Hybrid solver does serial LCD fanout and its result cache is caller-bypassable

Brouie commented

2026-06-03 07:12:05 +00:00

(Migrated from gitlab.com)

Severity: Medium-High (performance + request amplification)
Reachability: /route/solve and /route/solve/best — unauthenticated, though they do sit behind the lcd-heavy per-IP limit.
Affected: solve_global_best_execution (indexer/src/api/best_execution.rs) and the hybrid result cache (indexer/src/api/route_solver.rs).
Root cause: the solver evaluates every candidate path with serial LCD calls, and the result cache key varies on caller-controlled inputs, so the cache is easy to miss on purpose.

Summary

This is the "snappy solver" item. Two things make the solver slow and cheap to amplify:

Serial fanout. solve_global_best_execution loops over every candidate path and, inside the loop, awaits optimize_multihop_hybrid_joint (a per-hop LCD grid search) and then maybe_simulate (a router LCD sim) one path at a time. So a single request's latency is the sum of every candidate's LCD round-trips — hundreds of serial calls on a busy graph. Nothing here runs concurrently.
Bypassable cache. hybrid_cache_key is built from solver_version | token_in | token_out | amount_bucket | max_maker_fills | trader. max_maker_fills, the amount bucket, and trader are all caller-set, so bumping max_maker_fills 1→2→3… or hopping amount buckets misses the cache every time and forces a fresh full fanout. The cache barely protects the hot path.

The per-IP lcd-heavy limit caps request rate, but not per-request cost, and combined with the proxy-IP issue (separate report) the limit may not even bind. The fix is mostly performance work that also closes the amplification.

Current codebase

best_execution.rs solve_global_best_execution: for cand in &candidates { ... optimize_multihop_hybrid_joint(...).await ...; maybe_simulate(...).await ... } — serial, per candidate.
route_solver.rs hybrid_cache_key: includes max_maker_fills and trader; both caller-controlled.

Recommended direction

Run the per-candidate optimize+simulate concurrently (buffered join_all with a sane concurrency cap) instead of serially.
Bound the work: cap candidate-path count and per-hop grid resolution explicitly, and surface when the result was degraded by the cap (don't silently truncate).
Make the cache key robust: drop max_maker_fills from the key (or clamp it to a few discrete values), and bucket/normalize trader so honest variation still hits.

Acceptance criteria

Solver latency for an N-candidate request is ~slowest-candidate, not sum-of-candidates.
Varying max_maker_fills / amount within normal ranges hits the cache instead of forcing fresh fanouts.
Candidate count and grid size are explicitly bounded, and the response flags when a cap truncated the search.

Test plan (performance / abuse)

case	expect
dense graph, many candidates	bounded latency, concurrent LCD
repeated solves varying only max_maker_fills	cache hits after the first
solve with adversarially large fan	capped, flagged degraded, not unbounded

**Severity:** Medium-High (performance + request amplification) **Reachability:** `/route/solve` and `/route/solve/best` — unauthenticated, though they do sit behind the lcd-heavy per-IP limit. **Affected:** `solve_global_best_execution` (`indexer/src/api/best_execution.rs`) and the hybrid result cache (`indexer/src/api/route_solver.rs`). **Root cause:** the solver evaluates every candidate path with **serial** LCD calls, and the result cache key varies on caller-controlled inputs, so the cache is easy to miss on purpose. ## Summary This is the "snappy solver" item. Two things make the solver slow and cheap to amplify: 1. **Serial fanout.** `solve_global_best_execution` loops over every candidate path and, inside the loop, `await`s `optimize_multihop_hybrid_joint` (a per-hop LCD grid search) and then `maybe_simulate` (a router LCD sim) one path at a time. So a single request's latency is the *sum* of every candidate's LCD round-trips — hundreds of serial calls on a busy graph. Nothing here runs concurrently. 2. **Bypassable cache.** `hybrid_cache_key` is built from `solver_version | token_in | token_out | amount_bucket | max_maker_fills | trader`. `max_maker_fills`, the amount bucket, and `trader` are all caller-set, so bumping `max_maker_fills` 1→2→3… or hopping amount buckets misses the cache every time and forces a fresh full fanout. The cache barely protects the hot path. The per-IP lcd-heavy limit caps request *rate*, but not per-request *cost*, and combined with the proxy-IP issue (separate report) the limit may not even bind. The fix is mostly performance work that also closes the amplification. ## Current codebase - `best_execution.rs` `solve_global_best_execution`: `for cand in &candidates { ... optimize_multihop_hybrid_joint(...).await ...; maybe_simulate(...).await ... }` — serial, per candidate. - `route_solver.rs` `hybrid_cache_key`: includes `max_maker_fills` and `trader`; both caller-controlled. ## Recommended direction 1. Run the per-candidate optimize+simulate concurrently (buffered `join_all` with a sane concurrency cap) instead of serially. 2. Bound the work: cap candidate-path count and per-hop grid resolution explicitly, and surface when the result was degraded by the cap (don't silently truncate). 3. Make the cache key robust: drop `max_maker_fills` from the key (or clamp it to a few discrete values), and bucket/normalize `trader` so honest variation still hits. ## Acceptance criteria - [ ] Solver latency for an N-candidate request is ~slowest-candidate, not sum-of-candidates. - [ ] Varying `max_maker_fills` / amount within normal ranges hits the cache instead of forcing fresh fanouts. - [ ] Candidate count and grid size are explicitly bounded, and the response flags when a cap truncated the search. ## Test plan (performance / abuse) | case | expect | |---|---| | dense graph, many candidates | bounded latency, concurrent LCD | | repeated solves varying only max_maker_fills | cache hits after the first | | solve with adversarially large fan | capped, flagged degraded, not unbounded |

PlasticDigits commented

2026-06-03 10:46:31 +00:00

(Migrated from gitlab.com)

We must the orderbook and v2 pair data in postgres instead of relying on lcd calls for order simulation. There must be 0 lcd calls in the solver.

The indexer needs to simulate and optimize without any lcd calls.

Once that change is made, then can add concurrency, limit hops to 4, and fix the caching.

We must the orderbook and v2 pair data in postgres instead of relying on lcd calls for order simulation. There must be 0 lcd calls in the solver. The indexer needs to simulate and optimize without any lcd calls. Once that change is made, then can add concurrency, limit hops to 4, and fix the caching.

Brouie commented

2026-06-04 05:27:01 +00:00

(Migrated from gitlab.com)

mentioned in issue #285

Brouie commented

2026-06-04 05:27:04 +00:00

(Migrated from gitlab.com)

mentioned in merge request !744

Brouie commented

2026-06-04 06:29:25 +00:00

(Migrated from gitlab.com)

Implementation plan (your "0-LCD solver" direction). Large — the architectural keystone; phase it.

Phase 1 (must land first): mirror state into Postgres so the solver does 0 LCD calls. Two NEW sqlx migrations:

pair_reserves — brand new; reserves aren't event-sourced today, they only live on-chain behind {pool:{}}. Mirror RESERVES + fee_bps per pair (+ snapshot_at, height).
resting_limit_orders — a current-state materialized book, distinct from the append-only limit_order_placements/_cancellations/_fills logs (those lack current remaining + FIFO prev/next the walk needs).
Plus revert/*.down.sql. No backfill — a background snapshot loop populates both.

New modules: db queries (pair_reserves, resting_orders), a book_snapshot loop (copy oracle.rs::run_oracle_loop / trader_tracker::run_tier_sync_loop), and db_orderbook_sim (port the pool+book math — reuse orderbook_sim.rs/hybrid_orderbook_sim.rs already built for CG/CMC #220). Rewire hybrid_route_opt (grid search → DB) + best_execution (serial loop → join_all).

Phase 2: bump GET_DEFAULT_MAX_HOPS 3→4. Phase 3: cache-key fix — #283 folds in here, don't ship it separately.

Top risk: fidelity drift — if db_orderbook_sim diverges from on-chain HybridSimulation (rounding, dust-flush, expiry, tier discount) the solver quotes wrong; keep the final maybe_simulate LCD validation on the single winning route. Snapshot staleness → tight interval + treat missing/old snapshot as degrade-not-error. quote_kind enum rename (LCD→indexed) is a BREAKING API change → coordinate the frontend. Re-check the 4-hop bump against #286's path-candidate budget. Strong precedent on all three halves (CG/CMC sim, the snapshot-loop pattern, the synced tier in traders.tier_id). @PlasticDigits

Implementation plan (your "0-LCD solver" direction). Large — the architectural keystone; phase it. **Phase 1 (must land first): mirror state into Postgres so the solver does 0 LCD calls.** Two NEW sqlx migrations: - `pair_reserves` — brand new; reserves aren't event-sourced today, they only live on-chain behind `{pool:{}}`. Mirror RESERVES + fee_bps per pair (+ snapshot_at, height). - `resting_limit_orders` — a current-state materialized book, distinct from the append-only `limit_order_placements/_cancellations/_fills` logs (those lack current `remaining` + FIFO prev/next the walk needs). - Plus `revert/*.down.sql`. No backfill — a background snapshot loop populates both. New modules: db queries (pair_reserves, resting_orders), a `book_snapshot` loop (copy `oracle.rs::run_oracle_loop` / `trader_tracker::run_tier_sync_loop`), and `db_orderbook_sim` (port the pool+book math — **reuse `orderbook_sim.rs`/`hybrid_orderbook_sim.rs` already built for CG/CMC #220**). Rewire `hybrid_route_opt` (grid search → DB) + `best_execution` (serial loop → join_all). **Phase 2:** bump GET_DEFAULT_MAX_HOPS 3→4. **Phase 3:** cache-key fix — **#283 folds in here**, don't ship it separately. **Top risk: fidelity drift** — if `db_orderbook_sim` diverges from on-chain `HybridSimulation` (rounding, dust-flush, expiry, tier discount) the solver quotes wrong; keep the final `maybe_simulate` LCD validation on the single winning route. Snapshot staleness → tight interval + treat missing/old snapshot as degrade-not-error. **`quote_kind` enum rename (LCD→indexed) is a BREAKING API change → coordinate the frontend.** Re-check the 4-hop bump against #286's path-candidate budget. Strong precedent on all three halves (CG/CMC sim, the snapshot-loop pattern, the synced tier in `traders.tier_id`). @PlasticDigits

Brouie commented

2026-06-04 06:29:26 +00:00

(Migrated from gitlab.com)

mentioned in issue #283

Brouie commented

2026-06-05 01:35:20 +00:00

(Migrated from gitlab.com)

mentioned in merge request !751

Brouie commented

2026-06-05 03:23:17 +00:00

(Migrated from gitlab.com)

Touching this for status. It's the keystone — two new PG migrations (pair_reserves + resting_limit_orders), and the quote_kind rename is breaking into the frontend; #283 folds into its cache phase (I shipped the standalone cache-key correctness part as MR !751). I'm deliberately not landing a breaking, migration-heavy 0-LCD solver rewrite in one shot — it wants its own focused pass off the phase plan in my earlier note. Ready to take Phase 1 (mirror reserves + resting orders into Postgres, solver does 0 LCD) as a dedicated piece whenever you want to sequence it. @PlasticDigits

Brouie commented

2026-06-05 03:46:19 +00:00

(Migrated from gitlab.com)

Started it — Phase 1a is up (the schema foundation), MR above. Two new tables + their query layer, fully tested, and deliberately NOT wired into the solver yet so there's zero fidelity risk and nothing breaking:

pair_reserves — mirrors v2 pool state (reserve_0/1, fee_bps, height), one row/pair, upserted per snapshot (reserves aren't event-sourced today, so this is what the snapshot loop will fill).
resting_limit_orders — materialized current-state book with live remaining, indexed for the walk (price + FIFO), distinct from the append-only lifecycle logs.
Queries: pair_reserves upsert/get (missing -> None = degrade-not-error) and resting_orders atomic replace-pair-book + get-book ordered best-price-then-FIFO. revert/*.down.sql for both. Tests 3/3.

Next, as separate pieces because they need live fidelity validation: 1b the book_snapshot loop (populate from chain, modeled on the oracle/tier-sync loops), 1c db_orderbook_sim + rewire hybrid_route_opt/best_execution to read DB not LCD — keeping the final maybe_simulate LCD validation on the winning route as the drift guard. The breaking quote_kind rename rides with 1c, so we coordinate the frontend at that point. @PlasticDigits

Started it — Phase 1a is up (the schema foundation), MR above. Two new tables + their query layer, fully tested, and deliberately NOT wired into the solver yet so there's zero fidelity risk and nothing breaking: - pair_reserves — mirrors v2 pool state (reserve_0/1, fee_bps, height), one row/pair, upserted per snapshot (reserves aren't event-sourced today, so this is what the snapshot loop will fill). - resting_limit_orders — materialized current-state book with live remaining, indexed for the walk (price + FIFO), distinct from the append-only lifecycle logs. - Queries: pair_reserves upsert/get (missing -> None = degrade-not-error) and resting_orders atomic replace-pair-book + get-book ordered best-price-then-FIFO. revert/*.down.sql for both. Tests 3/3. Next, as separate pieces because they need live fidelity validation: 1b the book_snapshot loop (populate from chain, modeled on the oracle/tier-sync loops), 1c db_orderbook_sim + rewire hybrid_route_opt/best_execution to read DB not LCD — keeping the final maybe_simulate LCD validation on the winning route as the drift guard. The breaking quote_kind rename rides with 1c, so we coordinate the frontend at that point. @PlasticDigits

Brouie commented

2026-06-05 03:46:20 +00:00

(Migrated from gitlab.com)

mentioned in merge request !761

PlasticDigits commented

2026-06-05 03:49:54 +00:00

(Migrated from gitlab.com)

As the scope has expanded, please take all pieces into seperate issues and close this parent issue

PlasticDigits commented

2026-06-05 04:04:09 +00:00

(Migrated from gitlab.com)

mentioned in merge request !764

PlasticDigits commented

2026-06-05 04:08:16 +00:00

(Migrated from gitlab.com)

mentioned in issue #306

PlasticDigits commented

2026-06-05 04:19:52 +00:00

(Migrated from gitlab.com)

mentioned in issue #319

PlasticDigits commented

2026-06-05 04:19:53 +00:00

(Migrated from gitlab.com)

marked as related to #319

PlasticDigits commented

2026-06-05 04:35:30 +00:00

(Migrated from gitlab.com)

mentioned in commit 8ea4bc1420

mentioned in commit 8ea4bc1420869273b102acca681c6836f9e13fb1

Brouie commented

2026-06-05 08:23:01 +00:00

(Migrated from gitlab.com)

mentioned in issue #322

Brouie commented

2026-06-05 08:23:02 +00:00

(Migrated from gitlab.com)

mentioned in issue #323

Brouie commented

2026-06-05 08:23:04 +00:00

(Migrated from gitlab.com)

mentioned in issue #324

Brouie commented

2026-06-05 08:23:45 +00:00

(Migrated from gitlab.com)

Split into separate issues per your note. Mapping so nothing's lost — the parent's three acceptance items all carry over to the children:

0-LCD hybrid solver program:

Phase 1a (mirror schema: pair_reserves + resting_limit_orders + query layer) — DONE, MR !761 (merged 8ea4bc1).
Phase 1b (book_snapshot loop that fills the mirror from chain + freshness/staleness contract) — #322. The hard dependency 1c blocks on.
Phase 1c (db_orderbook_sim + rewire solver to 0-LCD + breaking quote_kind rename + poisoned-mirror fidelity) — #319 (already open).
Phase 2 (raise max hops 3->4) — #323.
Phase 3 (concurrent candidate solve + cache-key robustness: clamp max_maker_fills, bucket trader; the #283 tier keying already shipped in !751 and stays) — #324.

Sequencing matches your direction (0-LCD first, then concurrency / 4 hops / caching): 1a done -> 1b #322 -> 1c #319 -> 2 #323 + 3 #324. Each child carries the code anchors, acceptance criteria, and the fidelity/freshness guards.

Closing this parent. @PlasticDigits

Split into separate issues per your note. Mapping so nothing's lost — the parent's three acceptance items all carry over to the children: 0-LCD hybrid solver program: - Phase 1a (mirror schema: pair_reserves + resting_limit_orders + query layer) — DONE, MR !761 (merged 8ea4bc1). - Phase 1b (book_snapshot loop that fills the mirror from chain + freshness/staleness contract) — #322. The hard dependency 1c blocks on. - Phase 1c (db_orderbook_sim + rewire solver to 0-LCD + breaking quote_kind rename + poisoned-mirror fidelity) — #319 (already open). - Phase 2 (raise max hops 3->4) — #323. - Phase 3 (concurrent candidate solve + cache-key robustness: clamp max_maker_fills, bucket trader; the #283 tier keying already shipped in !751 and stays) — #324. Sequencing matches your direction (0-LCD first, then concurrency / 4 hops / caching): 1a done -> 1b #322 -> 1c #319 -> 2 #323 + 3 #324. Each child carries the code anchors, acceptance criteria, and the fidelity/freshness guards. Closing this parent. @PlasticDigits

Brouie (Migrated from gitlab.com) closed this issue

2026-06-05 08:23:46 +00:00

PlasticDigits commented

2026-06-05 10:05:46 +00:00

(Migrated from gitlab.com)

mentioned in merge request !787