Indexer hybrid route optimizer: pass book_start_hint for first live-side order (GitLab #289 follow-up) #332

New issue

2026-06-05T13:44:35Z

PlasticDigits commented

2026-06-05 13:44:35 +00:00

(Migrated from gitlab.com)

Parent

Follow-up to GitLab #289 (book head-clog / hybrid scan cap — open). Product hardening: indexer route optimizer should pass book_start_hint for hybrid hops.

Current codebase

On-chain: HybridSwapParams.book_start_hint: Option<u64> (smartcontracts/packages/dex-common/src/pair.rs) — optional start order id for book walk; validated for side (#272). Wrong-side hints fall back to head without cross-escrow drain.
Indexer optimizer: hybrid_route_opt.rs hybrid_sim_query always sends "book_start_hint": null to LCD HybridSimulation (line ~110). Optimized plan sets book_start_hint: None on output HybridHopJson (line ~327).
Route solve API: route_solver.rs forwards book_start_hint from hop JSON to router_operations when non-null — but optimizer never populates it.
#289 context: Legit takers can use book_start_hint to start past a clogged expired head once side validation exists — indexer should supply first live order id on the matching side when mirror/LCD book snapshot is available.

Why this is needed

Hybrid grid search simulates from book head every time — pessimistic when expired orders clog the head (#289 griefing vector mitigation for honest takers).
Quotes may degrade to pool-only (book_start_hint: null, degraded: true) when head simulation fails, even when live liquidity exists deeper in the book.
Frontend/dapp submitting indexer-solved routes cannot skip clogs without manual hint — product gap vs on-chain capability.

Constraints / guardrails

Side safety (#272): Hint must be on the correct side for the taker's match direction (bid vs ask). Reject or omit hint if mirror row side disagrees.
Staleness: Mirror/book snapshot TTL (book_snapshot_max_staleness_ms) — do not emit hint from stale mirror; degrade to null.
Simulation parity: Hint used in HybridSimulation must match hint in final router_operations payload.
Backward compatible: null hint remains valid default; no contract change required.
No hint forgery: Only emit order ids observed in indexer mirror or LCD book query for that pair/side.
Tests must not depend on production book depth — use seeded mirror fixtures.

Relevant files

Area	Path
Optimizer	`indexer/src/api/hybrid_route_opt.rs` — `hybrid_sim_query`, `optimize_hop_hybrid`, `HybridHopJson`
DB mirror	`indexer/src/api/db_orderbook_sim.rs`, `indexer/src/indexer/book_snapshot.rs`
Route solve	`indexer/src/api/route_solver.rs`, `best_execution.rs`
Tests	`indexer/tests/api_route_solve.rs`, `api_route_solve_db_hybrid.rs`
Contract validation	`smartcontracts/contracts/pair/src/orderbook.rs` — `resolve_book_start_hint`
Docs	`docs/route-solver.md`, `skills/AGENTS_INDEXER_HYBRID_BEST_EXECUTION.md`

Recommended direction

When loading hop mirror (or LCD book), find first live order on the taker's match side (not expired, remaining > 0).
Pass that order_id as book_start_hint in hybrid_sim_query and in returned HybridHopJson when book_input > 0.
If head is expired but live orders exist deeper, hint skips clog without changing contract scan cap.
If no live orders, leave null.
Add integration test: seeded book with expired head + live order at depth → optimizer returns non-null hint and better output than head-only sim.

Acceptance criteria

optimize_hop_hybrid sets book_start_hint to first live-side order id when book leg > 0 and mirror fresh.
HybridSimulation queries use same hint as output ops.
Stale/missing mirror → book_start_hint: null (no regression).
api_route_solve tests updated (replace hard-coded null expectations where hint expected).
Docs note hint behavior in route-solver glossary.

Test plan — all paths

Path	Expected
Pool-only hop	`book_start_hint` null
Book leg, empty book	null
Book leg, live head	hint = head id (or null if head is valid live)
Expired clog at head, live at depth	hint = first live id
Stale mirror	null; degraded if needed
Wrong-side id in mirror (corrupt)	Omit hint; fall back to contract head validation
Multihop optimize	Per-hop hints independent

Test plan — attack / abuse / hack vectors

Vector	Expected
Hint points to another user's order on wrong side	Contract #272 rejects; no cross-escrow
Stale hint from old mirror	Staleness gate omits hint
Attacker manipulates indexer mirror	Hint only from indexer DB/LCD — out of scope for on-chain; mirror integrity is separate (#190)

Verification criteria

cargo test --test api_route_solve green.
EXPLAIN not required — functional book tests pass.
Manual: LocalTerra pair with limit book → POST /route/solve returns non-null book_start_hint on book hop when live orders exist.

## Parent Follow-up to GitLab **#289** (book head-clog / hybrid scan cap — **open**). Product hardening: indexer route optimizer should pass `book_start_hint` for hybrid hops. ## Current codebase - **On-chain:** `HybridSwapParams.book_start_hint: Option<u64>` (`smartcontracts/packages/dex-common/src/pair.rs`) — optional start order id for book walk; validated for side (#272). Wrong-side hints fall back to head without cross-escrow drain. - **Indexer optimizer:** `hybrid_route_opt.rs` `hybrid_sim_query` always sends `"book_start_hint": null` to LCD `HybridSimulation` (line ~110). Optimized plan sets `book_start_hint: None` on output `HybridHopJson` (line ~327). - **Route solve API:** `route_solver.rs` forwards `book_start_hint` from hop JSON to `router_operations` when non-null — but optimizer never populates it. - **#289 context:** Legit takers can use `book_start_hint` to start past a clogged expired head once side validation exists — indexer should supply first **live** order id on the matching side when mirror/LCD book snapshot is available. ## Why this is needed - Hybrid grid search simulates from book **head** every time — pessimistic when expired orders clog the head (#289 griefing vector mitigation for honest takers). - Quotes may degrade to pool-only (`book_start_hint: null`, `degraded: true`) when head simulation fails, even when live liquidity exists deeper in the book. - Frontend/dapp submitting indexer-solved routes cannot skip clogs without manual hint — product gap vs on-chain capability. ## Constraints / guardrails - **Side safety (#272):** Hint must be on the **correct** side for the taker's match direction (bid vs ask). Reject or omit hint if mirror row side disagrees. - **Staleness:** Mirror/book snapshot TTL (`book_snapshot_max_staleness_ms`) — do not emit hint from stale mirror; degrade to `null`. - **Simulation parity:** Hint used in `HybridSimulation` must match hint in final `router_operations` payload. - **Backward compatible:** `null` hint remains valid default; no contract change required. - **No hint forgery:** Only emit order ids observed in indexer mirror or LCD book query for that pair/side. - Tests must not depend on production book depth — use seeded mirror fixtures. ## Relevant files | Area | Path | |------|------| | Optimizer | `indexer/src/api/hybrid_route_opt.rs` — `hybrid_sim_query`, `optimize_hop_hybrid`, `HybridHopJson` | | DB mirror | `indexer/src/api/db_orderbook_sim.rs`, `indexer/src/indexer/book_snapshot.rs` | | Route solve | `indexer/src/api/route_solver.rs`, `best_execution.rs` | | Tests | `indexer/tests/api_route_solve.rs`, `api_route_solve_db_hybrid.rs` | | Contract validation | `smartcontracts/contracts/pair/src/orderbook.rs` — `resolve_book_start_hint` | | Docs | `docs/route-solver.md`, `skills/AGENTS_INDEXER_HYBRID_BEST_EXECUTION.md` | ## Recommended direction 1. When loading hop mirror (or LCD book), find first **live** order on the taker's match side (not expired, remaining > 0). 2. Pass that `order_id` as `book_start_hint` in `hybrid_sim_query` and in returned `HybridHopJson` when `book_input > 0`. 3. If head is expired but live orders exist deeper, hint skips clog without changing contract scan cap. 4. If no live orders, leave `null`. 5. Add integration test: seeded book with expired head + live order at depth → optimizer returns non-null hint and better output than head-only sim. ## Acceptance criteria - [ ] `optimize_hop_hybrid` sets `book_start_hint` to first live-side order id when book leg > 0 and mirror fresh. - [ ] `HybridSimulation` queries use same hint as output ops. - [ ] Stale/missing mirror → `book_start_hint: null` (no regression). - [ ] `api_route_solve` tests updated (replace hard-coded `null` expectations where hint expected). - [ ] Docs note hint behavior in route-solver glossary. ## Test plan — all paths | Path | Expected | |------|----------| | Pool-only hop | `book_start_hint` null | | Book leg, empty book | null | | Book leg, live head | hint = head id (or null if head is valid live) | | Expired clog at head, live at depth | hint = first live id | | Stale mirror | null; degraded if needed | | Wrong-side id in mirror (corrupt) | Omit hint; fall back to contract head validation | | Multihop optimize | Per-hop hints independent | ## Test plan — attack / abuse / hack vectors | Vector | Expected | |--------|----------| | Hint points to another user's order on wrong side | Contract #272 rejects; no cross-escrow | | Stale hint from old mirror | Staleness gate omits hint | | Attacker manipulates indexer mirror | Hint only from indexer DB/LCD — out of scope for on-chain; mirror integrity is separate (#190) | ## Verification criteria - [ ] `cargo test --test api_route_solve` green. - [ ] `EXPLAIN` not required — functional book tests pass. - [ ] Manual: LocalTerra pair with limit book → `POST /route/solve` returns non-null `book_start_hint` on book hop when live orders exist.

PlasticDigits commented

2026-06-05 13:44:36 +00:00

(Migrated from gitlab.com)

marked as related to #289

ghost1 commented

2026-06-05 14:00:34 +00:00

(Migrated from gitlab.com)

mentioned in commit 63640f1c74

mentioned in commit 63640f1c746b9adbaacf4a299eeef69f345b2c91

PlasticDigits commented

2026-06-05 14:00:58 +00:00

(Migrated from gitlab.com)

mentioned in merge request !816

PlasticDigits commented

2026-06-05 14:01:07 +00:00

(Migrated from gitlab.com)

Implementation complete in !816.

Summary: global_v2 optimizer now emits book_start_hint from the first live same-side resting order when the mirror is fresh and book_input > 0. Stale mirror / LCD fallback omits hint.

Verification:

cd indexer && cargo test --lib db_orderbook_sim — PASS
cd indexer && cargo test --test api_route_solve --test api_route_solve_db_hybrid -- --test-threads=1 — PASS (26 tests)
Manual LocalTerra — SKIP (DB integration test covers hint paths)

Issue left open pending MR review/merge.

Implementation complete in [!816](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/98). **Summary:** `global_v2` optimizer now emits `book_start_hint` from the first live same-side resting order when the mirror is fresh and `book_input > 0`. Stale mirror / LCD fallback omits hint. **Verification:** - `cd indexer && cargo test --lib db_orderbook_sim` — PASS - `cd indexer && cargo test --test api_route_solve --test api_route_solve_db_hybrid -- --test-threads=1` — PASS (26 tests) - Manual LocalTerra — SKIP (DB integration test covers hint paths) Issue left **open** pending MR review/merge.

PlasticDigits commented

2026-06-05 14:26:53 +00:00

(Migrated from gitlab.com)

mentioned in commit 8edf0509e5

mentioned in commit 8edf0509e5534e1ca7851ad6467e2fec9741fca5

Brouie commented

2026-06-06 01:47:10 +00:00

(Migrated from gitlab.com)

#332 verified — checked on current main (merged #332 commit 63640f1). Indexer route-optimizer change; I covered source + unit + integration AND the live LocalTerra /route/solve check the implementation note marked SKIP.

The skipped manual check — done. Brought the indexer up in DB-hybrid mode (ROUTE_SOLVER_DB_HYBRID=1, global_v4), seeded a resting bid on a leaf pair (IRON/EMBER — IRON only routes through that one pair, so no multi-hop can steal the route), let the book_snapshot loop mirror it, then GET /api/v1/route/solve token_in=IRON token_out=EMBER amount_in=10000000:

quote_kind indexer_hybrid_db, single hop on the IRON/EMBER pair
router_operations[0].terra_swap.hybrid.book_input = 10000000, book_start_hint = 1 (the resting bid's order id) — non-null hint on the book hop, exactly the criterion.
Output ~2.97 EMBER/IRON sits between the pool (2.0) and the bid (3.0), so the book leg is really being priced with the hint. Negative case also checked live: when the global solver routed EMBER->CORAL through a pool-only multi-hop (no book leg on the winning path), book_start_hint stayed null — no false hint.

Acceptance criteria:

optimize_hop_hybrid sets book_start_hint to first live-side order when book leg>0 and mirror fresh: live (hint=1) + unit first_live_book_start_hint_picks_match_side; wrong-side omission covered by first_live_book_start_hint_omits_wrong_side_in_bid_list.
HybridSimulation uses same hint as output ops: hint applied in hybrid_sim_query and surfaced in router_operations (book-priced output confirms it).
Stale/missing mirror -> null: integration route_solve_db_hybrid_book_start_hint_paths (stale case) + Lcd source returns None by construction.
api_route_solve tests updated: route_solve_db_hybrid_book_start_hint_paths covers live-head=42, expired-head->skip-to-live=77, stale->null; passes.
Docs: route-solver.md glossary entry for book_start_hint (global_v2 first-live-side, fresh mirror, wrong-side omitted, on-chain L17/#272 authoritative at execute).

Verification criteria:

cargo test --test api_route_solve green: 23/23 + api_route_solve_db_hybrid 3/3 (incl the hint test).
EXPLAIN n/a.
Manual LocalTerra non-null hint on book hop: done — hint=1 as above.

One note: the hint path is DB-mirror-only (Lcd source -> None), so it only surfaces with ROUTE_SOLVER_DB_HYBRID=1 (global_v4), which is the #319+ production mode. Side safety stays on-chain authoritative (#272 / L17). @PlasticDigits — over to you for the verify-agent + close.

#332 verified — checked on current main (merged #332 commit 63640f1). Indexer route-optimizer change; I covered source + unit + integration AND the live LocalTerra /route/solve check the implementation note marked SKIP. The skipped manual check — done. Brought the indexer up in DB-hybrid mode (ROUTE_SOLVER_DB_HYBRID=1, global_v4), seeded a resting bid on a leaf pair (IRON/EMBER — IRON only routes through that one pair, so no multi-hop can steal the route), let the book_snapshot loop mirror it, then GET /api/v1/route/solve token_in=IRON token_out=EMBER amount_in=10000000: - quote_kind indexer_hybrid_db, single hop on the IRON/EMBER pair - router_operations[0].terra_swap.hybrid.book_input = 10000000, book_start_hint = 1 (the resting bid's order id) — non-null hint on the book hop, exactly the criterion. Output ~2.97 EMBER/IRON sits between the pool (2.0) and the bid (3.0), so the book leg is really being priced with the hint. Negative case also checked live: when the global solver routed EMBER->CORAL through a pool-only multi-hop (no book leg on the winning path), book_start_hint stayed null — no false hint. Acceptance criteria: - optimize_hop_hybrid sets book_start_hint to first live-side order when book leg>0 and mirror fresh: live (hint=1) + unit first_live_book_start_hint_picks_match_side; wrong-side omission covered by first_live_book_start_hint_omits_wrong_side_in_bid_list. - HybridSimulation uses same hint as output ops: hint applied in hybrid_sim_query and surfaced in router_operations (book-priced output confirms it). - Stale/missing mirror -> null: integration route_solve_db_hybrid_book_start_hint_paths (stale case) + Lcd source returns None by construction. - api_route_solve tests updated: route_solve_db_hybrid_book_start_hint_paths covers live-head=42, expired-head->skip-to-live=77, stale->null; passes. - Docs: route-solver.md glossary entry for book_start_hint (global_v2 first-live-side, fresh mirror, wrong-side omitted, on-chain L17/#272 authoritative at execute). Verification criteria: - cargo test --test api_route_solve green: 23/23 + api_route_solve_db_hybrid 3/3 (incl the hint test). - EXPLAIN n/a. - Manual LocalTerra non-null hint on book hop: done — hint=1 as above. One note: the hint path is DB-mirror-only (Lcd source -> None), so it only surfaces with ROUTE_SOLVER_DB_HYBRID=1 (global_v4), which is the #319+ production mode. Side safety stays on-chain authoritative (#272 / L17). @PlasticDigits — over to you for the verify-agent + close.

PlasticDigits commented

2026-06-06 06:54:41 +00:00

(Migrated from gitlab.com)

Verification — GitLab #332 (agent:verify)

Verified on branch cursor/gitlab-issue-verification-5f14 (includes merged commit 63640f1).

Issue: https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/332

Acceptance criteria

Criterion	Result	Evidence
`optimize_hop_hybrid` sets `book_start_hint` to first live-side order id when `book_input > 0` and mirror fresh	PASS	`resolve_hop_book_start_hint` → `first_live_book_start_hint` in `hybrid_route_opt.rs`; set on `HybridHopJson` when `best_book > 0`. Unit: `first_live_book_start_hint_picks_match_side`, `first_live_book_start_hint_omits_wrong_side_in_bid_list`. Integration: live head → hint `42`, expired head → hint `77`.
`HybridSimulation` queries use same hint as output ops	PASS	`book_start_hint` resolved once per hop; passed to `query_hybrid_sim_unified` (DB mirror + LCD fallback) and stored on returned `HybridHopJson`. LCD path omits hint when `book_input == 0`.
Stale/missing mirror → `book_start_hint: null`	PASS	`first_live_book_start_hint` returns `None` when `freshness != Fresh`; integration stale case asserts `book_start_hint` null; `HybridSimSource::Lcd` → `None` by construction.
`api_route_solve` tests updated	PASS	`route_solve_db_hybrid_book_start_hint_paths` covers live head, expired-head skip, stale mirror.
Docs note hint behavior in route-solver glossary	PASS	`docs/route-solver.md` glossary entry for `book_start_hint`; `skills/AGENTS_INDEXER_HYBRID_BEST_EXECUTION.md` + `skills/AGENTS_BOOK_MATCH_HINT_SECURITY.md`.

Verification criteria

Check	Result	Command / notes
`cargo test --test api_route_solve`	PASS	23/23
`cargo test --test api_route_solve_db_hybrid`	PASS	3/3 (incl. `route_solve_db_hybrid_book_start_hint_paths`)
`cargo test --lib db_orderbook_sim`	PASS	6/6
EXPLAIN	SKIP	Not required per issue
Manual LocalTerra `/route/solve` non-null hint on book hop	PASS	Prior live verification on `main` by @Brouie (hint=1 on IRON→EMBER book hop, `ROUTE_SOLVER_DB_HYBRID=1` / `global_v4`). This run did not re-provision LocalTerra (no `.env.local`); integration test exercises the same HTTP + DB-mirror path.

Test plan paths (automated coverage)

Path	Result
Pool-only hop → null hint	PASS (pool-only returns `None` hop json; hint only when `best_book > 0`)
Book leg, empty book	PASS (no live orders → `first_live_book_start_hint` → `None`)
Book leg, live head	PASS (hint = head id `42`)
Expired clog at head, live at depth	PASS (hint = `77`)
Stale mirror	PASS (hint null, degraded quote)
Wrong-side corrupt mirror row	PASS (unit `first_live_book_start_hint_omits_wrong_side_in_bid_list`)
Multihop per-hop hints independent	PASS (hint resolved per `HopDescriptor` in `optimize_one_hop`)

All acceptance and verification criteria pass. Closing #332.

## Verification — GitLab #332 (agent:verify) Verified on branch `cursor/gitlab-issue-verification-5f14` (includes merged commit `63640f1`). Issue: https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/332 ### Acceptance criteria | Criterion | Result | Evidence | |-----------|--------|----------| | `optimize_hop_hybrid` sets `book_start_hint` to first live-side order id when `book_input > 0` and mirror fresh | **PASS** | `resolve_hop_book_start_hint` → `first_live_book_start_hint` in `hybrid_route_opt.rs`; set on `HybridHopJson` when `best_book > 0`. Unit: `first_live_book_start_hint_picks_match_side`, `first_live_book_start_hint_omits_wrong_side_in_bid_list`. Integration: live head → hint `42`, expired head → hint `77`. | | `HybridSimulation` queries use same hint as output ops | **PASS** | `book_start_hint` resolved once per hop; passed to `query_hybrid_sim_unified` (DB mirror + LCD fallback) and stored on returned `HybridHopJson`. LCD path omits hint when `book_input == 0`. | | Stale/missing mirror → `book_start_hint: null` | **PASS** | `first_live_book_start_hint` returns `None` when `freshness != Fresh`; integration stale case asserts `book_start_hint` null; `HybridSimSource::Lcd` → `None` by construction. | | `api_route_solve` tests updated | **PASS** | `route_solve_db_hybrid_book_start_hint_paths` covers live head, expired-head skip, stale mirror. | | Docs note hint behavior in route-solver glossary | **PASS** | `docs/route-solver.md` glossary entry for `book_start_hint`; `skills/AGENTS_INDEXER_HYBRID_BEST_EXECUTION.md` + `skills/AGENTS_BOOK_MATCH_HINT_SECURITY.md`. | ### Verification criteria | Check | Result | Command / notes | |-------|--------|-----------------| | `cargo test --test api_route_solve` | **PASS** | 23/23 | | `cargo test --test api_route_solve_db_hybrid` | **PASS** | 3/3 (incl. `route_solve_db_hybrid_book_start_hint_paths`) | | `cargo test --lib db_orderbook_sim` | **PASS** | 6/6 | | EXPLAIN | **SKIP** | Not required per issue | | Manual LocalTerra `/route/solve` non-null hint on book hop | **PASS** | Prior live verification on `main` by @Brouie (hint=1 on IRON→EMBER book hop, `ROUTE_SOLVER_DB_HYBRID=1` / `global_v4`). This run did not re-provision LocalTerra (no `.env.local`); integration test exercises the same HTTP + DB-mirror path. | ### Test plan paths (automated coverage) | Path | Result | |------|--------| | Pool-only hop → null hint | **PASS** (pool-only returns `None` hop json; hint only when `best_book > 0`) | | Book leg, empty book | **PASS** (no live orders → `first_live_book_start_hint` → `None`) | | Book leg, live head | **PASS** (hint = head id `42`) | | Expired clog at head, live at depth | **PASS** (hint = `77`) | | Stale mirror | **PASS** (hint null, degraded quote) | | Wrong-side corrupt mirror row | **PASS** (unit `first_live_book_start_hint_omits_wrong_side_in_bid_list`) | | Multihop per-hop hints independent | **PASS** (hint resolved per `HopDescriptor` in `optimize_one_hop`) | All acceptance and verification criteria pass. Closing #332.

PlasticDigits (Migrated from gitlab.com) closed this issue

2026-06-06 06:54:46 +00:00

PlasticDigits commented

2026-06-08 08:43:13 +00:00

(Migrated from gitlab.com)

mentioned in commit bb38c9c219

mentioned in commit bb38c9c219392c13cd0825cde58aa001f8dc4d28

PlasticDigits commented

2026-06-08 08:43:13 +00:00

(Migrated from gitlab.com)

mentioned in commit eaa159e0b4

mentioned in commit eaa159e0b4142a0cb138ea3a10963ba3fdd3cebb