Indexer: short-circuit empty-book hybrid grid + stop LCD 500 fan-out on CosmWasm rejects #493

New issue

Closed

opened 2026-07-15 03:49:13 +00:00 by PlasticDigits · 8 comments

PlasticDigits commented

2026-07-15 03:49:13 +00:00

(Migrated from gitlab.com)

Summary

Follow-ups from production Ember→Coral swap latency on dex.cl8y.com / indexer.dex.cl8y.com (while still on global_v3).

Ops note: ROUTE_SOLVER_DB_HYBRID=1 (global_v4) has been enabled on prod and is redeploying — that is the main latency fix (#319). This issue covers two remaining hardening items that still matter under LCD fallback / flaky public LCDs.

Short-circuit empty-book / pool-winning hops so we do not run the full 17×3 hybrid LCD (or even DB) grid when the book cannot improve on pool-only.
Tighten LCD HTTP 500 handling so non-transient CosmWasm smart-query rejects do not fan out across all configured LCD endpoints.

Related: #485 (distant-pair latency / progress), #491 (prod indexer 503), ADR 0002 / docs/route-solver.md.

Evidence (pre–DB-hybrid redeploy)

Cold GET /api/v1/route/solve for mainnet EMBER→CORAL (direct pair exists):

Mode	Wall time	Result
Default hybrid (`global_v3`)	~4.5–6.6s	`quote_kind=indexer_pool_lcd`, `hybrid: null` (pool-only)
Warm cache (≤12s TTL)	~0.8s	same body
`pool_only=true`	~0.7–0.9s	same output

Indexer log pattern:

route best execution solver_version="global_v3" paths_considered=1 db_queries=0 lcd_queries=51
  candidate_ms=5350 total_ms=5350
WARN lcd: LCD upstream returned non-success status endpoint_idx=0..2 status=500 path="/cosmwasm/..."

Direct 1-hop path; solver still estimated 51 LCD hybrid sims (estimate_lcd_calls: 17 + 2×17 per hop).
Final plan was pool-only — same as skipping the grid.
Concurrent WARNs show endpoint fan-out 0→1→2 on HTTP 500 during the solve.

Public LCD smart-query samples were ~0.3–0.6s each; sequential grid explains the ~5s cold path.

1) Short-circuit empty-book / pool-winning hops

Problem

optimize_one_hop / optimize_multihop_hybrid_joint always runs the full 17 book fractions grid, then 2 coordinate passes, even when:

the hop has no resting book that can match the offer side, or
the best split is already book_input = 0 after the first pass.

Under global_v3 this is dozens of sequential LCD HybridSimulation calls. Under global_v4 it is cheaper but still wasted CPU on empty books; LCD fallback paths remain expensive when mirrors are stale/missing.

Proposed behavior

Without shrinking documented optimality bounds for hops that do have a live book:

If mirror (DB mode) or a cheap book probe shows no fillable resting orders on the taker match side → price pool-only once and skip the 17-point grid + coordinate re-grid for that hop.
Optional: if baseline pass picks book_input = 0 on a 1-hop path, skip further coordinate passes (they re-run the same empty-book grid).
Keep solver_version / optimality_scope honest; document any early-exit in hybrid_notes / route-solver docs if behavior is user-visible.

Code map

indexer/src/api/hybrid_route_opt.rs — optimize_one_hop, optimize_multihop_hybrid_joint, GRID_POINTS=17
indexer/src/api/best_execution.rs — estimate_lcd_calls, candidate evaluation
indexer/src/api/db_orderbook_sim.rs — mirror / resting orders for DB short-circuit

Acceptance

Empty-book direct pair (e.g. EMBER/CORAL with no live ask/bid for the offer) cold-solves without full 17×3 grid work (assert via db_hybrid_queries / actual LCD fallback counters in tests).
Hops with a live book still run the grid; no silent optimality regression without a version bump + docs.
Unit/integration coverage for empty-book short-circuit + non-empty book still optimizes.
Docs: docs/route-solver.md / invariants note the short-circuit.

2) Stop LCD 500 fan-out on CosmWasm rejects

Problem

In indexer/src/lcd/mod.rs, only a narrow set of HTTP 500 bodies are treated as deterministic contract rejects (pair not found, unknown variant, error parsing into type). Other CosmWasm 500s:

emit WARN per endpoint,
do not cooldown (only 429/408/502+),
continue to the next LCD endpoint — same deterministic contract answer, 3× latency + log spam.

Observed during Ember→Coral quoting when all three endpoints returned 500 on /cosmwasm/....

Proposed behavior

Broaden classification of deterministic CosmWasm smart-query failures (contract error / query reject encoded as HTTP 500) so we return ContractQueryRejected (or equivalent) without trying sibling endpoints.
Keep fan-out + cooldown for true infra failures (timeouts, 429, 502/503/504, connection errors).
Prefer DEBUG for deterministic rejects; reserve WARN for infra/outage paths (aligns with existing foreign-pair discovery behavior).

Acceptance

Same CosmWasm 500 body on endpoint 0 does not query endpoints 1 and 2.
Infra 502/503/timeout still fail over / cooldown as today.
Regression tests in indexer/src/lcd (wiremock) for reject-vs-failover.
No increase in false “all endpoints failed” for real outages.

Out of scope

Enabling ROUTE_SOLVER_DB_HYBRID on prod (already done / in progress).
Frontend Calculating… / progress UX (#485).
CoinGecko terrausd.usd oracle parse warnings (unrelated to quote path).

Test plan

cd indexer && cargo test --lib lcd hybrid_route_opt -- --test-threads=1
cd indexer && cargo test --test api_route_solve --test api_route_solve_db_hybrid -- --test-threads=1

After deploy: cold Ember→CORAL (and an empty-book pair) should not show multi-endpoint 500 WARN storms; hybrid metadata should show far fewer LCD fallbacks when the book is empty.

## Summary Follow-ups from production Ember→Coral swap latency on `dex.cl8y.com` / `indexer.dex.cl8y.com` (while still on `global_v3`). **Ops note:** `ROUTE_SOLVER_DB_HYBRID=1` (`global_v4`) has been enabled on prod and is redeploying — that is the main latency fix (#319). This issue covers two remaining hardening items that still matter under LCD fallback / flaky public LCDs. 1. **Short-circuit empty-book / pool-winning hops** so we do not run the full 17×3 hybrid LCD (or even DB) grid when the book cannot improve on pool-only. 2. **Tighten LCD HTTP 500 handling** so non-transient CosmWasm smart-query rejects do not fan out across all configured LCD endpoints. Related: #485 (distant-pair latency / progress), #491 (prod indexer 503), ADR 0002 / `docs/route-solver.md`. --- ## Evidence (pre–DB-hybrid redeploy) Cold `GET /api/v1/route/solve` for mainnet **EMBER→CORAL** (direct pair exists): | Mode | Wall time | Result | |------|-----------|--------| | Default hybrid (`global_v3`) | **~4.5–6.6s** | `quote_kind=indexer_pool_lcd`, `hybrid: null` (pool-only) | | Warm cache (≤12s TTL) | ~0.8s | same body | | `pool_only=true` | **~0.7–0.9s** | same output | Indexer log pattern: ```text route best execution solver_version="global_v3" paths_considered=1 db_queries=0 lcd_queries=51 candidate_ms=5350 total_ms=5350 WARN lcd: LCD upstream returned non-success status endpoint_idx=0..2 status=500 path="/cosmwasm/..." ``` - Direct 1-hop path; solver still estimated **51** LCD hybrid sims (`estimate_lcd_calls`: 17 + 2×17 per hop). - Final plan was **pool-only** — same as skipping the grid. - Concurrent WARNs show endpoint fan-out 0→1→2 on HTTP 500 during the solve. Public LCD smart-query samples were ~0.3–0.6s each; sequential grid explains the ~5s cold path. --- ## 1) Short-circuit empty-book / pool-winning hops ### Problem `optimize_one_hop` / `optimize_multihop_hybrid_joint` always runs the full **17 book fractions** grid, then **2 coordinate passes**, even when: - the hop has **no resting book** that can match the offer side, or - the best split is already `book_input = 0` after the first pass. Under `global_v3` this is dozens of sequential LCD `HybridSimulation` calls. Under `global_v4` it is cheaper but still wasted CPU on empty books; LCD fallback paths remain expensive when mirrors are stale/missing. ### Proposed behavior Without shrinking documented optimality bounds for hops that *do* have a live book: - If mirror (DB mode) or a cheap book probe shows **no fillable resting orders** on the taker match side → price **pool-only once** and skip the 17-point grid + coordinate re-grid for that hop. - Optional: if baseline pass picks `book_input = 0` on a **1-hop** path, skip further coordinate passes (they re-run the same empty-book grid). - Keep `solver_version` / `optimality_scope` honest; document any early-exit in `hybrid_notes` / route-solver docs if behavior is user-visible. ### Code map - `indexer/src/api/hybrid_route_opt.rs` — `optimize_one_hop`, `optimize_multihop_hybrid_joint`, `GRID_POINTS=17` - `indexer/src/api/best_execution.rs` — `estimate_lcd_calls`, candidate evaluation - `indexer/src/api/db_orderbook_sim.rs` — mirror / resting orders for DB short-circuit ### Acceptance - [ ] Empty-book direct pair (e.g. EMBER/CORAL with no live ask/bid for the offer) cold-solves without full 17×3 grid work (assert via `db_hybrid_queries` / actual LCD fallback counters in tests). - [ ] Hops with a live book still run the grid; no silent optimality regression without a version bump + docs. - [ ] Unit/integration coverage for empty-book short-circuit + non-empty book still optimizes. - [ ] Docs: `docs/route-solver.md` / invariants note the short-circuit. --- ## 2) Stop LCD 500 fan-out on CosmWasm rejects ### Problem In `indexer/src/lcd/mod.rs`, only a narrow set of HTTP 500 bodies are treated as deterministic contract rejects (`pair not found`, `unknown variant`, `error parsing into type`). Other CosmWasm 500s: - emit **WARN** per endpoint, - **do not** cooldown (only 429/408/502+), - **continue** to the next LCD endpoint — same deterministic contract answer, 3× latency + log spam. Observed during Ember→Coral quoting when all three endpoints returned 500 on `/cosmwasm/...`. ### Proposed behavior - Broaden classification of **deterministic CosmWasm smart-query failures** (contract error / query reject encoded as HTTP 500) so we return `ContractQueryRejected` (or equivalent) **without** trying sibling endpoints. - Keep fan-out + cooldown for true infra failures (timeouts, 429, 502/503/504, connection errors). - Prefer DEBUG for deterministic rejects; reserve WARN for infra/outage paths (aligns with existing foreign-pair discovery behavior). ### Acceptance - [ ] Same CosmWasm 500 body on endpoint 0 does not query endpoints 1 and 2. - [ ] Infra 502/503/timeout still fail over / cooldown as today. - [ ] Regression tests in `indexer/src/lcd` (wiremock) for reject-vs-failover. - [ ] No increase in false “all endpoints failed” for real outages. --- ## Out of scope - Enabling `ROUTE_SOLVER_DB_HYBRID` on prod (already done / in progress). - Frontend Calculating… / progress UX (#485). - CoinGecko `terrausd.usd` oracle parse warnings (unrelated to quote path). ## Test plan ```bash cd indexer && cargo test --lib lcd hybrid_route_opt -- --test-threads=1 cd indexer && cargo test --test api_route_solve --test api_route_solve_db_hybrid -- --test-threads=1 ``` After deploy: cold Ember→CORAL (and an empty-book pair) should not show multi-endpoint 500 WARN storms; hybrid metadata should show far fewer LCD fallbacks when the book is empty.

PlasticDigits commented

2026-07-15 03:49:14 +00:00

(Migrated from gitlab.com)

marked as related to #485