code/cl8y-dex-terraclassic

Fork 0

Indexer route/solve 502s ('mirror simulation failed') on a healthy pair when a candidate path touches a zero-reserve pair (DB-hybrid) #369

New issue

Closed

opened 2026-06-12 05:56:13 +00:00 by Brouie · 19 comments

Brouie commented

2026-06-12 05:56:13 +00:00

(Migrated from gitlab.com)

Symptom

GET /api/v1/route/solve (and /best) returns 502 "Route mirror simulation failed" for EMBER↔CORAL, both directions, every size (1 EMBER through pool-capacity), on main 8408689 with ROUTE_SOLVER_DB_HYBRID=1.

GET /api/v1/route/solve?token_in=<EMBER>&token_out=<CORAL>&amount_in=1000000  -> "Route mirror simulation failed"
GET /api/v1/route/solve?token_in=<CORAL>&token_out=<EMBER>&amount_in=1000000  -> "Route mirror simulation failed"
GET /api/v1/route/solve/best?...                                              -> "Route mirror simulation failed"

Indexer /health is ok and the indexer is caught up (last_indexed 1463243 vs chain 1463246).

Root cause

best_execution.rs:161-171 maps HybridSimError::Db → 502 "Route mirror simulation failed". Log detail: db hybrid sim error detail=insufficient pool liquidity.
db_orderbook_sim.rs::simulate_pool_leg returns InsufficientLiquidity when input_reserve == 0 || output_reserve == 0.
But the direct EMBER/CORAL pool (pair_id 1) is healthy: DB pair_reserves = 96883935727 / 95441935304, matching live LCD {pool} exactly, snapshot age 0. So the zero-reserve leg is not the direct pair.
There are 2 zero-reserve pairs in pair_reserves: pair_id 152 and 205 (reserve_0=0, reserve_1=0). The solver evaluates a candidate path that includes one of these and hard-fails the whole request instead of (a) skipping that path for the viable direct route, or (b) degrading. Note quote_kind_for does degrade gracefully for mirror_stale / mirror_missing, but a zero-reserve mirror is treated as a hard error.

A viable direct route demonstrably exists (pair 1 has deep liquidity and the dapp swaps EMBER→CORAL fine via LCD sim, e.g. tx 1D82DB21… code 0), yet route/solve 502s.

Why this matters beyond QA state

Empty/unfunded pairs are a normal state — every create_pair starts at zero reserves until liquidity is added (CP-00-02). Under ROUTE_SOLVER_DB_HYBRID, any such pair can poison route/solve for other pairs that share a candidate path, returning a hard 502 instead of the working direct route. Impact: indexer route-compare, the dapp's >30% retail-slippage warning UX (it reads route/solve slippage_percent), and integrator /route/solve/best all degrade. On-chain swaps are unaffected (dapp + on-chain min_return still protect funds).

How it surfaced here

ROUTE_SOLVER_DB_HYBRID=1 is set on the QA indexer. Pairs 152/205 were left at zero by a book_snapshot "All LCD endpoints failed" transient (2026-06-11), but the same poisoning reproduces for any genuinely empty pair, so the transient is incidental.

Suggested direction

In the DB-hybrid mirror sim, treat a zero-reserve pool leg the same as a missing/unusable route: skip that path candidate (or degrade), don't 502 the whole request when another candidate (the direct pool) is viable.
Confirm whether production runs ROUTE_SOLVER_DB_HYBRID; if so this is launch-relevant for any pair set containing empty pairs.

Found during #337 browser pass (SEC-00-05 re-confirm: the >30% guard didn't fire precisely because route/solve was 502-ing).

## Symptom `GET /api/v1/route/solve` (and `/best`) returns **502 "Route mirror simulation failed"** for **EMBER↔CORAL, both directions, every size** (1 EMBER through pool-capacity), on main 8408689 with `ROUTE_SOLVER_DB_HYBRID=1`. ``` GET /api/v1/route/solve?token_in=<EMBER>&token_out=<CORAL>&amount_in=1000000 -> "Route mirror simulation failed" GET /api/v1/route/solve?token_in=<CORAL>&token_out=<EMBER>&amount_in=1000000 -> "Route mirror simulation failed" GET /api/v1/route/solve/best?... -> "Route mirror simulation failed" ``` Indexer `/health` is ok and the indexer is caught up (last_indexed 1463243 vs chain 1463246). ## Root cause - `best_execution.rs:161-171` maps `HybridSimError::Db` → 502 "Route mirror simulation failed". Log detail: `db hybrid sim error detail=insufficient pool liquidity`. - `db_orderbook_sim.rs::simulate_pool_leg` returns `InsufficientLiquidity` when `input_reserve == 0 || output_reserve == 0`. - But the **direct** EMBER/CORAL pool (pair_id 1) is healthy: DB `pair_reserves` = `96883935727 / 95441935304`, matching live LCD `{pool}` exactly, snapshot age 0. So the zero-reserve leg is **not** the direct pair. - There are **2 zero-reserve pairs in `pair_reserves`: pair_id 152 and 205** (`reserve_0=0, reserve_1=0`). The solver evaluates a candidate path that includes one of these and **hard-fails the whole request** instead of (a) skipping that path for the viable direct route, or (b) degrading. Note `quote_kind_for` *does* degrade gracefully for `mirror_stale` / `mirror_missing`, but a **zero-reserve** mirror is treated as a hard error. A viable direct route demonstrably exists (pair 1 has deep liquidity and the dapp swaps EMBER→CORAL fine via LCD sim, e.g. tx 1D82DB21… code 0), yet route/solve 502s. ## Why this matters beyond QA state Empty/unfunded pairs are a **normal** state — every `create_pair` starts at zero reserves until liquidity is added (CP-00-02). Under `ROUTE_SOLVER_DB_HYBRID`, any such pair can poison route/solve for **other** pairs that share a candidate path, returning a hard 502 instead of the working direct route. Impact: indexer route-compare, the dapp's >30% retail-slippage warning UX (it reads route/solve `slippage_percent`), and integrator `/route/solve/best` all degrade. On-chain swaps are unaffected (dapp + on-chain `min_return` still protect funds). ## How it surfaced here `ROUTE_SOLVER_DB_HYBRID=1` is set on the QA indexer. Pairs 152/205 were left at zero by a `book_snapshot` "All LCD endpoints failed" transient (2026-06-11), but the same poisoning reproduces for any genuinely empty pair, so the transient is incidental. ## Suggested direction - In the DB-hybrid mirror sim, treat a zero-reserve pool leg the same as a missing/unusable route: skip that path candidate (or degrade), don't 502 the whole request when another candidate (the direct pool) is viable. - Confirm whether production runs `ROUTE_SOLVER_DB_HYBRID`; if so this is launch-relevant for any pair set containing empty pairs. Found during #337 browser pass (SEC-00-05 re-confirm: the >30% guard didn't fire precisely because route/solve was 502-ing).

Brouie commented

2026-06-12 06:16:26 +00:00

(Migrated from gitlab.com)

mentioned in issue #337

PlasticDigits commented

2026-06-12 06:35:06 +00:00

(Migrated from gitlab.com)

mentioned in commit d847275c89

mentioned in commit d847275c8997203b6a2c7447558101ebddc652e2

PlasticDigits commented

2026-06-12 06:36:27 +00:00

(Migrated from gitlab.com)

mentioned in merge request !878

PlasticDigits commented

2026-06-12 07:57:59 +00:00

(Migrated from gitlab.com)

mentioned in commit d64052c857

mentioned in commit d64052c857e9339004836f0daa8e03ffe199a146

PlasticDigits commented

2026-06-12 08:06:07 +00:00

(Migrated from gitlab.com)

mentioned in commit 2d46702d23

mentioned in commit 2d46702d237baea7ae0d36f180206c33d101e912

PlasticDigits commented

2026-06-12 08:06:32 +00:00

(Migrated from gitlab.com)

mentioned in merge request !880

PlasticDigits commented

2026-06-12 08:33:53 +00:00

(Migrated from gitlab.com)

mentioned in commit faf127ce0a

mentioned in commit faf127ce0ad7addc71e225ca8b3adad7a15d380b

PlasticDigits commented

2026-06-12 11:01:44 +00:00

(Migrated from gitlab.com)

mentioned in commit 15551045b6

mentioned in commit 15551045b6f9cfa99cd00270004177006d99ada3

PlasticDigits commented

2026-06-12 11:09:44 +00:00

(Migrated from gitlab.com)

mentioned in merge request !885

PlasticDigits commented

2026-06-12 11:28:51 +00:00

(Migrated from gitlab.com)

mentioned in issue #363

PlasticDigits commented

2026-06-12 11:49:46 +00:00

(Migrated from gitlab.com)

mentioned in commit d269d11312

mentioned in commit d269d11312b5c947d2c3f11ffa63b2f025871170

PlasticDigits commented

2026-06-12 11:59:47 +00:00

(Migrated from gitlab.com)

mentioned in commit 9e4582b52f

mentioned in commit 9e4582b52f82c335497655929c7d8a69229b75d2

PlasticDigits commented

2026-06-12 11:59:50 +00:00

(Migrated from gitlab.com)

mentioned in merge request !887

PlasticDigits commented

2026-06-13 02:37:42 +00:00

(Migrated from gitlab.com)

mentioned in commit 83ae37217a

mentioned in commit 83ae37217a148467a154b19106ed9208ca088150

PlasticDigits commented

2026-06-13 02:52:43 +00:00

(Migrated from gitlab.com)

mentioned in merge request !890

PlasticDigits commented

2026-06-13 03:01:28 +00:00

(Migrated from gitlab.com)

mentioned in commit 43853ec6a4

mentioned in commit 43853ec6a421922a514941af31789f4cce74b8a1

Brouie commented

2026-06-13 03:07:27 +00:00

(Migrated from gitlab.com)

Verified on my side on main 87b6f9a (indexer rebuilt + restarted off it).

The hard 502 is gone. EMBER↔CORAL both directions, /route/solve/best, and a pool-capacity-size request all come back 200 now — and they do it with pairs 152 and 205 still sitting at zero reserves in pair_reserves, so the skip path is genuinely being exercised, not masked by clean data. The solve returns a real route (3-hop, paths_considered 5, estimated_amount_out populated, quote_kind indexer_pool_db).

Source matches the symptom: hybrid_route_opt.rs treats an unusable (zero-reserve) pool hop as skip/degrade-to-LCD-fallback instead of failing the whole request, while genuinely missing reserves still error.

verify-issue-369.sh: 5/5 green

lib pool_reserves_unusable
lib concurrent_eval_skips (failed candidates skipped)
lib concurrent_solve (fail-fast preserved for real errors)
integration route_solve_db_hybrid_skips_zero_reserve
full api_route_solve_db_hybrid suite

Good to close from my side. @PlasticDigits

Verified on my side on main 87b6f9a (indexer rebuilt + restarted off it). The hard 502 is gone. EMBER↔CORAL both directions, /route/solve/best, and a pool-capacity-size request all come back 200 now — and they do it with pairs 152 and 205 still sitting at zero reserves in pair_reserves, so the skip path is genuinely being exercised, not masked by clean data. The solve returns a real route (3-hop, paths_considered 5, estimated_amount_out populated, quote_kind indexer_pool_db). Source matches the symptom: hybrid_route_opt.rs treats an unusable (zero-reserve) pool hop as skip/degrade-to-LCD-fallback instead of failing the whole request, while genuinely missing reserves still error. verify-issue-369.sh: 5/5 green - lib pool_reserves_unusable - lib concurrent_eval_skips (failed candidates skipped) - lib concurrent_solve (fail-fast preserved for real errors) - integration route_solve_db_hybrid_skips_zero_reserve - full api_route_solve_db_hybrid suite Good to close from my side. @PlasticDigits

PlasticDigits commented

2026-06-13 03:11:05 +00:00

(Migrated from gitlab.com)

mentioned in commit bfd30abba9

mentioned in commit bfd30abba9cbc3e6bc7f270387383f03182b8c7e

PlasticDigits (Migrated from gitlab.com) closed this issue

2026-06-13 03:50:46 +00:00

PlasticDigits commented

2026-07-13 10:33:12 +00:00

(Migrated from gitlab.com)

mentioned in issue #485