#694 - sec: indexer API4 — cap /gt/events, LCD-heavy progress + blacklist-check (RE-01/02/03) - code/cl8y-dex-terraclassic

PlasticDigits commented

2026-08-28 09:24:15 +00:00

(Migrated from gitlab.com)

Gate

P1 availability / OWASP API4. Internal audit INTERNAL_GROK46_1787908099 findings RE-01, RE-02, RE-03 (Chain G2). No fund-loss path. Request-count governors already exist (RATE_LIMIT_RPS=60, RATE_LIMIT_LCD_HEAVY_RPS=10, prod zero-clamp). This ticket is per-request cost: one HTTP hit must not do unbounded SQL or unbounded LCD work.

Related (do not reopen): #646 GT adapters, #485 progress poll, #379 / #363 / #355 RPS governors, #278 CG/CMC orderbook on LCD-heavy, #308 blacklist.

Current codebase

Rate limits are request-count (tower_governor, PeerIpKeyExtractor only — no XFF). LCD-heavy router (10 RPS) includes /api/v1/route/solve, /best, limit-book, CG/CMC orderbook. Everything else, including the three surfaces below, sits on the global 60 RPS router after merge(lcd_heavy_router).

ID	Surface	Today
RE-01	`GET /gt/events`	`MAX_EVENT_BLOCK_SPAN = 2000` then `SELECT` swaps + liquidity with no `LIMIT`, then `SELECT pair_id, reserve_0, reserve_1 FROM pair_reserves` (all rows) to stamp current reserves. Range rejection only in `tests/api_gt.rs`.
RE-02	`GET /api/v1/route/solve/progress`	Mounted on the global router. Handler calls `resolve_discount_bps`, which LCD-`GetDiscount` when `trader`/`sender` is set and `fee_discount_address` is configured. dApp `useRouteSolveProgress` polls ~1 Hz while a sim quote is in flight. Solve itself is cached (512 / 12s–90s); progress is not on LCD-heavy and has no 429 test.
RE-03	`GET /api/v1/compliance/blacklist-check`	`tokens` and `pairs` are comma-split with no max length, then forwarded to factory `BlacklistCheck` on LCD. Global governor only. Tests: sanitized 502 only.

Honest Swap/Trade tabs already amplify RE-02 (progress + refetchInterval). Behind NAT, one peer IP shares a bucket.

Why the new implementation is needed

OWASP API4 (unrestricted resource consumption): limits that count requests miss work per request. A busy 2k-block /gt/events window can return a multi-MB JSON payload and scan every reserve row. Progress + live GetDiscount at 60 RPS (distinct trader values) amplifies public LCD. Unbounded blacklist lists do the same to the factory query. Saturation looks like “DEX down” even though contracts still execute.

Listing partners (GeckoTerminal) and the official dApp are friendly load. An unauthenticated public API must bound both.

Constraints / guardrails

Do not trust X-Forwarded-For / CF-Connecting-IP (#282). Keep PeerIpKeyExtractor.
Do not disable or zero rate limits on non-loopback without ALLOW_ZERO_RATE_LIMITS=1. Prod clamp stays.
Do not change GeckoTerminal wire shape except to truncate or 400 when over cap. Gem exclusion (EXCLUDED_CW20 / L639-2) stays. Reserves stay current snapshot, not historical reconstruction.
Do not make progress poll gate submit or change You Receive (#484). Progress stays advisory.
Do not fail-open blacklist. Oversize lists → 400, not a silent truncate that under-blocks. Factory query errors stay 502 sanitized (LCD_UPSTREAM_GATEWAY_MSG).
Do not add per-swap FoT math or new HTTP write APIs.
Keep CORS allowlist. Swagger may stay mounted.
Frontend backoff (RE-04 class) is in scope only as: stop sending trader on progress if discount is already on the in-flight quote, and/or back off after N consecutive progress failures. Do not rewrite all refetchIntervals.

Relevant files

Path	Role
`indexer/src/api/gt.rs`	`/gt/events` queries + full `pair_reserves` scan
`indexer/src/api/mod.rs`	Router split: `lcd_heavy_router` vs global
`indexer/src/api/route_solve_progress.rs`	Progress handler
`indexer/src/api/route_solver.rs`	`resolve_discount_bps` LCD `GetDiscount`
`indexer/src/api/compliance.rs`	Unbounded `tokens` / `pairs`
`indexer/src/config.rs`	`DEFAULT_RATE_LIMIT_*`, prod clamp
`indexer/tests/security.rs`	429 / 400 / 413 / 502 matrix
`indexer/tests/api_gt.rs`	GT range tests
`indexer/tests/api_route_solve.rs`	Progress lifecycle
`frontend-dapp/src/hooks/useRouteSolveProgress.ts`	~1 Hz poll + `trader`
`frontend-dapp/src/utils/quoteDebounce.ts`	Sim refetch skip (leave as-is)
`docs/indexer-invariants.md`	Document new caps

Recommended direction

RE-01: Add MAX_GT_EVENT_ROWS (recommend 5000 combined swap+liq). If the window would exceed, return 400 with a stable message (or return the first N and set a documented truncated field — pick one and test it). Load reserves only for pair_ids present in the window (WHERE pair_id = ANY($ids)), never the full table.
RE-02: Move /api/v1/route/solve/progress onto lcd_heavy_router (same 10 RPS as solve). Cache discount_bps per (trader, sender) with the existing 12s route-solve TTL (or reuse the progress registry entry) so a 1 Hz poll does not LCD every tick. Skip LCD when both trader fields are absent (already returns 0).
RE-03: Cap tokens at 16 and pairs at 8 (after trim/empty-drop). Oversize → 400. Move the route onto lcd_heavy_router.
Frontend (small): useRouteSolveProgress — omit trader when the parent quote already resolved discount; abort/backoff after consecutive failures. Do not change quote math.
Tests: extend security.rs (and GT/progress files) per test plan. make verify-issue-<this> greps the caps + router membership + new tests.

Acceptance criteria

/gt/events never returns more than the documented row cap; over-cap is 400 or explicit truncation (documented).
/gt/events does not SELECT all of pair_reserves.
Progress and blacklist-check are on the LCD-heavy governor (same layer as /route/solve).
Progress with trader set does not LCD-GetDiscount on every poll within the cache TTL.
Blacklist tokens/pairs over the cap → 400; at-cap still queries factory; factory error still sanitized 502.
Existing GT gem exclusion, solve cache, CORS, prod RPS clamp, and Peer-IP extractor unchanged.
indexer/tests/security.rs covers 429 on progress + blacklist-check (and solve /best if missing) and the new 400s.
make verify-issue-<N> exists and is wired in the Makefile / AGENTS.md indexer table.
docs/indexer-invariants.md records the new caps and which routes are LCD-heavy.

Test plan — happy / bad path

Path	Expect
`/gt/events` small window, few rows	200, same JSON shape, gems still excluded
`/gt/events` `to-from+1 > 2000`	400 (existing)
`/gt/events` window with >cap rows (seed)	400 or truncated per chosen policy
`/gt/events` many pairs in window	Reserves only for those pair_ids (unit or SQL assert)
`GET /route/solve` + `/progress` without trader	200; progress cheap; no LCD discount call
Same with `trader` set, 2+ polls inside TTL	At most one LCD `GetDiscount` (mock LCD call count)
Progress burst at test RPS	429 on LCD-heavy
`blacklist-check` 1 wallet + 2 tokens + 1 pair	200 (LCD mock)
`tokens` with 17 addresses	400
`pairs` with 9 addresses	400
Empty tokens/pairs omitted	Still 200
Factory LCD down	502, body has no sqlx/stack
Prod `RATE_LIMIT_*=0`	Still clamps to 60/10 (existing)
dApp progress: quote in flight	Still advisory; submit not gated
dApp progress: omit trader when discount known	No trader query param

Test plan — attack / abuse vectors

Vector	Expect
Repeat max-span `/gt/events` at 60 RPS	Row cap + optional GT governor; no unbounded JSON; no full-table reserve scan
Progress flood with distinct `trader`s	LCD-heavy 10 RPS; discount cache prevents 1:1 LCD
`blacklist-check?tokens=a,a,a…` (thousands)	400 before LCD
XFF spoof to bypass limiter	Still keyed on peer IP
SQL / comma injection in tokens	Still bound LCD JSON; 400 if over cap; no SQLi
Compression bomb / huge GT response	Cap makes payload bounded
NAT-shared honest tabs	May 429 together (accepted); backoff reduces friendly load

Verification criteria

cd indexer && cargo test --test security -- --test-threads=1 includes the new 400/429 cases.
make verify-issue-<N> green: greps MAX_GT_EVENT_ROWS (or equivalent), progress + blacklist-check on lcd_heavy_router, list caps, and the new test names.
Coolify/prod: after deploy, curl max-span /gt/events does not grow without bound; progress with a registered trader does not 1:1 LCD (log or mock).
Swap page still shows progress while a slow quote runs; Place/Swap still works.
Audit note: RE-01/02/03 closed in INTERNAL_GROK46_1787908099 follow-up when this lands.

## Gate **P1 availability / OWASP API4.** Internal audit `INTERNAL_GROK46_1787908099` findings **RE-01**, **RE-02**, **RE-03** (Chain G2). No fund-loss path. Request-count governors already exist (`RATE_LIMIT_RPS=60`, `RATE_LIMIT_LCD_HEAVY_RPS=10`, prod zero-clamp). This ticket is **per-request cost**: one HTTP hit must not do unbounded SQL or unbounded LCD work. Related (do not reopen): [#646](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/646) GT adapters, [#485](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/485) progress poll, [#379](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/379) / [#363](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/363) / [#355](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/355) RPS governors, [#278](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/278) CG/CMC orderbook on LCD-heavy, [#308](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/308) blacklist. ## Current codebase Rate limits are **request-count** (`tower_governor`, `PeerIpKeyExtractor` only — no XFF). LCD-heavy router (10 RPS) includes `/api/v1/route/solve`, `/best`, limit-book, CG/CMC orderbook. Everything else, including the three surfaces below, sits on the **global 60 RPS** router after `merge(lcd_heavy_router)`. | ID | Surface | Today | |----|---------|--------| | RE-01 | `GET /gt/events` | `MAX_EVENT_BLOCK_SPAN = 2000` then `SELECT` swaps + liquidity **with no `LIMIT`**, then `SELECT pair_id, reserve_0, reserve_1 FROM pair_reserves` (**all rows**) to stamp current reserves. Range rejection only in `tests/api_gt.rs`. | | RE-02 | `GET /api/v1/route/solve/progress` | Mounted on the **global** router. Handler calls `resolve_discount_bps`, which LCD-`GetDiscount` when `trader`/`sender` is set and `fee_discount_address` is configured. dApp `useRouteSolveProgress` polls ~1 Hz while a sim quote is in flight. Solve itself is cached (512 / 12s–90s); progress is not on LCD-heavy and has **no 429 test**. | | RE-03 | `GET /api/v1/compliance/blacklist-check` | `tokens` and `pairs` are comma-split with **no max length**, then forwarded to factory `BlacklistCheck` on LCD. Global governor only. Tests: sanitized 502 only. | Honest Swap/Trade tabs already amplify RE-02 (progress + `refetchInterval`). Behind NAT, one peer IP shares a bucket. ## Why the new implementation is needed OWASP API4 (unrestricted resource consumption): limits that count requests miss **work per request**. A busy 2k-block `/gt/events` window can return a multi-MB JSON payload and scan every reserve row. Progress + live `GetDiscount` at 60 RPS (distinct `trader` values) amplifies public LCD. Unbounded blacklist lists do the same to the factory query. Saturation looks like “DEX down” even though contracts still execute. Listing partners (GeckoTerminal) and the official dApp are **friendly** load. An unauthenticated public API must bound both. ## Constraints / guardrails 1. **Do not** trust `X-Forwarded-For` / `CF-Connecting-IP` (`#282`). Keep `PeerIpKeyExtractor`. 2. **Do not** disable or zero rate limits on non-loopback without `ALLOW_ZERO_RATE_LIMITS=1`. Prod clamp stays. 3. **Do not** change GeckoTerminal wire shape except to **truncate or 400** when over cap. Gem exclusion (`EXCLUDED_CW20` / L639-2) stays. Reserves stay **current snapshot**, not historical reconstruction. 4. **Do not** make progress poll gate submit or change `You Receive` (#484). Progress stays advisory. 5. **Do not** fail-open blacklist. Oversize lists → **400**, not a silent truncate that under-blocks. Factory query errors stay 502 sanitized (`LCD_UPSTREAM_GATEWAY_MSG`). 6. **Do not** add per-swap FoT math or new HTTP write APIs. 7. Keep CORS allowlist. Swagger may stay mounted. 8. Frontend backoff (RE-04 class) is **in scope only** as: stop sending `trader` on progress if discount is already on the in-flight quote, and/or back off after N consecutive progress failures. Do not rewrite all `refetchInterval`s. ## Relevant files | Path | Role | |------|------| | `indexer/src/api/gt.rs` | `/gt/events` queries + full `pair_reserves` scan | | `indexer/src/api/mod.rs` | Router split: `lcd_heavy_router` vs global | | `indexer/src/api/route_solve_progress.rs` | Progress handler | | `indexer/src/api/route_solver.rs` | `resolve_discount_bps` LCD `GetDiscount` | | `indexer/src/api/compliance.rs` | Unbounded `tokens` / `pairs` | | `indexer/src/config.rs` | `DEFAULT_RATE_LIMIT_*`, prod clamp | | `indexer/tests/security.rs` | 429 / 400 / 413 / 502 matrix | | `indexer/tests/api_gt.rs` | GT range tests | | `indexer/tests/api_route_solve.rs` | Progress lifecycle | | `frontend-dapp/src/hooks/useRouteSolveProgress.ts` | ~1 Hz poll + `trader` | | `frontend-dapp/src/utils/quoteDebounce.ts` | Sim refetch skip (leave as-is) | | `docs/indexer-invariants.md` | Document new caps | ## Recommended direction 1. **RE-01:** Add `MAX_GT_EVENT_ROWS` (recommend **5000** combined swap+liq). If the window would exceed, return **400** with a stable message (or return the first N and set a documented `truncated` field — pick one and test it). Load reserves **only for `pair_id`s present in the window** (`WHERE pair_id = ANY($ids)`), never the full table. 2. **RE-02:** Move `/api/v1/route/solve/progress` onto `lcd_heavy_router` (same 10 RPS as solve). Cache `discount_bps` per `(trader, sender)` with the existing 12s route-solve TTL (or reuse the progress registry entry) so a 1 Hz poll does **not** LCD every tick. Skip LCD when both trader fields are absent (already returns 0). 3. **RE-03:** Cap `tokens` at **16** and `pairs` at **8** (after trim/empty-drop). Oversize → **400**. Move the route onto `lcd_heavy_router`. 4. **Frontend (small):** `useRouteSolveProgress` — omit `trader` when the parent quote already resolved discount; abort/backoff after consecutive failures. Do not change quote math. 5. **Tests:** extend `security.rs` (and GT/progress files) per test plan. `make verify-issue-<this>` greps the caps + router membership + new tests. ## Acceptance criteria - [ ] `/gt/events` never returns more than the documented row cap; over-cap is 400 or explicit truncation (documented). - [ ] `/gt/events` does not `SELECT` all of `pair_reserves`. - [ ] Progress and blacklist-check are on the LCD-heavy governor (same layer as `/route/solve`). - [ ] Progress with `trader` set does not LCD-`GetDiscount` on every poll within the cache TTL. - [ ] Blacklist `tokens`/`pairs` over the cap → 400; at-cap still queries factory; factory error still sanitized 502. - [ ] Existing GT gem exclusion, solve cache, CORS, prod RPS clamp, and Peer-IP extractor unchanged. - [ ] `indexer/tests/security.rs` covers 429 on progress + blacklist-check (and solve `/best` if missing) and the new 400s. - [ ] `make verify-issue-<N>` exists and is wired in the Makefile / AGENTS.md indexer table. - [ ] `docs/indexer-invariants.md` records the new caps and which routes are LCD-heavy. ## Test plan — happy / bad path | Path | Expect | |------|--------| | `/gt/events` small window, few rows | 200, same JSON shape, gems still excluded | | `/gt/events` `to-from+1 > 2000` | 400 (existing) | | `/gt/events` window with >cap rows (seed) | 400 or truncated per chosen policy | | `/gt/events` many pairs in window | Reserves only for those pair_ids (unit or SQL assert) | | `GET /route/solve` + `/progress` without trader | 200; progress cheap; no LCD discount call | | Same with `trader` set, 2+ polls inside TTL | At most one LCD `GetDiscount` (mock LCD call count) | | Progress burst at test RPS | 429 on LCD-heavy | | `blacklist-check` 1 wallet + 2 tokens + 1 pair | 200 (LCD mock) | | `tokens` with 17 addresses | 400 | | `pairs` with 9 addresses | 400 | | Empty tokens/pairs omitted | Still 200 | | Factory LCD down | 502, body has no sqlx/stack | | Prod `RATE_LIMIT_*=0` | Still clamps to 60/10 (existing) | | dApp progress: quote in flight | Still advisory; submit not gated | | dApp progress: omit trader when discount known | No trader query param | ## Test plan — attack / abuse vectors | Vector | Expect | |--------|--------| | Repeat max-span `/gt/events` at 60 RPS | Row cap + optional GT governor; no unbounded JSON; no full-table reserve scan | | Progress flood with distinct `trader`s | LCD-heavy 10 RPS; discount cache prevents 1:1 LCD | | `blacklist-check?tokens=a,a,a…` (thousands) | 400 before LCD | | XFF spoof to bypass limiter | Still keyed on peer IP | | SQL / comma injection in tokens | Still bound LCD JSON; 400 if over cap; no SQLi | | Compression bomb / huge GT response | Cap makes payload bounded | | NAT-shared honest tabs | May 429 together (accepted); backoff reduces friendly load | ## Verification criteria 1. `cd indexer && cargo test --test security -- --test-threads=1` includes the new 400/429 cases. 2. `make verify-issue-<N>` green: greps `MAX_GT_EVENT_ROWS` (or equivalent), progress + blacklist-check on `lcd_heavy_router`, list caps, and the new test names. 3. Coolify/prod: after deploy, `curl` max-span `/gt/events` does not grow without bound; progress with a registered trader does not 1:1 LCD (log or mock). 4. Swap page still shows progress while a slow quote runs; Place/Swap still works. 5. Audit note: RE-01/02/03 closed in `INTERNAL_GROK46_1787908099` follow-up when this lands.

PlasticDigits commented

2026-08-28 09:24:17 +00:00

(Migrated from gitlab.com)

marked as related to #646

PlasticDigits commented

2026-08-28 09:24:18 +00:00

(Migrated from gitlab.com)

marked as related to #485

PlasticDigits commented

2026-08-28 09:48:46 +00:00

(Migrated from gitlab.com)

mentioned in commit 8ca32dc679

mentioned in commit 8ca32dc679712fe6459166e67e75a894d32d7132

PlasticDigits commented

2026-08-28 09:48:48 +00:00

(Migrated from gitlab.com)

mentioned in merge request !1193

PlasticDigits commented

2026-08-28 10:51:40 +00:00

(Migrated from gitlab.com)

mentioned in commit 3c4060aba0

mentioned in commit 3c4060aba03c2091565c13616cb39296fdf3bc2b

PlasticDigits (Migrated from gitlab.com) closed this issue

2026-08-28 10:51:41 +00:00

PlasticDigits commented

2026-08-28 11:07:01 +00:00

(Migrated from gitlab.com)

Merge sanity (!1193 landed on main)

make verify-issue-694 passed on the MR tip (5/5). All in-repo acceptance criteria are implemented: GT row cap 5000 → 400 (no truncation), no pair_reserves SELECT, progress + blacklist-check on lcd_heavy_router, 12s GetDiscount cache, blacklist tokens≤16 / pairs≤8 fail-closed 400, factory errors stay sanitized 502, frontend omits trader when discount_bps is known and backs off after progress failures.

Do not reopen this issue unless a merged invariant is wrong. Operator leftovers moved to a post-merge ticket.

Residual (non-blocking, already out of MR scope)

Coolify/prod: redeploy indexer, then confirm max-span /gt/events stays bounded and progress+trader is not 1:1 LCD.
Live Swap: progress stays advisory; Place / Swap still works.
Audit note close for RE-01 / RE-02 / RE-03 in INTERNAL_GROK46_1787908099 after prod evidence.
GT 400 tests live in api_gt.rs (not security.rs); functionally covered by the verify harness.
COUNT-then-SELECT on /gt/events is not one transaction (theoretical over-cap race).
Trade page does not wire useRouteSolveProgress (Swap only; documented as optional).

## Merge sanity (!1193 landed on main) `make verify-issue-694` passed on the MR tip (5/5). All in-repo acceptance criteria are implemented: GT row cap 5000 → **400** (no truncation), no `pair_reserves` SELECT, progress + blacklist-check on `lcd_heavy_router`, 12s `GetDiscount` cache, blacklist tokens≤16 / pairs≤8 fail-closed **400**, factory errors stay sanitized **502**, frontend omits `trader` when `discount_bps` is known and backs off after progress failures. Do **not** reopen this issue unless a merged invariant is wrong. Operator leftovers moved to a post-merge ticket. ### Residual (non-blocking, already out of MR scope) - Coolify/prod: redeploy indexer, then confirm max-span `/gt/events` stays bounded and progress+trader is not 1:1 LCD. - Live Swap: progress stays advisory; Place / Swap still works. - Audit note close for RE-01 / RE-02 / RE-03 in `INTERNAL_GROK46_1787908099` after prod evidence. - GT 400 tests live in `api_gt.rs` (not `security.rs`); functionally covered by the verify harness. - COUNT-then-SELECT on `/gt/events` is not one transaction (theoretical over-cap race). - Trade page does not wire `useRouteSolveProgress` (Swap only; documented as optional).

PlasticDigits commented

2026-08-28 11:07:53 +00:00

(Migrated from gitlab.com)

mentioned in issue #698