Indexer: enforce LCD-heavy route rate limits in production #363

New issue

2026-06-12T05:05:44Z

PlasticDigits commented

2026-06-12 05:05:44 +00:00

(Migrated from gitlab.com)

Parent

Gap analysis follow-up from GitLab #361 — register item H7 (LCD amplification on heavy routes).

Current codebase

LCD-heavy routes are grouped and get a separate per-IP rate limit via rate_limit_lcd_heavy_rps (indexer/src/api/mod.rs:380-411).
Affected routes: limit-book (deep/shallow/head/insert-hints), route/solve, route/solve/best, cg/orderbook, cmc/orderbook/{market_pair}.
Config default: 10 RPS (indexer/src/config.rs); env RATE_LIMIT_LCD_HEAVY_RPS. Setting 0 disables the extra layer (see indexer/tests/security.rs).
#278 fixed cg/cmc bypass; global rate_limit_rps still applies separately.

Why needed

These endpoints fan out multiple LCD smart queries per request. A single client (or integrator) can amplify load on the indexer and upstream LCD. Rate limiting is implemented but optional when set to 0 — production deployments need enforced defaults and operator guidance.

Constraints / guardrails

Do not break legitimate retail dApp traffic (swap/trade debounced quotes) at expected concurrency.
Keep sanitized 502 bodies on LCD failure (#239).
No Prometheus /metrics scope (rejected in #361 parent triage).
Document integrator quotas; do not require MR-hosted CI gate.

Relevant files

indexer/src/api/mod.rs
indexer/src/config.rs
indexer/tests/security.rs
indexer/.env.example (if present) / docs/ operator config docs
scripts/qa/start-qa.sh (QA indexer env)

Recommended direction

Production default: refuse RATE_LIMIT_LCD_HEAVY_RPS=0 in RunMode::Production (or warn + clamp to minimum), while keeping 0 for local/integration tests.
Operator docs: document recommended RPS for QA vs mainnet, route list, and 429 response shape.
Optional: per-route cost weighting or tighter limit on limit-book depth params if abuse persists.
Wire QA start-qa / deploy env template with a sane non-zero value.

Acceptance criteria

Production indexer config cannot accidentally run with LCD-heavy limit disabled (or logs fatal/warn on startup).
Operator docs list all LCD-heavy routes and env knobs.
Security integration test proves 429 when limit exceeded (existing tests extended for production config path).
Retail swap/trade flows remain functional under default limits in manual QA.

Test plan

Path	Command / action
Security integration	`cd indexer && cargo test --test security`
Full integration	`make test-indexer-integration`
Manual	Burst `curl` against `/api/v1/route/solve/best` on QA indexer; expect 429 after threshold

Attack / abuse test plan

Single-IP burst on route/solve/best with deep hop list → 429, no LCD stampede.
Distributed low-rate requests still succeed within global + LCD-heavy budgets.
Verify cg/cmc orderbook endpoints share the same limiter (regression for #278).

Verification criteria

QA indexer starts with non-zero RATE_LIMIT_LCD_HEAVY_RPS in documented env template.
make test-indexer-integration green; security tests cover limiter enabled path.

## Parent Gap analysis follow-up from [GitLab #361](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/361) — register item **H7** (LCD amplification on heavy routes). ## Current codebase - LCD-heavy routes are grouped and get a **separate** per-IP rate limit via `rate_limit_lcd_heavy_rps` (`indexer/src/api/mod.rs:380-411`). - Affected routes: `limit-book` (deep/shallow/head/insert-hints), `route/solve`, `route/solve/best`, `cg/orderbook`, `cmc/orderbook/{market_pair}`. - Config default: **10 RPS** (`indexer/src/config.rs`); env `RATE_LIMIT_LCD_HEAVY_RPS`. Setting `0` disables the extra layer (see `indexer/tests/security.rs`). - [#278](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/278) fixed cg/cmc bypass; global `rate_limit_rps` still applies separately. ## Why needed These endpoints fan out multiple LCD `smart` queries per request. A single client (or integrator) can amplify load on the indexer **and** upstream LCD. Rate limiting is implemented but **optional** when set to 0 — production deployments need enforced defaults and operator guidance. ## Constraints / guardrails - Do not break legitimate retail dApp traffic (swap/trade debounced quotes) at expected concurrency. - Keep sanitized 502 bodies on LCD failure ([#239](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/239)). - No Prometheus `/metrics` scope (rejected in #361 parent triage). - Document integrator quotas; do not require MR-hosted CI gate. ## Relevant files - `indexer/src/api/mod.rs` - `indexer/src/config.rs` - `indexer/tests/security.rs` - `indexer/.env.example` (if present) / `docs/` operator config docs - `scripts/qa/start-qa.sh` (QA indexer env) ## Recommended direction 1. **Production default**: refuse `RATE_LIMIT_LCD_HEAVY_RPS=0` in `RunMode::Production` (or warn + clamp to minimum), while keeping `0` for local/integration tests. 2. **Operator docs**: document recommended RPS for QA vs mainnet, route list, and 429 response shape. 3. **Optional**: per-route cost weighting or tighter limit on `limit-book` depth params if abuse persists. 4. Wire QA `start-qa` / deploy env template with a sane non-zero value. ## Acceptance criteria - [ ] Production indexer config cannot accidentally run with LCD-heavy limit disabled (or logs fatal/warn on startup). - [ ] Operator docs list all LCD-heavy routes and env knobs. - [ ] Security integration test proves 429 when limit exceeded (existing tests extended for production config path). - [ ] Retail swap/trade flows remain functional under default limits in manual QA. ## Test plan | Path | Command / action | |------|------------------| | Security integration | `cd indexer && cargo test --test security` | | Full integration | `make test-indexer-integration` | | Manual | Burst `curl` against `/api/v1/route/solve/best` on QA indexer; expect 429 after threshold | ## Attack / abuse test plan - Single-IP burst on `route/solve/best` with deep hop list → 429, no LCD stampede. - Distributed low-rate requests still succeed within global + LCD-heavy budgets. - Verify cg/cmc orderbook endpoints share the same limiter (regression for #278). ## Verification criteria - QA indexer starts with non-zero `RATE_LIMIT_LCD_HEAVY_RPS` in documented env template. - `make test-indexer-integration` green; security tests cover limiter enabled path.

PlasticDigits commented

2026-06-12 05:06:04 +00:00

(Migrated from gitlab.com)

mentioned in issue #361

PlasticDigits commented

2026-06-12 05:17:29 +00:00

(Migrated from gitlab.com)

mentioned in commit f29d2de543

mentioned in commit f29d2de543bd09522429fc7e20cdd12de96c1dc9

PlasticDigits commented

2026-06-12 05:18:32 +00:00

(Migrated from gitlab.com)

mentioned in merge request !873

Brouie commented

2026-06-12 07:28:32 +00:00

(Migrated from gitlab.com)

Verified !873 at the source + test layer (this is my #355 rate-limiter territory).

AC (prod can't run with the LCD-heavy limit disabled): config.rs — in RunMode::Prod, RATE_LIMIT_LCD_HEAVY_RPS=0 logs a warn ("not allowed; using 10 RPS (#363)") and clamps to DEFAULT_RATE_LIMIT_LCD_HEAVY_RPS (10). Same guard added for the general RATE_LIMIT_RPS. So a prod deploy can't accidentally ship with the extra layer off.
Doesn't regress QA/dev: the clamp is gated on RunMode::Prod only — Dev/default still honors 0, so the QA indexer (heavy-RPS 0 for browser testing) is unaffected. RunMode::from_env maps only "prod"/"production" to Prod.
Test coverage: security.rs prod_lcd_heavy_rate_limit_enforced_when_env_zero sets RUN_MODE=prod + RATE_LIMIT_LCD_HEAVY_RPS=0 and asserts the config clamps to the default — direct AC coverage.
Operator guidance ACs covered in the diff: docs/integrators.md, docs/operator-secrets.md, indexer-invariants.md, .env.example, deploy-dex-local.sh wiring.

The underlying lcd-heavy limiter mechanism was already proven live on this stack (#355 — true RPS restored after the per_second period fix). Note: I did not run the cargo security test here (indexer rebuild, disk at 93%); the clamp logic + the dedicated test are sound, and it'll exercise on the clean-host/CI run once the rust-1.83 CI break (#370) is fixed. Good from my side at the source layer.

Verified !873 at the source + test layer (this is my #355 rate-limiter territory). - AC (prod can't run with the LCD-heavy limit disabled): config.rs — in RunMode::Prod, RATE_LIMIT_LCD_HEAVY_RPS=0 logs a warn ("not allowed; using 10 RPS (#363)") and clamps to DEFAULT_RATE_LIMIT_LCD_HEAVY_RPS (10). Same guard added for the general RATE_LIMIT_RPS. So a prod deploy can't accidentally ship with the extra layer off. - Doesn't regress QA/dev: the clamp is gated on RunMode::Prod only — Dev/default still honors 0, so the QA indexer (heavy-RPS 0 for browser testing) is unaffected. RunMode::from_env maps only "prod"/"production" to Prod. - Test coverage: security.rs prod_lcd_heavy_rate_limit_enforced_when_env_zero sets RUN_MODE=prod + RATE_LIMIT_LCD_HEAVY_RPS=0 and asserts the config clamps to the default — direct AC coverage. - Operator guidance ACs covered in the diff: docs/integrators.md, docs/operator-secrets.md, indexer-invariants.md, .env.example, deploy-dex-local.sh wiring. The underlying lcd-heavy limiter mechanism was already proven live on this stack (#355 — true RPS restored after the per_second period fix). Note: I did not run the cargo security test here (indexer rebuild, disk at 93%); the clamp logic + the dedicated test are sound, and it'll exercise on the clean-host/CI run once the rust-1.83 CI break (#370) is fixed. Good from my side at the source layer.

PlasticDigits commented

2026-06-12 07:55:42 +00:00

(Migrated from gitlab.com)

mentioned in commit 823daf68ee

mentioned in commit 823daf68ee948028a72e6dc7f20355b00963805d

PlasticDigits commented

2026-06-12 08:14:36 +00:00

(Migrated from gitlab.com)

mentioned in commit 3a757c7714

mentioned in commit 3a757c7714a6703b0aafbeb5654f9ea31b066d28

PlasticDigits commented

2026-06-12 08:15:00 +00:00

(Migrated from gitlab.com)

mentioned in merge request !882

ghost1 commented

2026-06-12 08:20:41 +00:00

(Migrated from gitlab.com)

mentioned in commit 9de66003d1

mentioned in commit 9de66003d142af2ee7fd5a5fb83866c4030674b7

PlasticDigits commented

2026-06-12 08:34:20 +00:00

(Migrated from gitlab.com)

mentioned in commit 4d33747d7f

mentioned in commit 4d33747d7f1ccf24239f6e349b2ff394da096ca1

PlasticDigits commented

2026-06-12 08:34:25 +00:00

(Migrated from gitlab.com)

mentioned in merge request !874

PlasticDigits commented

2026-06-12 08:34:45 +00:00

(Migrated from gitlab.com)

mentioned in commit d72c80f2c8

mentioned in commit d72c80f2c8d5421eebea5d094671ebf0e9c58a8a

PlasticDigits commented

2026-06-12 08:35:47 +00:00

(Migrated from gitlab.com)

mentioned in merge request !883

PlasticDigits commented

2026-06-12 11:02:02 +00:00

(Migrated from gitlab.com)

mentioned in commit 2f9c79676a

mentioned in commit 2f9c79676a8d499680fc6a28a96bfd02c930972e

PlasticDigits commented

2026-06-12 11:04:57 +00:00

(Migrated from gitlab.com)

mentioned in commit ee7828f038

mentioned in commit ee7828f0388ad031bd4e11dac3670500d7f85da4

PlasticDigits commented

2026-06-12 11:28:38 +00:00

(Migrated from gitlab.com)

Verification — GitLab #363 (LCD-heavy route rate limits in production)

Verified on main @ 2f9c796 (includes merged !873). No repo changes required.

Acceptance criteria

Item	Result	How verified
Production cannot run with LCD-heavy limit disabled	PASS	`indexer/src/config.rs`: `RUN_MODE=prod` clamps `RATE_LIMIT_LCD_HEAVY_RPS=0` → 10 with `tracing::warn!`; same for global `RATE_LIMIT_RPS` → 60. Unit: `cargo test --lib config::tests::prod_forces_nonzero_rate_limits_when_zero`. Startup log in `main.rs` prints effective RPS.
Operator docs list LCD-heavy routes and env knobs	PASS	`docs/operator-secrets.md` (profiles, 429 shape, route cross-link), `skills/AGENTS_INDEXER_API_LCD_SECURITY.md`, `docs/indexer-invariants.md`, `indexer/.env.example`, `docs/route-solver.md`.
Security integration: 429 when limit exceeded (prod path)	PASS	`cargo test --test security prod_lcd_heavy -j 1 -- --test-threads=1` (2 tests); full `cargo test --test security` → 25/25.
Retail swap/trade functional under default limits	PASS	Live indexer (`RATE_LIMIT_LCD_HEAVY_RPS=10`, `RATE_LIMIT_RPS=0` from deploy template): 15 retail-paced `route/solve/best` curls @ 500 ms → all 200. `http://127.0.0.1:5173/swap` and `/trade/EMBER-CORAL` → 200.

Test plan / attack matrix

Check	Result	Command / output
Security suite	PASS	`cd indexer && cargo test --test security -j 1 -- --test-threads=1` → 25 passed
Full integration	SKIP (unrelated)	`make test-indexer-integration` fails on `api::db_orderbook_sim::tests::zero_reserve_mirror_returns_no_output` (#369); not #363 scope
Manual burst `route/solve/best`	PASS	80 rapid curls → 429 after ~21 OK; headers: `retry-after`, `x-ratelimit-limit: 20`, `x-ratelimit-remaining: 0`
CG/CMC shared limiter (#278)	PASS	80 rapid curls on `/cg/orderbook?...` → 429
QA env template non-zero heavy RPS	PASS	`scripts/deploy-dex-local.sh` writes `RATE_LIMIT_LCD_HEAVY_RPS=10`; confirmed in `indexer/.env` after `make setup-cloud-localterra`

Notes

Dev/QA may keep RATE_LIMIT_RPS=0 for Playwright/UI bursts; LCD-heavy stays at 10 unless both knobs are 0 in dev-only mode.
Prod clamp is gated on RUN_MODE=prod / production only.

Follow-ups

Optional from issue: per-route cost weighting on deep limit-book if abuse persists.
Unrelated: fix zero_reserve_mirror_returns_no_output (#369) so make test-indexer-integration is green on main.

## Verification — GitLab #363 (LCD-heavy route rate limits in production) Verified on `main` @ `2f9c796` (includes merged !873). No repo changes required. ### Acceptance criteria | Item | Result | How verified | |------|--------|--------------| | Production cannot run with LCD-heavy limit disabled | **PASS** | `indexer/src/config.rs`: `RUN_MODE=prod` clamps `RATE_LIMIT_LCD_HEAVY_RPS=0` → **10** with `tracing::warn!`; same for global `RATE_LIMIT_RPS` → **60**. Unit: `cargo test --lib config::tests::prod_forces_nonzero_rate_limits_when_zero`. Startup log in `main.rs` prints effective RPS. | | Operator docs list LCD-heavy routes and env knobs | **PASS** | `docs/operator-secrets.md` (profiles, 429 shape, route cross-link), `skills/AGENTS_INDEXER_API_LCD_SECURITY.md`, `docs/indexer-invariants.md`, `indexer/.env.example`, `docs/route-solver.md`. | | Security integration: 429 when limit exceeded (prod path) | **PASS** | `cargo test --test security prod_lcd_heavy -j 1 -- --test-threads=1` (2 tests); full `cargo test --test security` → **25/25**. | | Retail swap/trade functional under default limits | **PASS** | Live indexer (`RATE_LIMIT_LCD_HEAVY_RPS=10`, `RATE_LIMIT_RPS=0` from deploy template): 15 retail-paced `route/solve/best` curls @ 500 ms → all **200**. `http://127.0.0.1:5173/swap` and `/trade/EMBER-CORAL` → **200**. | ### Test plan / attack matrix | Check | Result | Command / output | |-------|--------|------------------| | Security suite | **PASS** | `cd indexer && cargo test --test security -j 1 -- --test-threads=1` → 25 passed | | Full integration | **SKIP** (unrelated) | `make test-indexer-integration` fails on `api::db_orderbook_sim::tests::zero_reserve_mirror_returns_no_output` (#369); not #363 scope | | Manual burst `route/solve/best` | **PASS** | 80 rapid curls → **429** after ~21 OK; headers: `retry-after`, `x-ratelimit-limit: 20`, `x-ratelimit-remaining: 0` | | CG/CMC shared limiter (#278) | **PASS** | 80 rapid curls on `/cg/orderbook?...` → **429** | | QA env template non-zero heavy RPS | **PASS** | `scripts/deploy-dex-local.sh` writes `RATE_LIMIT_LCD_HEAVY_RPS=10`; confirmed in `indexer/.env` after `make setup-cloud-localterra` | ### Notes - Dev/QA may keep `RATE_LIMIT_RPS=0` for Playwright/UI bursts; LCD-heavy stays at **10** unless both knobs are `0` in dev-only mode. - Prod clamp is gated on `RUN_MODE=prod` / `production` only. ### Follow-ups - Optional from issue: per-route cost weighting on deep `limit-book` if abuse persists. - Unrelated: fix `zero_reserve_mirror_returns_no_output` (#369) so `make test-indexer-integration` is green on `main`.

PlasticDigits (Migrated from gitlab.com) closed this issue

2026-06-12 11:28:39 +00:00

PlasticDigits commented

2026-06-13 03:17:11 +00:00

(Migrated from gitlab.com)

mentioned in merge request !891