Indexer: enforce LCD-heavy route rate limits in production (#363) #873

2026-06-12T05:17:47Z

PlasticDigits commented

2026-06-12 05:17:47 +00:00

(Migrated from gitlab.com)

Summary

GitLab #363 (H7 follow-up from #361): production indexers can no longer accidentally disable the LCD-heavy per-IP rate limit.

RUN_MODE=prod: RATE_LIMIT_LCD_HEAVY_RPS=0 (and RATE_LIMIT_RPS=0) are clamped to defaults 10 / 60 with startup tracing::warn! messages.
Startup logs now print effective global and LCD-heavy RPS (main.rs).
Local deploy template (deploy-dex-local.sh) sets RATE_LIMIT_LCD_HEAVY_RPS=10 while keeping RATE_LIMIT_RPS=0 for Playwright/UI bursts.
Operator docs: route list, env knobs, prod vs QA vs local profiles, and 429 response shape in docs/operator-secrets.md, cross-linked from skills/AGENTS_INDEXER_API_LCD_SECURITY.md and docs/indexer-invariants.md.
Security test prod_lcd_heavy_rate_limit_enforced_when_env_zero covers the prod config path (clamp + 429 + Retry-After).

Acceptance checklist

Criterion	Verification	Result
Production cannot run with LCD-heavy limit disabled	`cd indexer && cargo test --lib config::tests::prod_forces_nonzero_rate_limits_when_zero`	PASS
Operator docs list LCD-heavy routes and env knobs	Review `docs/operator-secrets.md`, `skills/AGENTS_INDEXER_API_LCD_SECURITY.md`	PASS
Security integration test: 429 when exceeded (prod path)	`cd indexer && cargo test --test security prod_lcd_heavy -j 1 -- --test-threads=1`	PASS
Retail swap/trade under default limits	`cd indexer && cargo test --test api_route_solve` (debounced quote paths succeed); live burst manual on QA indexer	PASS (integration) / SKIP (live burst — indexer not running on agent VM)

Verification for third parties

# Unit + security
cd indexer && cargo test --lib config::tests -j 1 -- --test-threads=1
cd indexer && cargo test --test security -j 1 -- --test-threads=1

# Full integration (needs Postgres)
make test-indexer-integration

# Prod clamp smoke (expect warn in logs if env had 0)
RUN_MODE=prod LCD_URLS=https://lcd.example.com DATABASE_URL=postgres://... \
  FACTORY_ADDRESS=terra1... CORS_ORIGINS=https://app.example.com \
  RATE_LIMIT_LCD_HEAVY_RPS=0 cargo run -p cl8y-dex-indexer 2>&1 | head

# Manual abuse (QA indexer): burst route/solve/best → 429 after ~20 requests (2× burst)
for i in $(seq 1 30); do curl -sS -o /dev/null -w "%{http_code}\n" \
  "http://127.0.0.1:3001/api/v1/route/solve/best?token_in=...&token_out=...&amount_in=1000000"; done

Related: #361, #278, #355, #239

Note

Low Risk
Changes are config clamping, logging, local deploy env defaults, docs, and a security test—no new API routes or LCD query logic.

Overview
Production hardening (GitLab #363): With RUN_MODE=prod, setting RATE_LIMIT_RPS or RATE_LIMIT_LCD_HEAVY_RPS to 0 still clamps to 60 / 10, but now emits tracing::warn! on startup. Shared defaults live in DEFAULT_RATE_LIMIT_RPS and DEFAULT_RATE_LIMIT_LCD_HEAVY_RPS, and main.rs logs the effective global and LCD-heavy RPS at boot.

Local QA: deploy-dex-local.sh now writes RATE_LIMIT_LCD_HEAVY_RPS=10 alongside RATE_LIMIT_RPS=0 so UI/health bursts stay unlimited while limit-book, route/solve, and CG/CMC orderbook paths remain capped.

Tests: New security integration test prod_lcd_heavy_rate_limit_enforced_when_env_zero asserts prod config clamping and 429 with Retry-After on an LCD-heavy route when env had 0.

Docs: operator-secrets.md adds env knobs, prod/QA/local profiles, and 429 header behavior; integrators.md, indexer-invariants.md, .env.example, and the LCD security agent doc cross-link #363 and document dual-governor behavior.

^{Reviewed by Cursor Bugbot for commit f29d2de543. Bugbot is set up for automated code reviews on this repo. Configure here.}

## Summary GitLab #363 (H7 follow-up from #361): production indexers can no longer accidentally disable the LCD-heavy per-IP rate limit. - **`RUN_MODE=prod`:** `RATE_LIMIT_LCD_HEAVY_RPS=0` (and `RATE_LIMIT_RPS=0`) are clamped to defaults **10** / **60** with startup `tracing::warn!` messages. - **Startup logs** now print effective global and LCD-heavy RPS (`main.rs`). - **Local deploy template** (`deploy-dex-local.sh`) sets `RATE_LIMIT_LCD_HEAVY_RPS=10` while keeping `RATE_LIMIT_RPS=0` for Playwright/UI bursts. - **Operator docs:** route list, env knobs, prod vs QA vs local profiles, and **429** response shape in `docs/operator-secrets.md`, cross-linked from `skills/AGENTS_INDEXER_API_LCD_SECURITY.md` and `docs/indexer-invariants.md`. - **Security test** `prod_lcd_heavy_rate_limit_enforced_when_env_zero` covers the prod config path (clamp + 429 + `Retry-After`). ## Acceptance checklist | Criterion | Verification | Result | |-----------|--------------|--------| | Production cannot run with LCD-heavy limit disabled | `cd indexer && cargo test --lib config::tests::prod_forces_nonzero_rate_limits_when_zero` | PASS | | Operator docs list LCD-heavy routes and env knobs | Review `docs/operator-secrets.md`, `skills/AGENTS_INDEXER_API_LCD_SECURITY.md` | PASS | | Security integration test: 429 when exceeded (prod path) | `cd indexer && cargo test --test security prod_lcd_heavy -j 1 -- --test-threads=1` | PASS | | Retail swap/trade under default limits | `cd indexer && cargo test --test api_route_solve` (debounced quote paths succeed); live burst manual on QA indexer | PASS (integration) / SKIP (live burst — indexer not running on agent VM) | ## Verification for third parties ```bash # Unit + security cd indexer && cargo test --lib config::tests -j 1 -- --test-threads=1 cd indexer && cargo test --test security -j 1 -- --test-threads=1 # Full integration (needs Postgres) make test-indexer-integration # Prod clamp smoke (expect warn in logs if env had 0) RUN_MODE=prod LCD_URLS=https://lcd.example.com DATABASE_URL=postgres://... \ FACTORY_ADDRESS=terra1... CORS_ORIGINS=https://app.example.com \ RATE_LIMIT_LCD_HEAVY_RPS=0 cargo run -p cl8y-dex-indexer 2>&1 | head # Manual abuse (QA indexer): burst route/solve/best → 429 after ~20 requests (2× burst) for i in $(seq 1 30); do curl -sS -o /dev/null -w "%{http_code}\n" \ "http://127.0.0.1:3001/api/v1/route/solve/best?token_in=...&token_out=...&amount_in=1000000"; done ``` Related: #361, #278, #355, #239  --- > [!NOTE] > **Low Risk** > Changes are config clamping, logging, local deploy env defaults, docs, and a security test—no new API routes or LCD query logic. > > **Overview** > **Production hardening (GitLab #363):** With `RUN_MODE=prod`, setting `RATE_LIMIT_RPS` or `RATE_LIMIT_LCD_HEAVY_RPS` to `0` still clamps to **60** / **10**, but now emits **`tracing::warn!`** on startup. Shared defaults live in **`DEFAULT_RATE_LIMIT_RPS`** and **`DEFAULT_RATE_LIMIT_LCD_HEAVY_RPS`**, and **`main.rs`** logs the effective global and LCD-heavy RPS at boot. > > **Local QA:** `deploy-dex-local.sh` now writes **`RATE_LIMIT_LCD_HEAVY_RPS=10`** alongside **`RATE_LIMIT_RPS=0`** so UI/health bursts stay unlimited while limit-book, route/solve, and CG/CMC orderbook paths remain capped. > > **Tests:** New security integration test **`prod_lcd_heavy_rate_limit_enforced_when_env_zero`** asserts prod config clamping and **429** with **`Retry-After`** on an LCD-heavy route when env had `0`. > > **Docs:** **`operator-secrets.md`** adds env knobs, prod/QA/local profiles, and **429** header behavior; **`integrators.md`**, **`indexer-invariants.md`**, **`.env.example`**, and the LCD security agent doc cross-link **#363** and document dual-governor behavior. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f29d2de543bd09522429fc7e20cdd12de96c1dc9. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

ghost1 commented

2026-06-12 05:18:55 +00:00

(Migrated from gitlab.com)

changed the description

ghost1 commented

2026-06-12 05:19:07 +00:00

(Migrated from gitlab.com)

changed the description

PlasticDigits commented

2026-06-12 05:20:35 +00:00

(Migrated from gitlab.com)

Security review — MR !873

Commit reviewed: f29d2de543bd09522429fc7e20cdd12de96c1dc9
Scope: Production LCD-heavy rate-limit enforcement (#363): config.rs prod clamp + startup warnings, main.rs effective-limit logging, deploy-dex-local.sh local profile, docs, security integration test.

Outcome: FINDINGS: 0 medium+ — NONE

Summary

Reviewed added/modified paths and traced attacker-controlled inputs to sinks. This MR is defensive hardening only; no new HTTP routes, LCD query logic, auth layers, or dependencies.

Area	Result
*Prod clamp (`RATE_LIMIT_=0`)**	`RunMode::Prod` still forces 60/10 RPS via existing `apply_rate_limit_layer` path; MR adds constants, `tracing::warn!`, and integration test coverage. Not bypassable via request input.
Rate-limit keying	Unchanged: `PeerIpKeyExtractor` (socket peer IP); no trusted `X-Forwarded-For`.
LCD-heavy route coverage	Unchanged dual-governor wiring in `api/mod.rs`; no new unthrottled LCD fanout paths.
Local deploy (`RATE_LIMIT_RPS=0`, `RATE_LIMIT_LCD_HEAVY_RPS=10`)	Dev/Playwright profile only; `API_BIND=127.0.0.1`. Does not weaken prod when `RUN_MODE=prod`.
Logging	Startup logs effective RPS values only — no secrets or upstream URLs added.
Injection / SSRF / auth	No new sinks or boundary changes in this diff.

Inline threads: none (no medium+ findings).

Prior security-review comments on this MR: none found.

## Security review — MR !873 **Commit reviewed:** `f29d2de543bd09522429fc7e20cdd12de96c1dc9` **Scope:** Production LCD-heavy rate-limit enforcement (#363): `config.rs` prod clamp + startup warnings, `main.rs` effective-limit logging, `deploy-dex-local.sh` local profile, docs, security integration test. **Outcome:** `FINDINGS: 0` medium+ — **NONE** ### Summary Reviewed added/modified paths and traced attacker-controlled inputs to sinks. This MR is defensive hardening only; no new HTTP routes, LCD query logic, auth layers, or dependencies. | Area | Result | |------|--------| | **Prod clamp (`RATE_LIMIT_*=0`)** | `RunMode::Prod` still forces 60/10 RPS via existing `apply_rate_limit_layer` path; MR adds constants, `tracing::warn!`, and integration test coverage. Not bypassable via request input. | | **Rate-limit keying** | Unchanged: `PeerIpKeyExtractor` (socket peer IP); no trusted `X-Forwarded-For`. | | **LCD-heavy route coverage** | Unchanged dual-governor wiring in `api/mod.rs`; no new unthrottled LCD fanout paths. | | **Local deploy (`RATE_LIMIT_RPS=0`, `RATE_LIMIT_LCD_HEAVY_RPS=10`)** | Dev/Playwright profile only; `API_BIND=127.0.0.1`. Does not weaken prod when `RUN_MODE=prod`. | | **Logging** | Startup logs effective RPS values only — no secrets or upstream URLs added. | | **Injection / SSRF / auth** | No new sinks or boundary changes in this diff. | **Inline threads:** none (no medium+ findings). Prior security-review comments on this MR: none found.

Brouie commented

2026-06-12 07:28:33 +00:00

(Migrated from gitlab.com)

mentioned in issue #363