code/yieldomega
Watch
4
0
Fork
You've already forked yieldomega
 0
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity

Add tower-governor rate limits and trim public indexer status metrics #328

New issue
Closed
opened 2026-06-14 22:32:00 +00:00 by PlasticDigits · 5 comments
PlasticDigits commented 2026-06-14 22:32:00 +00:00 (Migrated from gitlab.com)

Parent

GitLab #325 — Security review (findings F-07, F-08; approved recommendation #4).

Explicit scope from parent comment: implement in-process tower-governor rate limiting. Do not rely on Cloudflare or nginx edge rate limits (rejected — monopoly concern).

Current codebase

  • indexer/src/api.rs: router exposes /healthz (plain ok), /v1/status (rich JSON), and many unauthenticated GET routes. clamp_limit allows limit 1–200 on paginated endpoints.
  • indexer/src/api_arena.rs: expensive list routes (/v1/arena/buys, /v1/arena/activity, /v1/arena/platform-usage, etc.) with no per-IP throttling.
  • indexer/src/main.rs: stacks tower_http CORS + trace; no rate-limit layer today.
  • indexer/Cargo.toml: tower-http with cors, trace only; no tower-governor dependency yet.
  • /v1/status returns rpc_metrics (call rates, method/caller breakdown), chain_pointer, max_indexed_block, ingestion_alive, last_indexed_at_ms — useful for ops reconnaissance (F-08).
  • indexer/README.md documents INV-INDEXER-306-STATUS-METRICS for rpc_metrics on /v1/status.
  • Integration tests hit both /healthz and /v1/status in indexer/tests/integration_stage2.rs.

Why needed

Public read API has no rate limiting. Attackers can script pagination (limit=200) across large tables to pressure DB CPU/memory and degrade availability. Detailed /v1/status aids timing/recon for ingestion lag and RPC load patterns. Parent #325 approved in-process rate limits and tighter status exposure.

Constraints / guardrails

  • No Cloudflare/nginx requirement — solution must work self-hosted (Render, bare VM).
  • Keep /healthz minimal for load balancers.
  • Do not break frontend frontend/src/lib/indexerApi.ts polling (/v1/arena/timers, /v1/status or health checks) under normal traffic — tune limits above legitimate UI refresh rates.
  • Preserve x-schema-version header behavior on JSON routes.
  • SQL remains parameterized; rate limiting is availability-only.
  • Document new env vars (e.g. INDEXER_RATE_LIMIT_PER_MIN) with sensible defaults.

Relevant files

Area Paths
API indexer/src/api.rs, indexer/src/api_arena.rs, indexer/src/main.rs
Metrics indexer/src/rpc_metrics.rs
Deps indexer/Cargo.toml
Tests indexer/tests/integration_stage2.rs, scripts/verify-indexer-rpc-metrics.sh
Frontend consumers frontend/src/lib/indexerApi.ts, arena hooks
Docs indexer/README.md, docs/indexer/design.md, docs/testing/invariants-and-business-logic.md

Recommended direction

Rate limiting (F-07)

  1. Add tower-governor (or tower_governor + governor crates compatible with axum 0.8).
  2. Apply per-peer (IP from ConnectInfo or X-Forwarded-For when trusted) limits on public GET routes; stricter bucket for heavy list endpoints if needed.
  3. Return 429 Too Many Requests with generic JSON body (no stack traces).
  4. Exempt /healthz or give it a generous bucket.
  5. Env-configurable limits; default safe for single-user dev on 127.0.0.1.

Status split (F-08)

  1. Keep GET /healthz as liveness only (ok + optional database_connected if cheap).
  2. Trim GET /v1/status for public callers or add GET /v1/status/ops (or similar) for detailed metrics behind env flag / optional bearer token / internal bind only.
  3. Minimum public /v1/status: schema_version, database_connected, max_indexed_block, ingestion_alive — drop or gate rpc_metrics and detailed chain_pointer for production unless INDEXER_EXPOSE_OPS_METRICS=1.
  4. Update INV-INDEXER-306 docs and frontend if it depends on removed fields (check indexerApi.ts).

Acceptance criteria

  • In-process rate limiting active on indexer HTTP router; burst abuse returns 429.
  • Normal arena UI against local stack remains functional (manual or Playwright smoke).
  • Public /v1/status no longer exposes full rpc_metrics by default in production mode (or gated).
  • /healthz suitable for LB probes unchanged or improved.
  • cd indexer && cargo clippy --all-targets -- -D warnings && cargo test passes.
  • Integration tests updated for new status shape and rate-limit behavior (where deterministic).

Test plan

Path Command
Indexer cd indexer && cargo clippy --all-targets -- -D warnings && cargo test
Integration cd indexer && cargo test --test integration_stage2
RPC metrics smoke bash scripts/verify-indexer-rpc-metrics.sh (update if status path changes)
Arena anvil bash scripts/verify-podium-live-anvil.sh or bash scripts/verify-wallet-profile-anvil.sh
Frontend cd frontend && npm run typecheck && npm test (if indexer client types change)
Load probe small script: rapid curl loop to list endpoint → expect 429 after threshold

Attack / abuse test plan

  • From one IP, hammer GET /v1/arena/buys?limit=200 — observe 429 and indexer process stability.
  • Confirm distributed slowloris not in scope but DB connection pool not exhausted by single abusive client.
  • curl /v1/status on production-config indexer: rpc_metrics absent or empty unless ops flag set.
  • Verify frontend arena page still loads timers/podiums with indexer-first display (INV-FRONTEND-301).

Verification criteria

  • MR documents chosen rate limits and env knobs.
  • Parent #325 recommendation #4 closed; tower-governor approach matches approved comment (no Cloudflare dependency).
## Parent [GitLab #325 — Security review](https://gitlab.com/PlasticDigits/yieldomega/-/issues/325) (findings **F-07**, **F-08**; approved recommendation **#4**). **Explicit scope from parent comment:** implement **in-process `tower-governor` rate limiting**. **Do not** rely on Cloudflare or nginx edge rate limits (rejected — monopoly concern). ## Current codebase - [`indexer/src/api.rs`](indexer/src/api.rs): router exposes `/healthz` (plain `ok`), `/v1/status` (rich JSON), and many unauthenticated `GET` routes. `clamp_limit` allows `limit` 1–200 on paginated endpoints. - [`indexer/src/api_arena.rs`](indexer/src/api_arena.rs): expensive list routes (`/v1/arena/buys`, `/v1/arena/activity`, `/v1/arena/platform-usage`, etc.) with no per-IP throttling. - [`indexer/src/main.rs`](indexer/src/main.rs): stacks `tower_http` CORS + trace; **no** rate-limit layer today. - [`indexer/Cargo.toml`](indexer/Cargo.toml): `tower-http` with `cors`, `trace` only; no `tower-governor` dependency yet. - `/v1/status` returns `rpc_metrics` (call rates, method/caller breakdown), `chain_pointer`, `max_indexed_block`, `ingestion_alive`, `last_indexed_at_ms` — useful for ops reconnaissance (F-08). - [`indexer/README.md`](indexer/README.md) documents `INV-INDEXER-306-STATUS-METRICS` for `rpc_metrics` on `/v1/status`. - Integration tests hit both `/healthz` and `/v1/status` in [`indexer/tests/integration_stage2.rs`](indexer/tests/integration_stage2.rs). ## Why needed Public read API has no rate limiting. Attackers can script pagination (`limit=200`) across large tables to pressure DB CPU/memory and degrade availability. Detailed `/v1/status` aids timing/recon for ingestion lag and RPC load patterns. Parent #325 approved in-process rate limits and tighter status exposure. ## Constraints / guardrails - **No Cloudflare/nginx requirement** — solution must work self-hosted (Render, bare VM). - Keep `/healthz` minimal for load balancers. - Do not break frontend [`frontend/src/lib/indexerApi.ts`](frontend/src/lib/indexerApi.ts) polling (`/v1/arena/timers`, `/v1/status` or health checks) under normal traffic — tune limits above legitimate UI refresh rates. - Preserve `x-schema-version` header behavior on JSON routes. - SQL remains parameterized; rate limiting is availability-only. - Document new env vars (e.g. `INDEXER_RATE_LIMIT_PER_MIN`) with sensible defaults. ## Relevant files | Area | Paths | |------|-------| | API | `indexer/src/api.rs`, `indexer/src/api_arena.rs`, `indexer/src/main.rs` | | Metrics | `indexer/src/rpc_metrics.rs` | | Deps | `indexer/Cargo.toml` | | Tests | `indexer/tests/integration_stage2.rs`, `scripts/verify-indexer-rpc-metrics.sh` | | Frontend consumers | `frontend/src/lib/indexerApi.ts`, arena hooks | | Docs | `indexer/README.md`, `docs/indexer/design.md`, `docs/testing/invariants-and-business-logic.md` | ## Recommended direction ### Rate limiting (F-07) 1. Add **`tower-governor`** (or `tower_governor` + `governor` crates compatible with axum 0.8). 2. Apply per-peer (IP from `ConnectInfo` or `X-Forwarded-For` when trusted) limits on public `GET` routes; stricter bucket for heavy list endpoints if needed. 3. Return `429 Too Many Requests` with generic JSON body (no stack traces). 4. Exempt `/healthz` or give it a generous bucket. 5. Env-configurable limits; default safe for single-user dev on `127.0.0.1`. ### Status split (F-08) 1. Keep **`GET /healthz`** as liveness only (`ok` + optional `database_connected` if cheap). 2. Trim **`GET /v1/status`** for public callers **or** add **`GET /v1/status/ops`** (or similar) for detailed metrics behind env flag / optional bearer token / internal bind only. 3. Minimum public `/v1/status`: `schema_version`, `database_connected`, `max_indexed_block`, `ingestion_alive` — drop or gate `rpc_metrics` and detailed `chain_pointer` for production unless `INDEXER_EXPOSE_OPS_METRICS=1`. 4. Update `INV-INDEXER-306` docs and frontend if it depends on removed fields (check `indexerApi.ts`). ## Acceptance criteria - [ ] In-process rate limiting active on indexer HTTP router; burst abuse returns 429. - [ ] Normal arena UI against local stack remains functional (manual or Playwright smoke). - [ ] Public `/v1/status` no longer exposes full `rpc_metrics` by default in production mode (or gated). - [ ] `/healthz` suitable for LB probes unchanged or improved. - [ ] `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` passes. - [ ] Integration tests updated for new status shape and rate-limit behavior (where deterministic). ## Test plan | Path | Command | |------|---------| | Indexer | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` | | Integration | `cd indexer && cargo test --test integration_stage2` | | RPC metrics smoke | `bash scripts/verify-indexer-rpc-metrics.sh` (update if status path changes) | | Arena anvil | `bash scripts/verify-podium-live-anvil.sh` or `bash scripts/verify-wallet-profile-anvil.sh` | | Frontend | `cd frontend && npm run typecheck && npm test` (if indexer client types change) | | Load probe | small script: rapid `curl` loop to list endpoint → expect 429 after threshold | ## Attack / abuse test plan - From one IP, hammer `GET /v1/arena/buys?limit=200` — observe 429 and indexer process stability. - Confirm distributed slowloris not in scope but DB connection pool not exhausted by single abusive client. - `curl /v1/status` on production-config indexer: `rpc_metrics` absent or empty unless ops flag set. - Verify frontend arena page still loads timers/podiums with indexer-first display (`INV-FRONTEND-301`). ## Verification criteria - MR documents chosen rate limits and env knobs. - Parent #325 recommendation #4 closed; tower-governor approach matches approved comment (no Cloudflare dependency).
PlasticDigits commented 2026-06-14 22:32:10 +00:00 (Migrated from gitlab.com)

mentioned in issue #325

mentioned in issue #325
PlasticDigits commented 2026-06-14 23:04:18 +00:00 (Migrated from gitlab.com)

mentioned in merge request !502

mentioned in merge request !502
PlasticDigits commented 2026-06-15 01:02:46 +00:00 (Migrated from gitlab.com)

mentioned in commit 35cecc13fe

mentioned in commit 35cecc13fe770e4bcc4d167f3e7bfa95b00885e0
PlasticDigits commented 2026-06-15 01:15:39 +00:00 (Migrated from gitlab.com)

Verification — GitLab #328 (PASS)

Verified on main @ 35cecc13 (merged !502). No repo changes required.

Acceptance criteria

Criterion Result How verified
In-process rate limiting; burst abuse returns 429 PASS bash scripts/verify-indexer-rate-limit.sh → ok=3 throttled=5 healthz=200; unit tests rate_limit::excess_requests_return_429
Normal arena UI / indexer polling functional PASS bash scripts/verify-podium-live-anvil.sh → integration_stage2 4/4; cd frontend && npm run typecheck && npm test → 676/676 (includes indexerApi 429 backoff + indexerFirstDisplay)
Public /v1/status omits rpc_metrics by default PASS bash scripts/verify-indexer-rpc-metrics.sh; unit test public_status_omits_rpc_metrics_by_default
/healthz suitable for LB probes PASS Rate-limit smoke confirms GET /healthz → 200 while /v1/arena/buys throttles to 429
cargo clippy + cargo test PASS cd indexer && cargo clippy --all-targets -- -D warnings && cargo test → 74 unit + 4 integration
Integration tests for status shape / rate limit PASS status_exposure_tests, rate_limit unit tests; cargo test --test integration_stage2 (included in full suite)

Test plan (issue body)

Command Result
cd indexer && cargo clippy --all-targets -- -D warnings && cargo test PASS
cd indexer && cargo test --test integration_stage2 PASS (4/4)
bash scripts/verify-indexer-rpc-metrics.sh PASS (schema=2.13.0, ops metrics warm)
bash scripts/verify-podium-live-anvil.sh PASS (live_rows=12)
cd frontend && npm run typecheck && npm test PASS
Rapid curl → 429 on list endpoint PASS (via verify-indexer-rate-limit.sh)

Attack / abuse checks

Check Result
Hammer GET /v1/arena/buys?limit=200 from one IP → 429, process stable PASS
curl /v1/status without ops flag → no rpc_metrics PASS
Frontend indexer-first display (INV-FRONTEND-301) PASS (unit tests)

Constraints / parent #325

  • tower_governor in-process limits — no Cloudflare/nginx dependency (PASS)
  • Env knobs documented: INDEXER_RATE_LIMIT_PER_MIN (default 600), INDEXER_RATE_LIMIT_BURST (120), INDEXER_TRUST_PROXY, INDEXER_EXPOSE_OPS_METRICS — indexer/README.md, indexer/.env.example, invariants INV-INDEXER-328-*
  • Ops metrics gated on GET /v1/status/ops when INDEXER_EXPOSE_OPS_METRICS=1 (PASS)
  • x-schema-version preserved on JSON routes (PASS, smoke scripts)

Infra

  • Native Postgres 127.0.0.1:5433 — bash scripts/verify-cloud-postgres.sh PASS
## Verification — GitLab #328 (PASS) Verified on `main` @ `35cecc13` (merged !502). No repo changes required. ### Acceptance criteria | Criterion | Result | How verified | |-----------|--------|--------------| | In-process rate limiting; burst abuse returns 429 | **PASS** | `bash scripts/verify-indexer-rate-limit.sh` → `ok=3 throttled=5 healthz=200`; unit tests `rate_limit::excess_requests_return_429` | | Normal arena UI / indexer polling functional | **PASS** | `bash scripts/verify-podium-live-anvil.sh` → integration_stage2 4/4; `cd frontend && npm run typecheck && npm test` → 676/676 (includes `indexerApi` 429 backoff + `indexerFirstDisplay`) | | Public `/v1/status` omits `rpc_metrics` by default | **PASS** | `bash scripts/verify-indexer-rpc-metrics.sh`; unit test `public_status_omits_rpc_metrics_by_default` | | `/healthz` suitable for LB probes | **PASS** | Rate-limit smoke confirms `GET /healthz` → 200 while `/v1/arena/buys` throttles to 429 | | `cargo clippy` + `cargo test` | **PASS** | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` → 74 unit + 4 integration | | Integration tests for status shape / rate limit | **PASS** | `status_exposure_tests`, `rate_limit` unit tests; `cargo test --test integration_stage2` (included in full suite) | ### Test plan (issue body) | Command | Result | |---------|--------| | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` | **PASS** | | `cd indexer && cargo test --test integration_stage2` | **PASS** (4/4) | | `bash scripts/verify-indexer-rpc-metrics.sh` | **PASS** (schema=2.13.0, ops metrics warm) | | `bash scripts/verify-podium-live-anvil.sh` | **PASS** (live_rows=12) | | `cd frontend && npm run typecheck && npm test` | **PASS** | | Rapid `curl` → 429 on list endpoint | **PASS** (via `verify-indexer-rate-limit.sh`) | ### Attack / abuse checks | Check | Result | |-------|--------| | Hammer `GET /v1/arena/buys?limit=200` from one IP → 429, process stable | **PASS** | | `curl /v1/status` without ops flag → no `rpc_metrics` | **PASS** | | Frontend indexer-first display (`INV-FRONTEND-301`) | **PASS** (unit tests) | ### Constraints / parent #325 - **tower_governor** in-process limits — no Cloudflare/nginx dependency (**PASS**) - Env knobs documented: `INDEXER_RATE_LIMIT_PER_MIN` (default 600), `INDEXER_RATE_LIMIT_BURST` (120), `INDEXER_TRUST_PROXY`, `INDEXER_EXPOSE_OPS_METRICS` — `indexer/README.md`, `indexer/.env.example`, invariants `INV-INDEXER-328-*` - Ops metrics gated on `GET /v1/status/ops` when `INDEXER_EXPOSE_OPS_METRICS=1` (**PASS**) - `x-schema-version` preserved on JSON routes (**PASS**, smoke scripts) ### Infra - Native Postgres `127.0.0.1:5433` — `bash scripts/verify-cloud-postgres.sh` **PASS**
PlasticDigits (Migrated from gitlab.com) closed this issue 2026-06-15 01:15:40 +00:00
PlasticDigits commented 2026-06-21 03:43:53 +00:00 (Migrated from gitlab.com)

mentioned in issue #341

mentioned in issue #341
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
cac-design-issue-544
renovate/configure
No results found.
Labels
Clear labels   
agent:fix_bugfix

   
agent:fix_conflicts

   
agent:fix_security

   
agent:gap_analysis

   
agent:implement

   
agent:open_issues

   
agents

   
agent:security_audit

   
agent:verify
  
area
timecurve

  
arena

   
arena-v2

   
audit

   
block:feature_review

   
block:security

   
bug

   
contracts

   
design

   
devops

   
docs
Documentation

  
documentation

   
enhancement

   
epic
Epic / umbrella issue

  
feature

   
frontend

   
indexer

   
low priority
Low priority, investigate other issues first

  
megaeth

   
operations

   
ops
Deploy and operations

  
performance

   
QA

   
qa checklist

   
referrals

   
removal
Legacy removal

  
security

   
smart-contracts

   
tech-debt

   
testing

   
time-arena
TimeArena v2 revamp

  
timecurve

   
visual-qa
No labels
agent:fix_bugfix
agent:fix_conflicts
agent:fix_security
agent:gap_analysis
agent:implement
agent:open_issues
agents
agent:security_audit
agent:verify
area
timecurve
arena
arena-v2
audit
block:feature_review
block:security
bug
contracts
design
devops
docs
documentation
enhancement
epic
feature
frontend
indexer
low priority
megaeth
operations
ops
performance
QA
qa checklist
referrals
removal
security
smart-contracts
tech-debt
testing
time-arena
timecurve
visual-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
code/yieldomega#328
No description provided.