Benchmark indexer JSON-RPC load on localnet and reduce rate-limit risk

PlasticDigits commented

2026-06-10 09:21:04 +00:00

(Migrated from gitlab.com)

Problem

The indexer is a steady JSON-RPC consumer with two long-running loops that share RPC_URL:

Ingestion (indexer/src/ingestion.rs) — per indexed block: eth_getBlockByNumber + eth_getLogs; optional ancestor fetches on reorg; extra eth_calls during live podium snapshots (arena_podium_live, warbow_score) on qualifying events.
Chain-timer head poller (indexer/src/chain_timer.rs) — ~1 Hz when healthy (RPC_FAST_POLL_MS in rpc_poll_health.rs, aligned with frontend/src/lib/rpcConnectivity.ts): each cycle is eth_blockNumber, eth_getBlockByNumber, many sequential eth_calls (timer fields, 4× podium(), sale state, sale-head metadata), plus optional PodiumVaults pool balances.

We have backoff for failures and HTTP 429 (5s → 15s → 30s), but no measured baseline for calls/minute, burst peaks, or per-method breakdown on realistic localnet traffic. Without that, we cannot confidently size RPC quotas, compare mainnet vs Anvil, or prioritize optimizations that keep /arena reactive without hammering shared RPC infra.

Related: #168 (timeouts/liveness), #237 (deferred WSS lane), #301 (indexer-first display — UI should not duplicate head polling when indexer is healthy).

Goals

Quantify average and peak JSON-RPC requests per minute from a single indexer process on localnet (Anvil stack: scripts/start-local-anvil-stack.sh or scripts/start-qa-local-full-stack.sh).
Break down volume by method (eth_blockNumber, eth_getBlockByNumber, eth_getLogs, eth_call) and by subsystem (ingestion vs chain-timer vs ingestion-side live podium RPC).
Run scenarios that reflect product load:
- Idle — chain advancing, no arena txs
- Catch-up — indexer behind tip (restart with START_BLOCK lag or paused Anvil then fast-forward)
- Active arena — sustained buys / WarBow via existing verify scripts or QA swarm (scripts/verify-podium-live-anvil.sh, vault-funding verify, optional bot swarm)
Capture behavior under simulated RPC stress (artificial latency, intermittent 429) to validate RpcPollHealth backoff does not cause thundering herds.
Propose and rank mitigation strategies that preserve dapp reactivity (timer/podium/sale head freshness targets) while staying under provider rate limits.

Suggested approach

Instrumentation (deliverable 1)

Add lightweight RPC metrics in the indexer (feature-flagged or always-on at debug/info):

Counter + histogram per JSON-RPC method and caller tag (ingestion, chain_timer, podium_live, reorg, …)
Rolling window stats: 1m avg, 5m avg, p99 burst (max calls in any 10s window)
Optional: expose on GET /v1/status (schema bump) or structured log line every N seconds for CI/scripts

Prefer hooking at the shared transport layer (rpc_http.rs / provider wrapper) so all paths are counted consistently.

Benchmark harness (deliverable 2)

Script under scripts/ (e.g. scripts/benchmark-indexer-rpc-anvil.sh) that:

Starts native Postgres + Anvil + DeployDev indexer (reuse stack helpers)
Runs scenario matrix above for a fixed duration (e.g. 10–15 min each)
Parses metrics / logs into a markdown or JSON report committed to docs/indexer/ or pasted in this issue

Document exact env knobs: RPC_URL, INGESTION_ENABLED, INDEXER_RPC_REQUEST_TIMEOUT_SEC, Anvil block time.

Analysis & recommendations (deliverable 3)

Document findings and prioritized strategies, for example:

Strategy	Tradeoff
Batch/multicall for chain-timer `eth_call` fan-out	Fewer round-trips; slightly more complex decode
Adaptive poll interval when head unchanged (deadline/podium epochs stable)	Lower steady-state RPC; must not miss timer UX thresholds
Coalesce duplicate `eth_call` at same block tag within one poll	Cheap win in `poll_once`
Reduce ingestion-path `eth_call` during live podium snapshots (derive from decoded logs where possible)	Fewer calls on hot paths; verify `INV-INDEXER-PODIUM-PREDICT-LIVE`
Separate read RPC URL for head poller vs ingestion (operator config)	Isolates burst indexing from latency-sensitive polls
Ship #237 WSS/SSE for head hints	Best-effort mini-block awareness; RPC remains reconcile authority
Frontend: enforce indexer-first display (#301)	Cuts duplicate browser RPC, not indexer RPC

Include target SLOs proposal (e.g. max steady-state calls/min per instance, max lag for GET /v1/arena/timers freshness) for operator runbooks.

Acceptance criteria

Reproducible localnet benchmark script + documented runbook
Report with avg/peak RPC/min and per-method breakdown for ≥3 scenarios (idle, catch-up, active arena)
At least 3 concrete optimization proposals with estimated RPC reduction and reactivity impact
No regression to ingestion liveness (#168) or arena head API contracts (#254, #273)

Verification

# After instrumentation lands:
bash scripts/benchmark-indexer-rpc-anvil.sh

# Existing smoke (must stay green):
bash scripts/verify-podium-live-anvil.sh
cd indexer && cargo test

References

indexer/README.md — RPC resilience, chain-timer cadence
indexer/src/rpc_poll_health.rs, indexer/src/rpc_http.rs
indexer/src/ingestion.rs, indexer/src/chain_timer.rs, indexer/src/arena_podium_live.rs
docs/indexer/design.md — ingestion vs future WSS lane (#237)
docs/testing/qa-local-full-stack.md

## Problem The indexer is a steady JSON-RPC consumer with two long-running loops that share `RPC_URL`: 1. **Ingestion** (`indexer/src/ingestion.rs`) — per indexed block: `eth_getBlockByNumber` + `eth_getLogs`; optional ancestor fetches on reorg; extra `eth_call`s during live podium snapshots (`arena_podium_live`, `warbow_score`) on qualifying events. 2. **Chain-timer head poller** (`indexer/src/chain_timer.rs`) — **~1 Hz** when healthy (`RPC_FAST_POLL_MS` in `rpc_poll_health.rs`, aligned with `frontend/src/lib/rpcConnectivity.ts`): each cycle is `eth_blockNumber`, `eth_getBlockByNumber`, many sequential `eth_call`s (timer fields, 4× `podium()`, sale state, sale-head metadata), plus optional `PodiumVaults` pool balances. We have backoff for failures and HTTP **429** (5s → 15s → 30s), but **no measured baseline** for calls/minute, burst peaks, or per-method breakdown on realistic localnet traffic. Without that, we cannot confidently size RPC quotas, compare mainnet vs Anvil, or prioritize optimizations that keep `/arena` reactive without hammering shared RPC infra. Related: [#168](https://gitlab.com/PlasticDigits/yieldomega/-/issues/168) (timeouts/liveness), [#237](https://gitlab.com/PlasticDigits/yieldomega/-/issues/237) (deferred WSS lane), [#301](https://gitlab.com/PlasticDigits/yieldomega/-/issues/301) (indexer-first display — UI should not duplicate head polling when indexer is healthy). ## Goals - Quantify **average and peak JSON-RPC requests per minute** from a single indexer process on **localnet** (Anvil stack: `scripts/start-local-anvil-stack.sh` or `scripts/start-qa-local-full-stack.sh`). - Break down volume by **method** (`eth_blockNumber`, `eth_getBlockByNumber`, `eth_getLogs`, `eth_call`) and by **subsystem** (ingestion vs chain-timer vs ingestion-side live podium RPC). - Run scenarios that reflect product load: - **Idle** — chain advancing, no arena txs - **Catch-up** — indexer behind tip (restart with `START_BLOCK` lag or paused Anvil then fast-forward) - **Active arena** — sustained buys / WarBow via existing verify scripts or QA swarm (`scripts/verify-podium-live-anvil.sh`, vault-funding verify, optional bot swarm) - Capture behavior under **simulated RPC stress** (artificial latency, intermittent 429) to validate `RpcPollHealth` backoff does not cause thundering herds. - Propose and rank **mitigation strategies** that preserve dapp reactivity (timer/podium/sale head freshness targets) while staying under provider rate limits. ## Suggested approach ### Instrumentation (deliverable 1) Add lightweight RPC metrics in the indexer (feature-flagged or always-on at `debug`/`info`): - Counter + histogram per JSON-RPC method and caller tag (`ingestion`, `chain_timer`, `podium_live`, `reorg`, …) - Rolling window stats: **1m avg**, **5m avg**, **p99 burst** (max calls in any 10s window) - Optional: expose on `GET /v1/status` (schema bump) or structured log line every N seconds for CI/scripts Prefer hooking at the shared transport layer (`rpc_http.rs` / provider wrapper) so all paths are counted consistently. ### Benchmark harness (deliverable 2) Script under `scripts/` (e.g. `scripts/benchmark-indexer-rpc-anvil.sh`) that: 1. Starts native Postgres + Anvil + DeployDev indexer (reuse stack helpers) 2. Runs scenario matrix above for a fixed duration (e.g. 10–15 min each) 3. Parses metrics / logs into a markdown or JSON report committed to `docs/indexer/` or pasted in this issue Document exact env knobs: `RPC_URL`, `INGESTION_ENABLED`, `INDEXER_RPC_REQUEST_TIMEOUT_SEC`, Anvil block time. ### Analysis & recommendations (deliverable 3) Document findings and prioritized strategies, for example: | Strategy | Tradeoff | |----------|----------| | Batch/multicall for chain-timer `eth_call` fan-out | Fewer round-trips; slightly more complex decode | | Adaptive poll interval when head unchanged (deadline/podium epochs stable) | Lower steady-state RPC; must not miss timer UX thresholds | | Coalesce duplicate `eth_call` at same block tag within one poll | Cheap win in `poll_once` | | Reduce ingestion-path `eth_call` during live podium snapshots (derive from decoded logs where possible) | Fewer calls on hot paths; verify `INV-INDEXER-PODIUM-PREDICT-LIVE` | | Separate read RPC URL for head poller vs ingestion (operator config) | Isolates burst indexing from latency-sensitive polls | | Ship [#237](https://gitlab.com/PlasticDigits/yieldomega/-/issues/237) WSS/SSE for head hints | Best-effort mini-block awareness; RPC remains reconcile authority | | Frontend: enforce indexer-first display ([#301](https://gitlab.com/PlasticDigits/yieldomega/-/issues/301)) | Cuts duplicate browser RPC, not indexer RPC | Include **target SLOs** proposal (e.g. max steady-state calls/min per instance, max lag for `GET /v1/arena/timers` freshness) for operator runbooks. ## Acceptance criteria - [ ] Reproducible localnet benchmark script + documented runbook - [ ] Report with avg/peak RPC/min and per-method breakdown for ≥3 scenarios (idle, catch-up, active arena) - [ ] At least **3** concrete optimization proposals with estimated RPC reduction and reactivity impact - [ ] No regression to ingestion liveness ([#168](https://gitlab.com/PlasticDigits/yieldomega/-/issues/168)) or arena head API contracts ([#254](https://gitlab.com/PlasticDigits/yieldomega/-/issues/254), [#273](https://gitlab.com/PlasticDigits/yieldomega/-/issues/273)) ## Verification ```bash # After instrumentation lands: bash scripts/benchmark-indexer-rpc-anvil.sh # Existing smoke (must stay green): bash scripts/verify-podium-live-anvil.sh cd indexer && cargo test ``` ## References - `indexer/README.md` — RPC resilience, chain-timer cadence - `indexer/src/rpc_poll_health.rs`, `indexer/src/rpc_http.rs` - `indexer/src/ingestion.rs`, `indexer/src/chain_timer.rs`, `indexer/src/arena_podium_live.rs` - `docs/indexer/design.md` — ingestion vs future WSS lane ([#237](https://gitlab.com/PlasticDigits/yieldomega/-/issues/237)) - `docs/testing/qa-local-full-stack.md`

PlasticDigits commented

2026-06-10 11:16:24 +00:00

(Migrated from gitlab.com)

mentioned in commit 3277fda5b8

mentioned in commit 3277fda5b8a76e88afa7ef1f4c392d0a32154ba4

PlasticDigits commented

2026-06-10 11:16:47 +00:00

(Migrated from gitlab.com)

mentioned in merge request !454

PlasticDigits commented

2026-06-10 11:16:56 +00:00

(Migrated from gitlab.com)

Implementation (MR !454)

Branch local1/306-impl-rpc-benchmark adds indexer JSON-RPC instrumentation, localnet benchmark harness, and operator documentation for #306.

Deliverables

Instrumentation — indexer/src/rpc_metrics.rs + hooks in rpc_http.rs, ingestion, chain_timer, reorg, warbow_score, arena_podium_live. Structured logs every INDEXER_RPC_METRICS_LOG_SEC (default 60s).
Status API — GET /v1/status schema 2.11.0 includes rpc_metrics (1m/5m rates, peak_calls_10s, by_method, by_caller).
Benchmark harness — bash scripts/benchmark-indexer-rpc-anvil.sh (idle, catch-up, active arena). Sample artifacts: docs/indexer/benchmarks/rpc-benchmark-20260610T111415Z.{json,md}.
Docs / invariants — docs/indexer/rpc-load-benchmark.md, INV-INDEXER-306-* in invariants, design.md §306, play skill crosslink.

Sample baseline (Anvil, 30s scenarios)

Scenario	calls/min	peak/10s
idle	740	354
catch-up	820	434
active arena	1999	434

chain_timer eth_call fan-out dominates; ingestion spikes on catch-up and arena buys.

Verification checklist

bash scripts/verify-indexer-rpc-metrics.sh
BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh (optional longer run)
bash scripts/verify-podium-live-anvil.sh
cd indexer && cargo test
Confirm GET /v1/status → rpc_metrics.by_caller.chain_timer grows ~1 Hz when idle

Follow-ups (not in this MR)

Implement ranked mitigations (multicall batching, adaptive poll interval) as separate issues once baseline is reviewed on mainnet RPC.
#237 WSS lane remains deferred; #301 browser RPC reduction already shipped.

Issue stays open until MR !454 merges and checklist is signed off.

## Implementation (MR !454) Branch `local1/306-impl-rpc-benchmark` adds indexer JSON-RPC instrumentation, localnet benchmark harness, and operator documentation for #306. ### Deliverables 1. **Instrumentation** — `indexer/src/rpc_metrics.rs` + hooks in `rpc_http.rs`, `ingestion`, `chain_timer`, `reorg`, `warbow_score`, `arena_podium_live`. Structured logs every `INDEXER_RPC_METRICS_LOG_SEC` (default 60s). 2. **Status API** — `GET /v1/status` schema **2.11.0** includes `rpc_metrics` (1m/5m rates, `peak_calls_10s`, `by_method`, `by_caller`). 3. **Benchmark harness** — `bash scripts/benchmark-indexer-rpc-anvil.sh` (idle, catch-up, active arena). Sample artifacts: `docs/indexer/benchmarks/rpc-benchmark-20260610T111415Z.{json,md}`. 4. **Docs / invariants** — `docs/indexer/rpc-load-benchmark.md`, `INV-INDEXER-306-*` in invariants, design.md §306, play skill crosslink. ### Sample baseline (Anvil, 30s scenarios) | Scenario | calls/min | peak/10s | |----------|-----------|----------| | idle | 740 | 354 | | catch-up | 820 | 434 | | active arena | 1999 | 434 | `chain_timer` `eth_call` fan-out dominates; ingestion spikes on catch-up and arena buys. ### Verification checklist - [ ] `bash scripts/verify-indexer-rpc-metrics.sh` - [ ] `BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh` (optional longer run) - [ ] `bash scripts/verify-podium-live-anvil.sh` - [ ] `cd indexer && cargo test` - [ ] Confirm `GET /v1/status` → `rpc_metrics.by_caller.chain_timer` grows ~1 Hz when idle ### Follow-ups (not in this MR) - Implement ranked mitigations (multicall batching, adaptive poll interval) as separate issues once baseline is reviewed on mainnet RPC. - #237 WSS lane remains deferred; #301 browser RPC reduction already shipped. Issue stays open until MR !454 merges and checklist is signed off.

PlasticDigits commented

2026-06-10 11:28:04 +00:00

(Migrated from gitlab.com)

mentioned in commit 15b4923c38

mentioned in commit 15b4923c3865064e24b7cc360795e52449048eba

PlasticDigits commented

2026-06-10 11:29:06 +00:00

(Migrated from gitlab.com)

mentioned in commit 98259e2723

mentioned in commit 98259e272358fe1fff2e1fae11f89664f87a45f1

PlasticDigits commented

2026-06-10 11:30:57 +00:00

(Migrated from gitlab.com)

These need to be brought down to 50/10s in peak

PlasticDigits commented

2026-06-13 06:34:51 +00:00

(Migrated from gitlab.com)

mentioned in issue #307

PlasticDigits commented

2026-06-13 06:34:54 +00:00

(Migrated from gitlab.com)

mentioned in issue #308

PlasticDigits commented

2026-06-13 06:35:43 +00:00

(Migrated from gitlab.com)

Child issues opened (agent triage)

Parent baseline landed in MR !454 (instrumentation, GET /v1/status rpc_metrics, scripts/benchmark-indexer-rpc-anvil.sh, docs/indexer/rpc-load-benchmark.md). Follow-up directive: peak_calls_10s ≤ 50 (from ~354 idle).

Opened

Issue	Bundle	Scope
#307	Chain-timer RPC batching	Multicall3 aggregate for `poll_once` `eth_call` fan-out + coalesce duplicate calls at same block tag; target idle `peak_calls_10s` ≤ 50
#308	Adaptive poll cadence	Slow chain-timer polls when `read_block_number` / epochs stable; stay ≤ 3s `polled_at_ms` freshness near deadlines

Both issues reference the shared operator target from the 2026-06-10 comment and may be implemented in either order; combined they should close the ~7× gap to 50 calls / 10s.

Skipped — already done (parent / MR !454)

RPC instrumentation (rpc_metrics.rs, transport hooks)
GET /v1/status schema 2.11.0 rpc_metrics field
scripts/benchmark-indexer-rpc-anvil.sh + scripts/verify-indexer-rpc-metrics.sh
Baseline report + INV-INDEXER-306-* invariants
Analysis & ranked mitigation table in docs/indexer/rpc-load-benchmark.md

Skipped — rejected / deferred / shipped

Item	Status	Reason
#237 WSS/SSE head hints	Deferred	Explicit in MR !454 follow-up note
#301 indexer-first browser display	Shipped	Explicit in MR !454 follow-up note

Skipped — undecided (no approval in issue comments)

Derive ingestion-path live podium snapshots from decoded logs (vs block-tagged podium() RPC)
Separate read RPC_URL for head poller vs ingestion (operator isolation only)
Ingestion-side podium_live / warbow_score RPC reduction beyond current metrics
Mainnet RPC baseline re-run (operator task after mitigations land)
Bugbot MR !454 notes (chain-timer fallback overcount / eth_getBlockByNumber null handling) — not triaged as approved proposals on #306

Bundling rationale

#307 groups multicall batching + same-block coalesce (same poll_once refactor). #308 is separate poll-loop timing logic per MR !454 (“multicall batching, adaptive poll interval) as separate issues”).

## Child issues opened (agent triage) Parent baseline landed in **MR !454** (instrumentation, `GET /v1/status` `rpc_metrics`, `scripts/benchmark-indexer-rpc-anvil.sh`, `docs/indexer/rpc-load-benchmark.md`). Follow-up directive: **`peak_calls_10s` ≤ 50** (from ~354 idle). ### Opened | Issue | Bundle | Scope | |-------|--------|-------| | [#307](https://gitlab.com/PlasticDigits/yieldomega/-/issues/307) | Chain-timer RPC batching | Multicall3 aggregate for `poll_once` `eth_call` fan-out + coalesce duplicate calls at same block tag; target idle `peak_calls_10s` ≤ 50 | | [#308](https://gitlab.com/PlasticDigits/yieldomega/-/issues/308) | Adaptive poll cadence | Slow chain-timer polls when `read_block_number` / epochs stable; stay ≤ 3s `polled_at_ms` freshness near deadlines | Both issues reference the shared operator target from the 2026-06-10 comment and may be implemented in either order; combined they should close the ~7× gap to **50 calls / 10s**. ### Skipped — already done (parent / MR !454) - RPC instrumentation (`rpc_metrics.rs`, transport hooks) - `GET /v1/status` schema **2.11.0** `rpc_metrics` field - `scripts/benchmark-indexer-rpc-anvil.sh` + `scripts/verify-indexer-rpc-metrics.sh` - Baseline report + `INV-INDEXER-306-*` invariants - Analysis & ranked mitigation table in `docs/indexer/rpc-load-benchmark.md` ### Skipped — rejected / deferred / shipped | Item | Status | Reason | |------|--------|--------| | [#237](https://gitlab.com/PlasticDigits/yieldomega/-/issues/237) WSS/SSE head hints | **Deferred** | Explicit in MR !454 follow-up note | | [#301](https://gitlab.com/PlasticDigits/yieldomega/-/issues/301) indexer-first browser display | **Shipped** | Explicit in MR !454 follow-up note | ### Skipped — undecided (no approval in issue comments) - Derive ingestion-path live podium snapshots from decoded logs (vs block-tagged `podium()` RPC) - Separate read `RPC_URL` for head poller vs ingestion (operator isolation only) - Ingestion-side `podium_live` / `warbow_score` RPC reduction beyond current metrics - Mainnet RPC baseline re-run (operator task after mitigations land) - Bugbot MR !454 notes (chain-timer fallback overcount / `eth_getBlockByNumber` null handling) — not triaged as approved proposals on #306 ### Bundling rationale **#307** groups multicall batching + same-block coalesce (same `poll_once` refactor). **#308** is separate poll-loop timing logic per MR !454 (“multicall batching, adaptive poll interval) as **separate issues**”).

PlasticDigits commented

2026-06-13 07:02:54 +00:00

(Migrated from gitlab.com)