feat(indexer): JSON-RPC load metrics and Anvil benchmark #306 #454

Merged

PlasticDigits merged 2 commits from local1/306-impl-rpc-benchmark into main 2026-06-10 11:29:05 +00:00

Conversation 15 Commits 2 Files changed 22 +1533 -130

PlasticDigits commented 2026-06-10 11:16:44 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Summary

• Add rpc_metrics module hooking ingestion, chain-timer, reorg, podium_live, and warbow_score JSON-RPC calls at the shared transport layer.
• Expose rolling rates (calls_per_min_1m, peak_calls_10s, per-method/caller breakdown) on GET /v1/status (schema 2.11.0).
• Add scripts/verify-indexer-rpc-metrics.sh and scripts/benchmark-indexer-rpc-anvil.sh (idle / catch-up / active arena scenarios).
• Document invariants (INV-INDEXER-306-*), operator runbook (docs/indexer/rpc-load-benchmark.md), sample benchmark artifacts, and play-skill crosslink.

Test plan

• cd indexer && cargo clippy --all-targets -- -D warnings
• cd indexer && cargo test
• bash scripts/verify-indexer-rpc-metrics.sh — PASS (schema 2.11.0, total_calls>0)
• BENCHMARK_SCENARIO_SEC=30 bash scripts/benchmark-indexer-rpc-anvil.sh — PASS (artifacts under docs/indexer/benchmarks/)
• bash scripts/verify-podium-live-anvil.sh (reviewer — no ingestion contract change expected)

Related: #306

---

Note

Medium Risk
Metrics plumbing touches every hot-path RPC call (mutex record per request) and broad refactors chain-timer/sale_state signatures, but logic and API contracts are unchanged aside from additive status fields.

Overview
Adds JSON-RPC load observability for GitLab #306: a new rpc_metrics module counts logical RPC calls (one per attempt, not per fallback URL) with rolling 1m/5m rates, 10s burst peaks, and method/caller breakdowns.

Instrumentation runs through rpc_*_instrumented helpers in the shared transport layer and is wired into ingestion, chain-timer head polling, reorg walks, and ingest-side podium_live / warbow_score snapshots. GET /v1/status bumps schema to 2.11.0 and returns an rpc_metrics object; main optionally logs aggregates on INDEXER_RPC_METRICS_LOG_SEC.

Operators get scripts/verify-indexer-rpc-metrics.sh, scripts/benchmark-indexer-rpc-anvil.sh (idle / catch-up / active arena), runbook docs/indexer/rpc-load-benchmark.md, sample artifacts under docs/indexer/benchmarks/, and INV-INDEXER-306-* invariants plus README/design cross-links.

^{Reviewed by Cursor Bugbot for commit 15b4923c38. Bugbot is set up for automated code reviews on this repo. Configure here.}

## Summary - Add `rpc_metrics` module hooking ingestion, chain-timer, reorg, podium_live, and warbow_score JSON-RPC calls at the shared transport layer. - Expose rolling rates (`calls_per_min_1m`, `peak_calls_10s`, per-method/caller breakdown) on `GET /v1/status` (schema **2.11.0**). - Add `scripts/verify-indexer-rpc-metrics.sh` and `scripts/benchmark-indexer-rpc-anvil.sh` (idle / catch-up / active arena scenarios). - Document invariants (`INV-INDEXER-306-*`), operator runbook (`docs/indexer/rpc-load-benchmark.md`), sample benchmark artifacts, and play-skill crosslink. ## Test plan - [x] `cd indexer && cargo clippy --all-targets -- -D warnings` - [x] `cd indexer && cargo test` - [x] `bash scripts/verify-indexer-rpc-metrics.sh` — PASS (schema 2.11.0, total_calls>0) - [x] `BENCHMARK_SCENARIO_SEC=30 bash scripts/benchmark-indexer-rpc-anvil.sh` — PASS (artifacts under `docs/indexer/benchmarks/`) - [ ] `bash scripts/verify-podium-live-anvil.sh` (reviewer — no ingestion contract change expected) Related: #306 <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Metrics plumbing touches every hot-path RPC call (mutex record per request) and broad refactors chain-timer/sale_state signatures, but logic and API contracts are unchanged aside from additive status fields. > > **Overview** > Adds **JSON-RPC load observability** for GitLab #306: a new `rpc_metrics` module counts logical RPC calls (one per attempt, not per fallback URL) with rolling **1m/5m rates**, **10s burst peaks**, and **method/caller** breakdowns. > > Instrumentation runs through **`rpc_*_instrumented`** helpers in the shared transport layer and is wired into **ingestion**, **chain-timer** head polling, **reorg** walks, and ingest-side **podium_live** / **warbow_score** snapshots. **`GET /v1/status`** bumps schema to **2.11.0** and returns an **`rpc_metrics`** object; **`main`** optionally logs aggregates on **`INDEXER_RPC_METRICS_LOG_SEC`**. > > Operators get **`scripts/verify-indexer-rpc-metrics.sh`**, **`scripts/benchmark-indexer-rpc-anvil.sh`** (idle / catch-up / active arena), runbook **`docs/indexer/rpc-load-benchmark.md`**, sample artifacts under **`docs/indexer/benchmarks/`**, and **`INV-INDEXER-306-*`** invariants plus README/design cross-links. > > ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 15b4923c3865064e24b7cc360795e52449048eba. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).} <!-- /CURSOR_SUMMARY -->

PlasticDigits commented 2026-06-10 11:16:56 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in issue #306

mentioned in issue #306

project_80561744_bot_5118c68a88bff95eacb6ab1c47c19cae commented 2026-06-10 11:16:59 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

changed the description

changed the description

project_80561744_bot_5118c68a88bff95eacb6ab1c47c19cae commented 2026-06-10 11:17:08 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

changed the description

changed the description

project_80561744_bot_5118c68a88bff95eacb6ab1c47c19cae commented 2026-06-10 11:17:51 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Stale Security Review comment

Cursor Bugbot has reviewed your changes and found 1 potential issue.
^{Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issue.}

^{Reviewed by Cursor Bugbot for commit 3277fda5b8. Configure here.}

<details> <summary>Stale Security Review comment</summary> <!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes and found 1 potential issue.<!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> ^{Bugbot Autofix is [ON](https://www.cursor.com/dashboard/bugbot). A cloud agent has been kicked off to fix the reported issue. <!-- BUGBOT_AUTOFIX_AGENT_LINK -->} <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3277fda5b8a76e88afa7ef1f4c392d0a32154ba4. Configure [here](https://www.cursor.com/dashboard/bugbot).} </details>

project_80561744_bot_5118c68a88bff95eacb6ab1c47c19cae commented 2026-06-10 11:17:52 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Chain timer fallback overcounts

Medium Severity

When comma-separated RPC_URL fallbacks are configured, a failed mid-poll_once on the first provider still records every RPC attempt, then poll_any_provider runs a full poll_once on the next provider and records again. One chain-timer cycle can inflate rpc_metrics versus actual provider load, contradicting the “one logical call, not per fallback retry” rule used elsewhere.

Additional Locations (1)

• indexer/src/chain_timer.rs#L302-L327

 

^{Reviewed by Cursor Bugbot for commit 3277fda5b8. Configure here.}

### Chain timer fallback overcounts **Medium Severity** <!-- DESCRIPTION START --> When comma-separated `RPC_URL` fallbacks are configured, a failed mid-`poll_once` on the first provider still records every RPC attempt, then `poll_any_provider` runs a full `poll_once` on the next provider and records again. One chain-timer cycle can inflate `rpc_metrics` versus actual provider load, contradicting the “one logical call, not per fallback retry” rule used elsewhere. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 5d183640-b85b-489f-8261-40a76e5724f5 --> <!-- LOCATIONS START indexer/src/chain_timer.rs#L246-L260 indexer/src/chain_timer.rs#L302-L327 LOCATIONS END --> <details> <summary>Additional Locations (1)</summary> - [`indexer/src/chain_timer.rs#L302-L327`](https://gitlab.com/PlasticDigits/yieldomega/-//blob/3277fda5b8a76e88afa7ef1f4c392d0a32154ba4/indexer/src/chain_timer.rs#L302-L327) </details> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmZmODE5NWM5LTNmNjYtNDlkZS1hYzBlLWQxMWJkYTZlNjI4NCIsImVuY3J5cHRpb25LZXkiOiJMZG1rd0dtZFVISWxlNlJRQ0FxX3JVdjdtb3Z0TmxWM3R2WDlmUVJsYlJnIiwiYnJhbmNoIjoibG9jYWwxLzMwNi1pbXBsLXJwYy1iZW5jaG1hcmsiLCJyZXBvT3duZXIiOiJQbGFzdGljRGlnaXRzIiwicmVwb05hbWUiOiJ5aWVsZG9tZWdhIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmZmODE5NWM5LTNmNjYtNDlkZS1hYzBlLWQxMWJkYTZlNjI4NCIsImVuY3J5cHRpb25LZXkiOiJMZG1rd0dtZFVISWxlNlJRQ0FxX3JVdjdtb3Z0TmxWM3R2WDlmUVJsYlJnIiwiYnJhbmNoIjoibG9jYWwxLzMwNi1pbXBsLXJwYy1iZW5jaG1hcmsiLCJyZXBvT3duZXIiOiJQbGFzdGljRGlnaXRzIiwicmVwb05hbWUiOiJ5aWVsZG9tZWdhIiwicHJOdW1iZXIiOjg0LCJjb21taXRTaGEiOiIzMjc3ZmRhNWI4YTc2ZTg4YWZhN2VmMWY0YzM5MmQwYTMyMTU0YmE0IiwicHJvdmlkZXIiOiJnaXRsYWIiLCJob3N0bmFtZSI6ImdpdGxhYi5jb20ifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3277fda5b8a76e88afa7ef1f4c392d0a32154ba4. Configure [here](https://www.cursor.com/dashboard/bugbot).}

PlasticDigits commented 2026-06-10 11:28:05 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

changed this line in version 2 of the diff

changed this line in [version 2 of the diff](/PlasticDigits/yieldomega/-/merge_requests/84/diffs?diff_id=1844294982&start_sha=3277fda5b8a76e88afa7ef1f4c392d0a32154ba4#3440af45bad0179d2131631d0d2bfbfd3512be1b_255_253)

PlasticDigits commented 2026-06-10 11:28:06 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

added 1 commit

• 15b4923c - fix(indexer): count chain-timer RPC fallbacks as one logical call #306

Compare with previous version

added 1 commit <ul><li>15b4923c - fix(indexer): count chain-timer RPC fallbacks as one logical call #306</li></ul> [Compare with previous version](/PlasticDigits/yieldomega/-/merge_requests/84/diffs?diff_id=1844294982&start_sha=3277fda5b8a76e88afa7ef1f4c392d0a32154ba4)

project_80561744_bot_5118c68a88bff95eacb6ab1c47c19cae commented 2026-06-10 11:28:24 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

changed the description

changed the description

project_80561744_bot_5118c68a88bff95eacb6ab1c47c19cae commented 2026-06-10 11:28:47 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

resolved all threads

resolved all threads

PlasticDigits commented 2026-06-10 11:29:06 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in commit 98259e2723

mentioned in commit 98259e272358fe1fff2e1fae11f89664f87a45f1

PlasticDigits (Migrated from gitlab.com) merged commit 98259e2723 into main 2026-06-10 11:29:06 +00:00

project_80561744_bot_5118c68a88bff95eacb6ab1c47c19cae commented 2026-06-10 11:29:20 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is ON, but it could not run because the branch was deleted or merged before autofix could start.}

^{Reviewed by Cursor Bugbot for commit 15b4923c38. Configure here.}

<!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes and found 1 potential issue. <!-- BUGBOT_FIX_ALL --> <a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aTplNTVhMTI4My1mN2ZmLTQzNGMtODljOC0zMDIxMWM2NTU3MjMiLCJlbmNyeXB0aW9uS2V5IjoiODRvTE44Z29pamd0b0FTTEwyV1RkOEtZNEVTU0dnUHhFWU5MZU1HLTA3OCIsImJyYW5jaCI6ImxvY2FsMS8zMDYtaW1wbC1ycGMtYmVuY2htYXJrIiwicmVwb093bmVyIjoiUGxhc3RpY0RpZ2l0cyIsInJlcG9OYW1lIjoieWllbGRvbWVnYSJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <!-- /BUGBOT_FIX_ALL --> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> ^{Bugbot Autofix is ON, but it could not run because the branch was deleted or merged before autofix could start.} <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 15b4923c3865064e24b7cc360795e52449048eba. Configure [here](https://www.cursor.com/dashboard/bugbot).}

project_80561744_bot_5118c68a88bff95eacb6ab1c47c19cae commented 2026-06-10 11:29:21 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Chain timer block fetch fallback

Medium Severity

poll_once loads the head block via rpc_first_ok_instrumented for eth_getBlockByNumber. That helper treats Ok(None) as success and stops, so a primary RPC that returns JSON-RPC null never tries comma-separated fallbacks. Removing the old per-provider poll loop removed the previous recovery path, so multi-URL setups can fail head polls when only a secondary node has the block.

 

^{Reviewed by Cursor Bugbot for commit 15b4923c38. Configure here.}

### Chain timer block fetch fallback **Medium Severity** <!-- DESCRIPTION START --> `poll_once` loads the head block via `rpc_first_ok_instrumented` for `eth_getBlockByNumber`. That helper treats `Ok(None)` as success and stops, so a primary RPC that returns JSON-RPC null never tries comma-separated fallbacks. Removing the old per-provider poll loop removed the previous recovery path, so multi-URL setups can fail head polls when only a secondary node has the block. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 6afe9960-3597-4cdb-a321-483043915326 --> <!-- LOCATIONS START indexer/src/chain_timer.rs#L316-L325 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmU4ODk1YjA0LTYzMTQtNGVmZC04MWE1LTg4YjNjY2M4NDE4MSIsImVuY3J5cHRpb25LZXkiOiJyM1h1WHlJTDNySUQtOTBJVnJ0d2ZtRzJkVktaX1lmN0kzUHNDLXRJbTBNIiwiYnJhbmNoIjoibG9jYWwxLzMwNi1pbXBsLXJwYy1iZW5jaG1hcmsiLCJyZXBvT3duZXIiOiJQbGFzdGljRGlnaXRzIiwicmVwb05hbWUiOiJ5aWVsZG9tZWdhIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmU4ODk1YjA0LTYzMTQtNGVmZC04MWE1LTg4YjNjY2M4NDE4MSIsImVuY3J5cHRpb25LZXkiOiJyM1h1WHlJTDNySUQtOTBJVnJ0d2ZtRzJkVktaX1lmN0kzUHNDLXRJbTBNIiwiYnJhbmNoIjoibG9jYWwxLzMwNi1pbXBsLXJwYy1iZW5jaG1hcmsiLCJyZXBvT3duZXIiOiJQbGFzdGljRGlnaXRzIiwicmVwb05hbWUiOiJ5aWVsZG9tZWdhIiwicHJOdW1iZXIiOjg0LCJjb21taXRTaGEiOiIxNWI0OTIzYzM4NjUwNjRlMjRiN2NjMzYwNzk1ZTUyNDQ5MDQ4ZWJhIiwicHJvdmlkZXIiOiJnaXRsYWIiLCJob3N0bmFtZSI6ImdpdGxhYi5jb20ifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 15b4923c3865064e24b7cc360795e52449048eba. Configure [here](https://www.cursor.com/dashboard/bugbot).}

PlasticDigits commented 2026-06-13 06:34:51 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in issue #307

mentioned in issue #307

PlasticDigits commented 2026-06-13 06:34:54 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in issue #308

mentioned in issue #308

PlasticDigits commented 2026-06-14 05:39:45 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in merge request !488

mentioned in merge request !488

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

agent:fix_bugfix

  

agent:fix_conflicts

  

agent:fix_security

  

agent:gap_analysis

  

agent:implement

  

agent:open_issues

  

agents

  

agent:security_audit

  

agent:verify

  

area

timecurve

  

arena

  

arena-v2

  

audit

  

block:feature_review

  

block:security

  

bug

  

contracts

  

design

  

devops

  

docs

Documentation

  

documentation

  

enhancement

  

epic

Epic / umbrella issue

  

feature

  

frontend

  

indexer

  

low priority

Low priority, investigate other issues first

  

megaeth

  

operations

  

ops

Deploy and operations

  

performance

  

QA

  

qa checklist

  

referrals

  

removal

Legacy removal

  

security

  

smart-contracts

  

tech-debt

  

testing

  

time-arena

TimeArena v2 revamp

  

timecurve

  

visual-qa

No labels

agent:fix_bugfix

agent:fix_conflicts

agent:fix_security

agent:gap_analysis

agent:implement

agent:open_issues

agents

agent:security_audit

agent:verify

area

timecurve

arena

arena-v2

audit

block:feature_review

block:security

bug

contracts

design

devops

docs

documentation

enhancement

epic

feature

frontend

indexer

low priority

megaeth

operations

ops

performance

QA

qa checklist

referrals

removal

security

smart-contracts

tech-debt

testing

time-arena

timecurve

visual-qa

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

code/yieldomega!454

No description provided.