Arena page generates uncontrolled RPC retry storm under any RPC degradation (post-#216) #221

New issue

Closed

opened 2026-05-20 01:26:25 +00:00 by Brouie · 9 comments

Brouie commented 2026-05-20 01:26:25 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Found while running the #216 verification checklist. Arena (/timecurve/arena) is generating an uncontrolled retry storm against RPC endpoints. This appears to be a regression class separate from the polling-cadence work that #216 addresses — the retry layer never backs off.

Numbers from a fresh 60s observation

Page	Window	Total requests	Failed	Errors
/timecurve (Simple)	60s	5,906	small portion	4 console errors
/timecurve/arena	60s	126,893	majority	59,513 console errors
/timecurve/arena	10s	9,726	majority	3,531 console errors

Arena is firing ~2,100 requests per second under load. Tested in Incognito so wallet extensions are ruled out as the cause.

What I see in the network panel

Every visible request on Arena is one of:

• (failed) net::... — request killed before completing
• (pending) — stuck in Chrome's queue, never finishes

Both states across all RPC endpoints in the fallback list:

• rpc-megaeth-mainnet.globalstake.io
• gateway.liquify.com (api=MEGAETHM2ONGJ26QTI0N1N6)
• mainnet.megaeth.com/rpc (was healthy on direct curl from my laptop while this was happening — so the RPC is alive globally, the failures are browser-side)

Hypothesis

Chrome enforces a per-origin parallel connection cap. When the page generates many simultaneous outgoing requests faster than the network can drain them, Chrome starts dropping or queueing. The (pending) rows look like requests sitting in Chrome's queue. The (failed) rows look like requests Chrome killed for backpressure.

Once requests start failing, the existing retry/fallback path appears to fire more requests immediately rather than backing off. That's the retry loop.

I confirmed mainnet.megaeth.com/rpc is alive globally with:
curl -X POST https://mainnet.megaeth.com/rpc -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
returned 0xfae6cc — block 16,443,084

So the underlying transport works. The failures are happening at the browser layer under load.

Possibly related to the #216 known follow-up

The shipped notes say:

Known follow-up (non-blocking): Arena still polls a ~1s RPC multicall for core TimeCurve fields (mergedArenaTc) instead of reusing indexer sale-state like Simple — separate optimization if RPC load is still high.

Marked non-blocking, but this observation suggests it IS blocking — when RPC is even slightly degraded, the mergedArenaTc multicall doesn't drop in volume, the retry layer doesn't engage useRpcQueryHealthForRefetch (or it does but the multicall doesn't subscribe), and the page generates 100k+ requests/min.

Repro

1. Open https://yieldomega.com/timecurve/arena in a fresh Chrome tab (Incognito to rule out extensions)
2. F12 → Network tab → clear → Fetch/XHR filter
3. Wait 30-60 seconds
4. Observe request count climb past 50,000 with most rows showing (failed) or (pending)

Comparison data

Simple page (/timecurve) under the same conditions in the same session: 5,906 req/60s. So the storm is specific to Arena.

Severity

Launch-impact class. The Arena page is the WarBow leaderboard / steal / guard surface. If it's unusable when any RPC endpoint degrades (and one of them — globalstake — is the same one that was flagged in #216 as dead, still in position 1 of VITE_RPC_URL), most users hitting Arena during a partial RPC outage will see a broken page.

Related

• #216 architecture work (this finding is from the verification pass)
• The #216 verification doc says globalstake.io was removed from VITE_RPC_URL but on production /timecurve build-info panel I see VITE_RPC_URL = rpc-megaeth-mainnet.globalstake.io,megaeth.drpc.org,gateway.liquify.com/api/MEGAETHM2ONGJ26QTI0N1N6,mainnet.megaeth.com/rpc — globalstake is still position 1 (4 endpoints total now, vs the 2 from yesterday).

Asks of dev

@PlasticDigits — couple things:

1. Confirm whether the mergedArenaTc multicall is wired into useRpcQueryHealthForRefetch / shared backoff. If not, that's likely the retry-storm trigger.
2. The globalstake removal claim in the #216 verification doc — was that intended to land in a separate deploy, or did it not make it?
3. Should this be tracked as Arena-specific scope or rolled into the #216 known follow-up?

Found while running the #216 verification checklist. Arena (/timecurve/arena) is generating an uncontrolled retry storm against RPC endpoints. This appears to be a regression class separate from the polling-cadence work that #216 addresses — the retry layer never backs off. ## Numbers from a fresh 60s observation | Page | Window | Total requests | Failed | Errors | |---|---|---|---|---| | /timecurve (Simple) | 60s | 5,906 | small portion | 4 console errors | | /timecurve/arena | 60s | 126,893 | majority | 59,513 console errors | | /timecurve/arena | 10s | 9,726 | majority | 3,531 console errors | Arena is firing ~2,100 requests per second under load. Tested in Incognito so wallet extensions are ruled out as the cause. ## What I see in the network panel Every visible request on Arena is one of: - (failed) net::... — request killed before completing - (pending) — stuck in Chrome's queue, never finishes Both states across all RPC endpoints in the fallback list: - rpc-megaeth-mainnet.globalstake.io - gateway.liquify.com (api=MEGAETHM2ONGJ26QTI0N1N6) - mainnet.megaeth.com/rpc (was healthy on direct curl from my laptop while this was happening — so the RPC is alive globally, the failures are browser-side) ## Hypothesis Chrome enforces a per-origin parallel connection cap. When the page generates many simultaneous outgoing requests faster than the network can drain them, Chrome starts dropping or queueing. The (pending) rows look like requests sitting in Chrome's queue. The (failed) rows look like requests Chrome killed for backpressure. Once requests start failing, the existing retry/fallback path appears to fire more requests immediately rather than backing off. That's the retry loop. I confirmed mainnet.megaeth.com/rpc is alive globally with: curl -X POST https://mainnet.megaeth.com/rpc -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' returned 0xfae6cc — block 16,443,084 So the underlying transport works. The failures are happening at the browser layer under load. ## Possibly related to the #216 known follow-up The shipped notes say: > Known follow-up (non-blocking): Arena still polls a ~1s RPC multicall for core TimeCurve fields (mergedArenaTc) instead of reusing indexer sale-state like Simple — separate optimization if RPC load is still high. Marked non-blocking, but this observation suggests it IS blocking — when RPC is even slightly degraded, the mergedArenaTc multicall doesn't drop in volume, the retry layer doesn't engage useRpcQueryHealthForRefetch (or it does but the multicall doesn't subscribe), and the page generates 100k+ requests/min. ## Repro 1. Open https://yieldomega.com/timecurve/arena in a fresh Chrome tab (Incognito to rule out extensions) 2. F12 → Network tab → clear → Fetch/XHR filter 3. Wait 30-60 seconds 4. Observe request count climb past 50,000 with most rows showing (failed) or (pending) ## Comparison data Simple page (/timecurve) under the same conditions in the same session: 5,906 req/60s. So the storm is specific to Arena. ## Severity Launch-impact class. The Arena page is the WarBow leaderboard / steal / guard surface. If it's unusable when any RPC endpoint degrades (and one of them — globalstake — is the same one that was flagged in #216 as dead, still in position 1 of VITE_RPC_URL), most users hitting Arena during a partial RPC outage will see a broken page. ## Related - #216 architecture work (this finding is from the verification pass) - The #216 verification doc says globalstake.io was removed from VITE_RPC_URL but on production /timecurve build-info panel I see VITE_RPC_URL = rpc-megaeth-mainnet.globalstake.io,megaeth.drpc.org,gateway.liquify.com/api/MEGAETHM2ONGJ26QTI0N1N6,mainnet.megaeth.com/rpc — globalstake is still position 1 (4 endpoints total now, vs the 2 from yesterday). ## Asks of dev @PlasticDigits — couple things: 1. Confirm whether the mergedArenaTc multicall is wired into useRpcQueryHealthForRefetch / shared backoff. If not, that's likely the retry-storm trigger. 2. The globalstake removal claim in the #216 verification doc — was that intended to land in a separate deploy, or did it not make it? 3. Should this be tracked as Arena-specific scope or rolled into the #216 known follow-up?

Brouie commented 2026-05-20 01:28:19 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in issue #216

mentioned in issue #216

PlasticDigits commented 2026-05-20 04:06:47 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Keep here in 221

Keep here in 221

PlasticDigits commented 2026-05-20 04:10:47 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in commit 8f69b7b159

mentioned in commit 8f69b7b159c418b571f708240c8535308ace8226

PlasticDigits commented 2026-05-20 04:10:57 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Fix landed on main (8f69b7b)

Root cause: With VITE_INDEXER_URL set, Arena disabled the #216 sale-state RPC multicalls — so nothing was feeding reportRpcFetchAttempt, while six useWatchContractEvent log pollers + uncapped useBlock head reads kept hammering JSON-RPC. Under a degraded endpoint, viem fallback (4 URLs) × react-query retry amplified into ~100k+ req/min (per repro in this issue).

What changed:

1. LatestBlockProvider — useBlock reports into shared RPC health; MegaETH head poll uses useRpcBackoffPollInterval (5s → 15s → 30s).
2. useWarbowBpMovingEventWatch — off when indexer is configured (indexer WarBow ~1.5s polls cover live BP); off during RPC offline-tier backoff.
3. useRpcConnectivity / rpcBackedReadQueryOptions — reactive poll intervals + retry: 0 while offline-tier (Arena no-indexer multicalls).
4. Indexer-on Arena arenaWarbowPolicyRpc supplement multicall now reports RPC health.

Docs / invariants: INV-FRONTEND-221-ARENA-RPC · invariants §221 · timecurve-views §221 · manual QA #221 · guardrails + skills/play-timecurve-warbow/SKILL.md.

Automated: rpcConnectivity.test.ts, rpcReadQueryOptions.test.ts, tsc -b.

---

@brouie — please verify on production/staging when you have a moment:

• Open /timecurve/arena (Incognito), Network → Fetch/XHR, wait 60s — request count should stay well below Simple /timecurve (not 50k+).
• Block/throttle the first host in VITE_RPC_URL — within ~3s of failures, polls should slow (5s tier), not stay at ~1 Hz with endless (pending) / (failed) rows.
• With indexer set, no sustained eth_getLogs / filter traffic for WarBow events on Arena.
• After RPC recovers, WarBow rank / feed still updates within 1–2 indexer poll intervals (~3s baseline).
• Steal/guard/revenge submit still uses fresh RPC preflight (readFreshWarbowStealPreflight).

Leaving this issue open until your sign-off. Re: #216 follow-up item 2 (globalstake position in prod VITE_RPC_URL) — that is a deploy/env change, not covered by this frontend commit; happy to track separately if still desired.

## Fix landed on `main` (8f69b7b) **Root cause:** With `VITE_INDEXER_URL` set, Arena disabled the #216 sale-state RPC multicalls — so nothing was feeding `reportRpcFetchAttempt`, while **six** `useWatchContractEvent` log pollers + uncapped `useBlock` head reads kept hammering JSON-RPC. Under a degraded endpoint, viem **fallback** (4 URLs) × react-query **retry** amplified into ~100k+ req/min (per repro in this issue). **What changed:** 1. **`LatestBlockProvider`** — `useBlock` reports into shared RPC health; MegaETH head poll uses **`useRpcBackoffPollInterval`** (5s → 15s → 30s). 2. **`useWarbowBpMovingEventWatch`** — **off** when indexer is configured (indexer WarBow ~1.5s polls cover live BP); **off** during RPC offline-tier backoff. 3. **`useRpcConnectivity` / `rpcBackedReadQueryOptions`** — reactive poll intervals + `retry: 0` while offline-tier (Arena no-indexer multicalls). 4. **Indexer-on Arena** `arenaWarbowPolicyRpc` supplement multicall now reports RPC health. **Docs / invariants:** `INV-FRONTEND-221-ARENA-RPC` · [invariants §221](docs/testing/invariants-and-business-logic.md#arena-rpc-retry-storm-gitlab-221) · [timecurve-views §221](docs/frontend/timecurve-views.md#arena-rpc-backoff-gitlab-221) · [manual QA #221](docs/testing/manual-qa-checklists.md#manual-qa-issue-221) · guardrails + `skills/play-timecurve-warbow/SKILL.md`. **Automated:** `rpcConnectivity.test.ts`, `rpcReadQueryOptions.test.ts`, `tsc -b`. --- @brouie — please verify on production/staging when you have a moment: - [ ] Open `/timecurve/arena` (Incognito), Network → Fetch/XHR, wait **60s** — request count should stay **well below** Simple `/timecurve` (not 50k+). - [ ] Block/throttle the **first** host in `VITE_RPC_URL` — within ~3s of failures, polls should **slow** (5s tier), not stay at ~1 Hz with endless `(pending)` / `(failed)` rows. - [ ] With indexer set, **no** sustained `eth_getLogs` / filter traffic for WarBow events on Arena. - [ ] After RPC recovers, WarBow rank / feed still updates within **1–2** indexer poll intervals (~3s baseline). - [ ] Steal/guard/revenge **submit** still uses fresh RPC preflight (`readFreshWarbowStealPreflight`). Leaving this issue **open** until your sign-off. Re: #216 follow-up item 2 (globalstake position in prod `VITE_RPC_URL`) — that is a **deploy/env** change, not covered by this frontend commit; happy to track separately if still desired.

Brouie commented 2026-05-20 04:45:40 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Walked verification on production with the fix landed. Strong PASS on the core fix, two items needed additional testing I haven't done yet.

1. Arena 60s request count — PASS. 144 requests/60s on Arena in fresh tab observation. Compared to the 126,893/60s observation from when I filed this issue — that's a 99% reduction. Arena is also now LOWER than Simple's typical 5,906/60s, which is a nice inversion of the original problem.
2. RPC throttle + backoff — NOT WALKED. Would need DevTools network blocking on the first VITE_RPC_URL host to confirm the 5s tier kicks in within ~3s. Plan to walk this when I have a clean tab.
3. No sustained eth_getLogs / filter traffic for WarBow events — PASS. Zero eth_getLogs visible in the network panel during a 60s observation with indexer set. All visible traffic is indexer routes (guard-latest, steals-by-victim-day, podiums, chain-timer, sale-state) returning healthy 200s. The useWatchContractEvent off-when-indexer-configured change is working.
4. WarBow rank/feed update within ~3s after RPC recovery — NOT WALKED. Couldn't reproduce a clean failure-then-recovery cycle from a fresh tab. Depends on item 2.
5. Steal/guard/revenge submit fresh RPC preflight — PARTIAL. Triggered the Extend guard write flow on Arena. MetaMask popup fired correctly with right TimeCurve contract address (0x1b68b...2bf71), MegaETH network, sensible gas estimate. Network panel during popup was clean indexer-only traffic, no RPC preflight burst visible from Arena page itself in that window. Cancelled the tx in the wallet before submit, so I don't have a direct observation of realtime_sendRawTransaction on the wire or the post-receipt refetch. Plan to walk this on a future actual submit.

Re: globalstake position in prod VITE_RPC_URL — acknowledged separate concern, will not track here.

Bottom line: the storm is gone. The fix landed and the macro behavior is dramatically better. Items 2 + 4 + 5 still to walk, but the core regression is closed in my book.

@PlasticDigits

Walked verification on production with the fix landed. Strong PASS on the core fix, two items needed additional testing I haven't done yet. 1. Arena 60s request count — PASS. 144 requests/60s on Arena in fresh tab observation. Compared to the 126,893/60s observation from when I filed this issue — that's a 99% reduction. Arena is also now LOWER than Simple's typical 5,906/60s, which is a nice inversion of the original problem. 2. RPC throttle + backoff — NOT WALKED. Would need DevTools network blocking on the first VITE_RPC_URL host to confirm the 5s tier kicks in within \~3s. Plan to walk this when I have a clean tab. 3. No sustained eth_getLogs / filter traffic for WarBow events — PASS. Zero eth_getLogs visible in the network panel during a 60s observation with indexer set. All visible traffic is indexer routes (guard-latest, steals-by-victim-day, podiums, chain-timer, sale-state) returning healthy 200s. The useWatchContractEvent off-when-indexer-configured change is working. 4. WarBow rank/feed update within \~3s after RPC recovery — NOT WALKED. Couldn't reproduce a clean failure-then-recovery cycle from a fresh tab. Depends on item 2. 5. Steal/guard/revenge submit fresh RPC preflight — PARTIAL. Triggered the Extend guard write flow on Arena. MetaMask popup fired correctly with right TimeCurve contract address (0x1b68b...2bf71), MegaETH network, sensible gas estimate. Network panel during popup was clean indexer-only traffic, no RPC preflight burst visible from Arena page itself in that window. Cancelled the tx in the wallet before submit, so I don't have a direct observation of realtime_sendRawTransaction on the wire or the post-receipt refetch. Plan to walk this on a future actual submit. Re: globalstake position in prod VITE_RPC_URL — acknowledged separate concern, will not track here. Bottom line: the storm is gone. The fix landed and the macro behavior is dramatically better. Items 2 + 4 + 5 still to walk, but the core regression is closed in my book. @PlasticDigits

Brouie commented 2026-05-20 05:06:49 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Follow-up — also walked item 5 (steal submit fresh RPC preflight).

Switched to cl8y bot w2 (rank #4) to attempt a steal against a wallet in the 2×–10× band. Clicked STEAL on #1, MetaMask popup fired with warbowSteal selector 0x23df5215 on TimeCurve proxy. Cancelled before signing.

Item 5 of your verification checklist verified — fresh RPC preflight (readFreshWarbowStealPreflight) appears to be firing on click since the popup populates with correct contract + selector + victim address. Steal CTA path is intact post-fix.

Updated status of your checklist:

• Item 1 (request count below Simple): PASS (144 req/60s)
• Item 2 (throttle backoff): NOT WALKED
• Item 3 (no eth_getLogs): PASS
• Item 4 (recovery within ~3s): NOT WALKED
• Item 5 (steal/guard/revenge fresh RPC preflight): PASS

Follow-up — also walked item 5 (steal submit fresh RPC preflight). Switched to cl8y bot w2 (rank #4) to attempt a steal against a wallet in the 2×–10× band. Clicked STEAL on #1, MetaMask popup fired with warbowSteal selector 0x23df5215 on TimeCurve proxy. Cancelled before signing. Item 5 of your verification checklist verified — fresh RPC preflight (readFreshWarbowStealPreflight) appears to be firing on click since the popup populates with correct contract + selector + victim address. Steal CTA path is intact post-fix. Updated status of your checklist: - Item 1 (request count below Simple): PASS (144 req/60s) - Item 2 (throttle backoff): NOT WALKED - Item 3 (no eth_getLogs): PASS - Item 4 (recovery within ~3s): NOT WALKED - Item 5 (steal/guard/revenge fresh RPC preflight): PASS

Brouie commented 2026-05-20 05:21:58 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in issue #215

mentioned in issue #215

Brouie commented 2026-05-21 02:33:20 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Walked items 2 + 4 via DevTools URL block on indexer.yieldomega.com.

Item 2 (throttle backoff): PASS

Blocked indexer.yieldomega.com pattern. Page went quiet on indexer side as expected — no retry storm visible, no cascading failures, no pending/failed queue buildup. Sale clock kept counting client-side, on-screen WarBow data stayed at last-known-good state.

The offline-tier retry: 0 behavior is working — pre-fix this would have been a storm, post-fix the page just stops asking.

Item 4 (recovery within ~3s): PASS

Removed the block. Indexer requests resumed cleanly, normal cadence (~25 req/s), healthy 200 OK responses, response times 70-140ms range. No recovery delay, no burst pattern. WarBow data refreshed on the next poll interval as expected.

Full checklist now PASS from QA side:

• Item 1 (request count below Simple): PASS
• Item 2 (throttle backoff): PASS
• Item 3 (no eth_getLogs): PASS
• Item 4 (recovery within ~3s): PASS
• Item 5 (steal preflight): PASS

#221 verified clean.

Walked items 2 + 4 via DevTools URL block on indexer.yieldomega.com. **Item 2 (throttle backoff): PASS** Blocked `indexer.yieldomega.com` pattern. Page went quiet on indexer side as expected — no retry storm visible, no cascading failures, no pending/failed queue buildup. Sale clock kept counting client-side, on-screen WarBow data stayed at last-known-good state. The offline-tier `retry: 0` behavior is working — pre-fix this would have been a storm, post-fix the page just stops asking. **Item 4 (recovery within \~3s): PASS** Removed the block. Indexer requests resumed cleanly, normal cadence (\~25 req/s), healthy 200 OK responses, response times 70-140ms range. No recovery delay, no burst pattern. WarBow data refreshed on the next poll interval as expected. **Full checklist now PASS from QA side:** * Item 1 (request count below Simple): PASS * Item 2 (throttle backoff): PASS * Item 3 (no eth_getLogs): PASS * Item 4 (recovery within \~3s): PASS * Item 5 (steal preflight): PASS #221 verified clean.

PlasticDigits (Migrated from gitlab.com) closed this issue 2026-05-21 12:01:45 +00:00

PlasticDigits commented 2026-06-07 13:57:30 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in issue #301

mentioned in issue #301

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

cac-design-issue-544

renovate/configure

No results found.

Labels

Clear labels   

agent:fix_bugfix

  

agent:fix_conflicts

  

agent:fix_security

  

agent:gap_analysis

  

agent:implement

  

agent:open_issues

  

agents

  

agent:security_audit

  

agent:verify

  

area

timecurve

  

arena

  

arena-v2

  

audit

  

block:feature_review

  

block:security

  

bug

  

contracts

  

design

  

devops

  

docs

Documentation

  

documentation

  

enhancement

  

epic

Epic / umbrella issue

  

feature

  

frontend

  

indexer

  

low priority

Low priority, investigate other issues first

  

megaeth

  

operations

  

ops

Deploy and operations

  

performance

  

QA

  

qa checklist

  

referrals

  

removal

Legacy removal

  

security

  

smart-contracts

  

tech-debt

  

testing

  

time-arena

TimeArena v2 revamp

  

timecurve

  

visual-qa

No labels

agent:fix_bugfix

agent:fix_conflicts

agent:fix_security

agent:gap_analysis

agent:implement

agent:open_issues

agents

agent:security_audit

agent:verify

area

timecurve

arena

arena-v2

audit

block:feature_review

block:security

bug

contracts

design

devops

docs

documentation

enhancement

epic

feature

frontend

indexer

low priority

megaeth

operations

ops

performance

QA

qa checklist

referrals

removal

security

smart-contracts

tech-debt

testing

time-arena

timecurve

visual-qa

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

code/yieldomega#221

No description provided.