code/yieldomega
Watch
4
0
Fork
You've already forked yieldomega
 0
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity

Adaptive chain-timer poll interval when head state is unchanged #308

New issue
Closed
opened 2026-06-13 06:34:53 +00:00 by PlasticDigits · 7 comments
PlasticDigits commented 2026-06-13 06:34:53 +00:00 (Migrated from gitlab.com)

Parent

#306 — Benchmark indexer JSON-RPC load on localnet and reduce rate-limit risk

Baseline (MR !454): steady-state ~1 Hz chain-timer polls via RPC_FAST_POLL_MS even when read_block_number and timer epochs are unchanged. Parent comment: bring peak_calls_10s down to ≤ 50 (from ~354 idle).

Current codebase

  • indexer/src/chain_timer.rs — run_poll_loop always sleeps health.backoff_sleep() after every poll_once; no head-change short-circuit.
  • indexer/src/rpc_poll_health.rs — adaptive spacing only on failure / HTTP 429 (5s → 15s → 30s). Healthy cadence fixed at 1000 ms, aligned with frontend/src/lib/rpcConnectivity.ts.
  • indexer/src/rpc_metrics.rs — exposes peak_calls_10s, calls_per_min_1m, by_caller.chain_timer.
  • GET /v1/arena/timers — served from in-memory TimecurveHeadSnapshot cache; UX expects ~1s freshness near deadline boundaries.
  • docs/indexer/rpc-load-benchmark.md — ranks adaptive poll interval when head unchanged and timer epochs stable (est. 30–50% steady-state reduction).

Why needed

Approved follow-up from #306 / MR !454: implement adaptive poll interval as a separate mitigation. When Anvil (or mainnet) produces no new blocks and onchain timer/podium epochs are stable, full poll_once every second wastes RPC quota and contributes directly to burst peaks. This complements Multicall batching (sibling issue).

Constraints / guardrails

  • Never skip polls entirely — minimum cadence must guarantee GET /v1/arena/timers polled_at_ms within 3s when RPC healthy (per rpc-load-benchmark.md proposed SLO).
  • Accelerate to RPC_FAST_POLL_MS (1s) when:
    • read_block_number advances,
    • last_buy_epoch or any podium_epochs[cat] changes,
    • wall-clock approaches any podium_deadlines_sec or global deadline_sec within a configurable margin (e.g. ≤ 30s),
    • prior poll failed or returned HTTP 429 (RpcPollHealth already handles failure tiers).
  • Idle slow cadence should be env-configurable (e.g. CHAIN_TIMER_IDLE_POLL_MS, default TBD by benchmark — likely 2–5s) without breaking frontend indexer-first assumptions.
  • Do not change ingestion loop cadence or INGESTION_ENABLED behavior.
  • #237 WSS hints remain deferred — this issue is RPC poll spacing only.

Relevant files

Path Role
indexer/src/chain_timer.rs run_poll_loop, poll_once, snapshot comparison
indexer/src/rpc_poll_health.rs Failure backoff; extend or sibling for success-path spacing
indexer/src/rpc_metrics.rs Measure improvement
frontend/src/lib/rpcConnectivity.ts Reference cadence constants (read-only alignment)
scripts/benchmark-indexer-rpc-anvil.sh Idle scenario measurement
docs/indexer/rpc-load-benchmark.md SLO + mitigation table

Recommended direction

  1. After successful poll_once, compare new TimecurveHeadSnapshot to cached:
    • read_block_number, last_buy_epoch, podium_epochs, deadline_sec, podium_deadlines_sec.
  2. If unchanged and not in deadline proximity window, sleep CHAIN_TIMER_IDLE_POLL_MS instead of RPC_FAST_POLL_MS.
  3. Optionally still issue lightweight eth_blockNumber on idle cycles (1 call) to detect new heads without full poll_once — only if it reduces total RPC vs full poll; benchmark both variants.
  4. Log mode transitions at debug (fast vs idle) for operator forensics.
  5. Document env knobs in indexer/README.md and rpc-load-benchmark.md.

Acceptance criteria

  • Chain-timer uses fast 1s polls when head or epoch fields change or near deadline; slower idle polls otherwise.
  • bash scripts/benchmark-indexer-rpc-anvil.sh idle scenario: calls_per_min_1m drops ≥ 30% vs rpc-benchmark-20260610T111415Z.json baseline (or combined with Multicall issue, peak_calls_10s ≤ 50).
  • GET /v1/arena/timers polled_at_ms ≤ 3s stale under healthy RPC during idle Anvil run.
  • Near-miss test: with artificial wall-clock mock or short deadline on Anvil, poller returns to 1s cadence within one slow interval of epoch change.
  • RpcPollHealth failure / 429 backoff unchanged.
  • INV-INDEXER-306-NO-REGRESSION passes.

Test plan

export PATH="$HOME/.foundry/bin:$PATH"
source /usr/local/cargo/env 2>/dev/null || true

cd indexer && cargo clippy --all-targets -- -D warnings && cargo test
# New unit tests: snapshot diff → fast vs idle sleep selection; deadline proximity
bash scripts/verify-indexer-rpc-metrics.sh
bash scripts/verify-podium-live-anvil.sh
BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh
# Manual: curl -s http://127.0.0.1:3103/v1/arena/timers | jq '.polled_at_ms' twice 1s apart — should update in fast mode

Attack / abuse test plan

  • Stuck head — if RPC returns constant eth_blockNumber while chain advances on another node, ingestion lag alerts (#168) still fire; idle mode must not suppress ingestion.
  • Clock skew — deadline proximity uses wall clock; document NTP dependency; avoid sleeping through deadline if system clock jumps backward (use saturating comparisons).
  • Epoch edge — ensure first block of new last_buy_epoch triggers fast polls before UI shows stale timer chips.
  • 429 during idle — rate-limit still escalates backoff; idle spacing must not fight report_rate_limited.

Verification criteria

Check Command Expected
Unit + clippy cd indexer && cargo clippy --all-targets -- -D warnings && cargo test PASS, includes idle/fast selection tests
Metrics smoke bash scripts/verify-indexer-rpc-metrics.sh PASS
Arena regression bash scripts/verify-podium-live-anvil.sh PASS
Idle RPC reduction BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh lower calls_per_min_1m / peak_calls_10s vs baseline
Freshness poll /v1/arena/timers during idle polled_at_ms age < 3000 ms
## Parent [#306 — Benchmark indexer JSON-RPC load on localnet and reduce rate-limit risk](https://gitlab.com/PlasticDigits/yieldomega/-/issues/306) Baseline (MR !454): steady-state **~1 Hz** chain-timer polls via `RPC_FAST_POLL_MS` even when `read_block_number` and timer epochs are unchanged. Parent comment: bring **`peak_calls_10s` down to ≤ 50** (from ~354 idle). ## Current codebase - **`indexer/src/chain_timer.rs`** — `run_poll_loop` always sleeps `health.backoff_sleep()` after every `poll_once`; no head-change short-circuit. - **`indexer/src/rpc_poll_health.rs`** — adaptive spacing only on **failure** / HTTP **429** (5s → 15s → 30s). Healthy cadence fixed at **1000 ms**, aligned with `frontend/src/lib/rpcConnectivity.ts`. - **`indexer/src/rpc_metrics.rs`** — exposes `peak_calls_10s`, `calls_per_min_1m`, `by_caller.chain_timer`. - **`GET /v1/arena/timers`** — served from in-memory `TimecurveHeadSnapshot` cache; UX expects ~1s freshness near deadline boundaries. - **`docs/indexer/rpc-load-benchmark.md`** — ranks **adaptive poll interval when head unchanged and timer epochs stable** (est. 30–50% steady-state reduction). ## Why needed Approved follow-up from #306 / MR !454: implement **adaptive poll interval** as a separate mitigation. When Anvil (or mainnet) produces no new blocks and onchain timer/podium epochs are stable, full `poll_once` every second wastes RPC quota and contributes directly to burst peaks. This complements Multicall batching (sibling issue). ## Constraints / guardrails - **Never** skip polls entirely — minimum cadence must guarantee `GET /v1/arena/timers` `polled_at_ms` within **3s** when RPC healthy (per `rpc-load-benchmark.md` proposed SLO). - **Accelerate** to `RPC_FAST_POLL_MS` (1s) when: - `read_block_number` advances, - `last_buy_epoch` or any `podium_epochs[cat]` changes, - wall-clock approaches any `podium_deadlines_sec` or global `deadline_sec` within a configurable margin (e.g. ≤ 30s), - prior poll failed or returned HTTP 429 (`RpcPollHealth` already handles failure tiers). - Idle slow cadence should be **env-configurable** (e.g. `CHAIN_TIMER_IDLE_POLL_MS`, default TBD by benchmark — likely 2–5s) without breaking frontend indexer-first assumptions. - Do **not** change ingestion loop cadence or `INGESTION_ENABLED` behavior. - `#237` WSS hints remain **deferred** — this issue is RPC poll spacing only. ## Relevant files | Path | Role | |------|------| | `indexer/src/chain_timer.rs` | `run_poll_loop`, `poll_once`, snapshot comparison | | `indexer/src/rpc_poll_health.rs` | Failure backoff; extend or sibling for success-path spacing | | `indexer/src/rpc_metrics.rs` | Measure improvement | | `frontend/src/lib/rpcConnectivity.ts` | Reference cadence constants (read-only alignment) | | `scripts/benchmark-indexer-rpc-anvil.sh` | Idle scenario measurement | | `docs/indexer/rpc-load-benchmark.md` | SLO + mitigation table | ## Recommended direction 1. After successful `poll_once`, compare new `TimecurveHeadSnapshot` to cached: - `read_block_number`, `last_buy_epoch`, `podium_epochs`, `deadline_sec`, `podium_deadlines_sec`. 2. If unchanged and not in **deadline proximity window**, sleep `CHAIN_TIMER_IDLE_POLL_MS` instead of `RPC_FAST_POLL_MS`. 3. Optionally still issue lightweight `eth_blockNumber` on idle cycles (1 call) to detect new heads without full `poll_once` — only if it **reduces** total RPC vs full poll; benchmark both variants. 4. Log mode transitions at `debug` (`fast` vs `idle`) for operator forensics. 5. Document env knobs in `indexer/README.md` and `rpc-load-benchmark.md`. ## Acceptance criteria - [ ] Chain-timer uses **fast 1s** polls when head or epoch fields change or near deadline; **slower idle** polls otherwise. - [ ] `bash scripts/benchmark-indexer-rpc-anvil.sh` idle scenario: `calls_per_min_1m` drops ≥ **30%** vs `rpc-benchmark-20260610T111415Z.json` baseline (or combined with Multicall issue, `peak_calls_10s` ≤ **50**). - [ ] `GET /v1/arena/timers` `polled_at_ms` ≤ **3s** stale under healthy RPC during idle Anvil run. - [ ] Near-miss test: with artificial wall-clock mock or short deadline on Anvil, poller returns to 1s cadence within one slow interval of epoch change. - [ ] `RpcPollHealth` failure / 429 backoff unchanged. - [ ] `INV-INDEXER-306-NO-REGRESSION` passes. ## Test plan ```bash export PATH="$HOME/.foundry/bin:$PATH" source /usr/local/cargo/env 2>/dev/null || true cd indexer && cargo clippy --all-targets -- -D warnings && cargo test # New unit tests: snapshot diff → fast vs idle sleep selection; deadline proximity bash scripts/verify-indexer-rpc-metrics.sh bash scripts/verify-podium-live-anvil.sh BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh # Manual: curl -s http://127.0.0.1:3103/v1/arena/timers | jq '.polled_at_ms' twice 1s apart — should update in fast mode ``` ## Attack / abuse test plan - **Stuck head** — if RPC returns constant `eth_blockNumber` while chain advances on another node, ingestion lag alerts ([#168](https://gitlab.com/PlasticDigits/yieldomega/-/issues/168)) still fire; idle mode must not suppress ingestion. - **Clock skew** — deadline proximity uses wall clock; document NTP dependency; avoid sleeping through deadline if system clock jumps backward (use saturating comparisons). - **Epoch edge** — ensure first block of new `last_buy_epoch` triggers fast polls before UI shows stale timer chips. - **429 during idle** — rate-limit still escalates backoff; idle spacing must not fight `report_rate_limited`. ## Verification criteria | Check | Command | Expected | |-------|---------|----------| | Unit + clippy | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` | PASS, includes idle/fast selection tests | | Metrics smoke | `bash scripts/verify-indexer-rpc-metrics.sh` | PASS | | Arena regression | `bash scripts/verify-podium-live-anvil.sh` | PASS | | Idle RPC reduction | `BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh` | lower `calls_per_min_1m` / `peak_calls_10s` vs baseline | | Freshness | poll `/v1/arena/timers` during idle | `polled_at_ms` age < 3000 ms |
PlasticDigits commented 2026-06-13 06:35:43 +00:00 (Migrated from gitlab.com)

mentioned in issue #306

mentioned in issue #306
PlasticDigits commented 2026-06-13 07:27:08 +00:00 (Migrated from gitlab.com)

mentioned in merge request !455

mentioned in merge request !455
PlasticDigits commented 2026-06-13 09:26:46 +00:00 (Migrated from gitlab.com)

mentioned in commit 11397d80e9

mentioned in commit 11397d80e933dd0ad344f40a165e3fd5debe9584
PlasticDigits commented 2026-06-13 09:31:07 +00:00 (Migrated from gitlab.com)

mentioned in commit dff689cad3

mentioned in commit dff689cad32eaeca286ee587c6d24a0156f83c7e
PlasticDigits commented 2026-06-13 09:31:12 +00:00 (Migrated from gitlab.com)

mentioned in merge request !456

mentioned in merge request !456
PlasticDigits commented 2026-06-13 13:59:29 +00:00 (Migrated from gitlab.com)

Verification — GitLab #308 (adaptive chain-timer poll interval)

Verified against main @ 1a7b8ed5 (includes merged MR !455). Local checkout was initially stale; pulled before re-running checks.

Results

Criterion Command / step Result
Fast 1s vs idle 3s adaptive spacing cd indexer && cargo test --lib chain_timer_poll (17 tests: unchanged_head_is_idle, block_advance_is_fast, epoch_change_is_fast, podium_deadline_proximity_is_fast, failure_backoff_*, rate_limit_overrides_idle_on_success, …) PASS
Unit + clippy cd indexer && cargo clippy --all-targets -- -D warnings && cargo test PASS (60 lib + 4 integration without YIELDOMEGA_PG_TEST_URL)
RpcPollHealth failure / 429 backoff unchanged chain_timer_poll + rpc_poll_health tests PASS
Metrics smoke (INV-INDEXER-306-STATUS-METRICS) bash scripts/verify-indexer-rpc-metrics.sh (after cargo build --release) PASS — schema=2.11.0 total_calls=56 peak_10s=20
Idle RPC reduction ≥30% vs baseline BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh vs docs/indexer/benchmarks/rpc-benchmark-20260610T111415Z.json idle last sample PASS — calls_per_min_1m 740 → 84 (−89%); peak_calls_10s 354 → 20 (target ≤50)
Timer freshness SLO (≤3s) CHAIN_TIMER_IDLE_POLL_MS clamped to 3000; refresh_snapshot_polled_at_ms on idle short-circuit; unit tests PASS (indirect — see note)
Near-miss: epoch/block → fast within one idle interval Unit tests block_advance_is_fast, podium_epoch_change_is_fast PASS
Arena regression (INV-INDEXER-306-NO-REGRESSION) bash scripts/verify-podium-live-anvil.sh FAIL (unrelated — see below)
Manual: curl …/v1/arena/timers | jq .polled_at_ms API response inspection SKIP — polled_at_ms is maintained on the in-memory snapshot but not serialized on GET /v1/arena/timers; freshness enforced by 3s idle cap + short-circuit refresh

How verified

  1. Pulled latest main (merge 11397d80 / MR !455).
  2. cargo clippy --all-targets -- -D warnings && cargo test — all green.
  3. cargo test --lib chain_timer_poll — 17/17 adaptive poll selection tests.
  4. Pre-built release indexer (cargo build --release) so Anvil verify scripts are not blocked by cold-compile timeouts.
  5. bash scripts/verify-indexer-rpc-metrics.sh — status exposes rpc_metrics with chain-timer traffic after warm-up.
  6. BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh — wrote docs/indexer/benchmarks/rpc-benchmark-20260613T135052Z.json; idle steady-state matches shipped benchmark doc (~84 calls/min, peak 20/10s).
  7. bash scripts/verify-podium-live-anvil.sh — podium/chain-timer HTTP checks completed; failed at tail cargo test --test integration_stage2 with arena_wallet_stats_two_epochs_and_bonus_fields: warbow_battle_points 1000 vs expected 750 (stale assertion after WarBow BP routing #310; not introduced by chain-timer spacing).

#308-specific acceptance: PASS

Adaptive poll implementation (indexer/src/chain_timer_poll.rs, chain_timer.rs loop) meets spacing, RPC reduction, failure-backoff, and near-deadline fast-mode requirements.

Follow-ups (out of scope for #308)

  • Update arena_wallet_stats_two_epochs_and_bonus_fields expected warbow_battle_points to 1000 (or file under #310) so verify-podium-live-anvil.sh regains green INV-INDEXER-306-NO-REGRESSION.
  • Optional: expose polled_at_ms on GET /v1/arena/timers if operators want direct HTTP freshness probes (SLO is already enforced in-process).
## Verification — GitLab #308 (adaptive chain-timer poll interval) Verified against **`main` @ `1a7b8ed5`** (includes merged MR !455). Local checkout was initially stale; pulled before re-running checks. ### Results | Criterion | Command / step | Result | |-----------|----------------|--------| | Fast 1s vs idle 3s adaptive spacing | `cd indexer && cargo test --lib chain_timer_poll` (17 tests: `unchanged_head_is_idle`, `block_advance_is_fast`, `epoch_change_is_fast`, `podium_deadline_proximity_is_fast`, `failure_backoff_*`, `rate_limit_overrides_idle_on_success`, …) | **PASS** | | Unit + clippy | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` | **PASS** (60 lib + 4 integration without `YIELDOMEGA_PG_TEST_URL`) | | `RpcPollHealth` failure / 429 backoff unchanged | `chain_timer_poll` + `rpc_poll_health` tests | **PASS** | | Metrics smoke (`INV-INDEXER-306-STATUS-METRICS`) | `bash scripts/verify-indexer-rpc-metrics.sh` (after `cargo build --release`) | **PASS** — `schema=2.11.0` `total_calls=56` `peak_10s=20` | | Idle RPC reduction ≥30% vs baseline | `BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh` vs `docs/indexer/benchmarks/rpc-benchmark-20260610T111415Z.json` idle last sample | **PASS** — `calls_per_min_1m` **740 → 84** (−89%); `peak_calls_10s` **354 → 20** (target ≤50) | | Timer freshness SLO (≤3s) | `CHAIN_TIMER_IDLE_POLL_MS` clamped to 3000; `refresh_snapshot_polled_at_ms` on idle short-circuit; unit tests | **PASS** (indirect — see note) | | Near-miss: epoch/block → fast within one idle interval | Unit tests `block_advance_is_fast`, `podium_epoch_change_is_fast` | **PASS** | | Arena regression (`INV-INDEXER-306-NO-REGRESSION`) | `bash scripts/verify-podium-live-anvil.sh` | **FAIL** (unrelated — see below) | | Manual: `curl …/v1/arena/timers \| jq .polled_at_ms` | API response inspection | **SKIP** — `polled_at_ms` is maintained on the in-memory snapshot but **not** serialized on `GET /v1/arena/timers`; freshness enforced by 3s idle cap + short-circuit refresh | ### How verified 1. Pulled latest `main` (merge `11397d80` / MR !455). 2. `cargo clippy --all-targets -- -D warnings && cargo test` — all green. 3. `cargo test --lib chain_timer_poll` — 17/17 adaptive poll selection tests. 4. Pre-built release indexer (`cargo build --release`) so Anvil verify scripts are not blocked by cold-compile timeouts. 5. `bash scripts/verify-indexer-rpc-metrics.sh` — status exposes `rpc_metrics` with chain-timer traffic after warm-up. 6. `BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh` — wrote `docs/indexer/benchmarks/rpc-benchmark-20260613T135052Z.json`; idle steady-state matches shipped benchmark doc (~84 calls/min, peak 20/10s). 7. `bash scripts/verify-podium-live-anvil.sh` — podium/chain-timer HTTP checks completed; **failed at tail** `cargo test --test integration_stage2` with `arena_wallet_stats_two_epochs_and_bonus_fields`: `warbow_battle_points` **1000** vs expected **750** (stale assertion after WarBow BP routing #310; not introduced by chain-timer spacing). ### #308-specific acceptance: **PASS** Adaptive poll implementation (`indexer/src/chain_timer_poll.rs`, `chain_timer.rs` loop) meets spacing, RPC reduction, failure-backoff, and near-deadline fast-mode requirements. ### Follow-ups (out of scope for #308) - Update `arena_wallet_stats_two_epochs_and_bonus_fields` expected `warbow_battle_points` to **1000** (or file under #310) so `verify-podium-live-anvil.sh` regains green `INV-INDEXER-306-NO-REGRESSION`. - Optional: expose `polled_at_ms` on `GET /v1/arena/timers` if operators want direct HTTP freshness probes (SLO is already enforced in-process).
PlasticDigits (Migrated from gitlab.com) closed this issue 2026-06-13 13:59:30 +00:00
PlasticDigits commented 2026-06-15 02:27:09 +00:00 (Migrated from gitlab.com)

mentioned in merge request !503

mentioned in merge request !503
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
cac-design-issue-544
renovate/configure
No results found.
Labels
Clear labels   
agent:fix_bugfix

   
agent:fix_conflicts

   
agent:fix_security

   
agent:gap_analysis

   
agent:implement

   
agent:open_issues

   
agents

   
agent:security_audit

   
agent:verify
  
area
timecurve

  
arena

   
arena-v2

   
audit

   
block:feature_review

   
block:security

   
bug

   
contracts

   
design

   
devops

   
docs
Documentation

  
documentation

   
enhancement

   
epic
Epic / umbrella issue

  
feature

   
frontend

   
indexer

   
low priority
Low priority, investigate other issues first

  
megaeth

   
operations

   
ops
Deploy and operations

  
performance

   
QA

   
qa checklist

   
referrals

   
removal
Legacy removal

  
security

   
smart-contracts

   
tech-debt

   
testing

   
time-arena
TimeArena v2 revamp

  
timecurve

   
visual-qa
No labels
agent:fix_bugfix
agent:fix_conflicts
agent:fix_security
agent:gap_analysis
agent:implement
agent:open_issues
agents
agent:security_audit
agent:verify
area
timecurve
arena
arena-v2
audit
block:feature_review
block:security
bug
contracts
design
devops
docs
documentation
enhancement
epic
feature
frontend
indexer
low priority
megaeth
operations
ops
performance
QA
qa checklist
referrals
removal
security
smart-contracts
tech-debt
testing
time-arena
timecurve
visual-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
code/yieldomega#308
No description provided.