code/yieldomega
Watch
4
0
Fork
You've already forked yieldomega
 0
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity

Indexer: per-block Multicall3 batching for warbow_score + podium_live ingest RPC #356

New issue
Closed
opened 2026-06-28 05:40:31 +00:00 by PlasticDigits · 9 comments
PlasticDigits commented 2026-06-28 05:40:31 +00:00 (Migrated from gitlab.com)

Summary

Batch warbow_score and arena_podium_live post-log eth_call side-effects during block ingestion into one Multicall3 aggregate3 per block (at BlockId::Number(block)), mirroring the chain-timer head poller (#307). Today each Buy / WarBow log triggers ~9 sequential eth_calls; per-block planning + dedupe should collapse that to ~1 counted RPC per block (plus unchanged single-call paths where batching has no win).

Parent / related: #306 (RPC metrics) · #307 (chain-timer Multicall3) · #140 (transactional block ingest) · #317 (ingest side-effects fail-closed) · #273 (live podiums) · #254 (WarBow BP snapshots).

Current codebase

Ingestion loop (indexer/src/ingestion.rs)

  • Walks blocks sequentially: eth_getBlockByNumber + eth_getLogs per block.
  • Decodes logs and persists inside one Postgres transaction per block (INV-INDEXER-140).
  • For each TimeArena log, after persist_decoded_log_conn, ingestion runs two independent RPC paths:
    1. warbow_score::snapshot_warbow_players_after_log — podiumEpoch(3) + battlePoints(player) per affected player (2+ eth_calls per log).
    2. arena_podium_live::snapshot_live_podium_after_log — sequential epoch lookups + podium(cat) snapshots (up to 7 eth_calls on ArenaBuy).

WarBow snapshots (indexer/src/warbow_score.rs)

  • eth_call_u256 records one RpcMethod::EthCall per call with RpcCaller::WarbowScore.
  • Triggered on ArenaBuy, steal/revenge/flag events (not on ArenaWarbowEpochScore — already emitted on-chain).
  • Uses encode_u8_call / encode_address_call against TimeArena at d.block_number.

Live podium projections (indexer/src/arena_podium_live.rs)

  • snapshot_live_podium_after_log branches on event type; ArenaBuy is the expensive path:
    • lastBuyEpoch → podium(0) → podiumEpoch(1,2) + podium(1,2) → podiumEpoch(3) → DB rollup for WarBow top-3.
  • podium_at_block records RpcCaller::PodiumLive per eth_call.
  • podiumEpoch(3) is fetched twice on buys (warbow path + podium path).

Existing Multicall3 transport (indexer/src/multicall.rs)

  • MulticallBatch builder with dedupe on (target, callData).
  • aggregate3_at_block — allowFailure: false for every sub-call; records one logical eth_call in rpc_metrics.
  • Used by chain_timer::poll_once_multicall (#307); falls back to sequential reads on failure.

RPC metrics (indexer/src/rpc_metrics.rs)

  • by_caller / by_method are raw call counts in a 5-minute window, not per logical update.
  • Full-stack benchmark with bot swarm (2026-06-28): ~809 calls/min total; warbow_score + podium_live ~1,100 eth_calls / 5 min vs ~180/min for chain_timer.

Policy (docs/indexer/design.md, INV-INDEXER-317-INGEST-SIDE-EFFECTS)

  • RPC snapshot failure aborts the block transaction — no warn-and-continue partial derived state.

Why this is needed

  1. RPC amplification under load — Bot swarm / busy arena produces many Buy logs per block; each buy currently adds ~9 sequential eth_calls on top of per-block getBlock + getLogs. This dominates indexer RPC during active play (see docs/indexer/benchmarks/rpc-benchmark-fullstack-swarm-20260628T052907Z.json).
  2. Duplicate reads — Same-block podiumEpoch / epoch head reads are repeated across logs and across warbow vs podium paths; MulticallBatch dedupe eliminates redundant sub-calls.
  3. Provider quota / burst — #306 tracks peak_calls_10s; ingest-side eth_call bursts scale with tx count, not block time.
  4. Parity with chain-timer — Head poller already batches ~30 contract reads into one Multicall3 call; ingest side-effects should follow the same pattern.

Constraints and guardrails

Guardrail Requirement
INV-INDEXER-140 Single Postgres transaction per block; DB writes for warbow/podium remain per log in decode order after RPC results are available.
INV-INDEXER-317-INGEST-SIDE-EFFECTS Multicall or sub-call failure must abort the block tx (same as today). No silent skip of derived tables.
INV-INDEXER-273-* GET /v1/arena/podiums live predictions must match pre-change behavior; podium_prediction semantics unchanged.
INV-INDEXER-307-MULTICALL-BATCH Reuse multicall.rs; allowFailure: false on all sub-calls — historical eth_calls at log block should not revert on healthy chain state; a revert indicates indexer bug or corrupt log context and must fail the block.
INV-INDEXER-273-CHAIN-TIMER-SELECTORS Continue using podiumEpoch(uint256) / podiumDeadline(uint256) array getters (not uint8 overloads).
Onchain authority Indexer remains derived-only; batching must not change decoded values or business rules.
Multicall3 presence Anvil bootstrap via scripts/lib/anvil_multicall3.sh (already in stack). Production MegaETH ships canonical 0xcA11…CA11.
Fallback If aggregate3 fails (missing Multicall3, transport error), fall back to existing sequential path for that block (same as chain_timer poll_once_sequential) — preserves liveness; log at debug/warn.
Scope Indexer-only; no frontend or contract changes.

Relevant files

File Role
indexer/src/ingestion.rs Replace per-log dual RPC calls with per-block batch orchestration
indexer/src/warbow_score.rs Refactor to consume batched results (or move planning to new module)
indexer/src/arena_podium_live.rs Same — event-specific read planning
indexer/src/multicall.rs Add push_address_arg; possibly export decode helpers
indexer/src/chain_timer.rs encode_u8_call, decode_podium_return, selector constants
indexer/src/rpc_metrics.rs Attribution: one eth_call per aggregate3 (consider RpcCaller::Ingestion vs split warbow/podium — document choice)
indexer/src/rpc_http.rs Transport (unchanged unless sticky provider helper needed)
indexer/tests/integration_stage2.rs Parity + rollback smoke
scripts/verify-podium-live-anvil.sh E2E live podium ingest
scripts/verify-wallet-profile-anvil.sh WarBow BP / wallet stats
scripts/benchmark-indexer-rpc-anvil.sh RPC regression (active-arena scenario)
docs/indexer/design.md Document new ingest batching + invariant IDs
docs/testing/invariants-and-business-logic.md Add INV-INDEXER-329-* (or next free ID)
docs/indexer/rpc-load-benchmark.md Refresh baseline notes

Recommended solution direction

1. New module: indexer/src/arena_ingest_rpc.rs (name TBD)

  • plan_block_rpc_batch(block, arena_logs: &[DecodedLog]) -> BlockRpcPlan
    • Walk logs in order; for each TimeArena log, append required sub-calls to a MulticallBatch (dedupe handles shared podiumEpoch reads).
    • Map each logical need (e.g. battlePoints(player), podium(cat, epoch)) to a stable result index in the batch.
  • execute_block_rpc_batch(providers, block, batch, metrics) -> Vec<Bytes>
    • Single aggregate3_at_block with allowFailure: false.
    • On failure → sequential fallback executing the same plan (counts N eth_calls — acceptable degradation).

2. Per-event read sets (planning)

Event Sub-calls to plan
ArenaBuy lastBuyEpoch, podiumEpoch(0..3), podium(0..2) at block, battlePoints(buyer)
ArenaWarbowSteal / Revenge / FlagClaimed / Guard podiumEpoch(3), battlePoints per player
ArenaPodiumEpochRolled podium(category)
ArenaLastBuyEpochStarted podium(0)
ArenaWarbowEpochScore No RPC (on-chain score row; warbow path already skipped)

3. Ingestion integration

for each block:
  decode all logs → persist rows in tx
  plan + execute ONE multicall (or fallback) BEFORE or interleaved with per-log DB side-effects
  apply results to warbow_score + arena_podium_live writers (pure DB + decode, no network)
  commit tx

Preferred: plan from decoded logs before RPC, execute one multicall, then run existing insert helpers with pre-fetched bytes (minimize behavior drift).

4. MulticallBatch::push_address_arg

Mirror warbow_score::encode_address_call for battlePoints(address).

5. Metrics

  • Record one eth_call per successful aggregate3 (match #307).
  • Optional: structured log ingest_multicall_subcalls=N deduped_from=M.

Acceptance criteria

  • AC-1: For a block with one or more ArenaBuy logs, ingest performs at most one Multicall3 aggregate3 eth_call for all warbow + podium live reads at that block (excluding fallback path).
  • AC-2: idx_warbow_epoch_score and idx_arena_podium_live rows for a given test fixture are byte-identical to pre-change sequential ingest (same players, scores, epochs, block tags).
  • AC-3: GET /v1/arena/podiums and GET /v1/arena/wallet/{addr}/stats responses unchanged for verify-podium-live-anvil.sh and verify-wallet-profile-anvil.sh.
  • AC-4: allowFailure: false on all sub-calls; any sub-call failure fails the batch and rolls back the block transaction.
  • AC-5: Multicall3 unavailable → sequential fallback produces same DB state; ingest completes (Anvil without Multicall3 deploy test or mock).
  • AC-6: Reorg rollback still clears idx_arena_podium_live / warbow tables (integration_stage2 rollback assertions pass).
  • AC-7: Active-arena RPC benchmark: eth_call:warbow_score + eth_call:podium_live per-minute counts drop ≥80% vs pre-change baseline on same harness (benchmark-indexer-rpc-anvil.sh, active_arena scenario).
  • AC-8: New invariant IDs documented in invariants-and-business-logic.md + design.md.

Test plan — functional paths

# Path Command / test
T-1 Unit: batch planning for single ArenaBuy cargo test in arena_ingest_rpc — assert sub-call count + dedupe of podiumEpoch(3)
T-2 Unit: multi-buy same block Plan 3 buys → shared epoch reads deduped to 1 sub-call each
T-3 Unit: WarBow steal (2 players) Plan includes 2× battlePoints + podiumEpoch(3)
T-4 Unit: ArenaPodiumEpochRolled Single podium(cat) sub-call
T-5 Unit: ArenaLastBuyEpochStarted Single podium(0)
T-6 Unit: ArenaWarbowEpochScore Plan empty RPC (no sub-calls)
T-7 Integration: persist all events + rollback integration_stage2.rs::postgres_stage2_persist_all_events_and_rollback_after
T-8 Integration: live podiums smoke integration_stage2.rs::arena_podiums_live_predictions_smoke
T-9 Anvil E2E: podium live bash scripts/verify-podium-live-anvil.sh
T-10 Anvil E2E: wallet profile / WarBow bash scripts/verify-wallet-profile-anvil.sh
T-11 RPC metrics smoke bash scripts/verify-indexer-rpc-metrics.sh
T-12 RPC benchmark regression BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh — compare active_arena eth_call totals
T-13 Full stack swarm (manual) START_BOT_SWARM=1 bash scripts/start-qa-local-full-stack.sh --live-sale + sample /v1/status/ops rpc_metrics
T-14 Clippy + unit cd indexer && cargo clippy --all-targets -- -D warnings && cargo test

Test plan — attack, hack, and abuse vectors

Vector Threat Test / verification
A-1: RPC griefing / rate limit Provider returns 429 mid-block Ingest aborts block tx; retries block; no partial idx_arena_podium_live rows (INV-INDEXER-317). Simulate with mock provider or timeout.
A-2: Malicious log ordering Attacker stuffs many Buy logs in one block to amplify RPC Per-block single multicall caps RPC at O(unique reads), not O(logs); benchmark block with N buys shows flat eth_call count.
A-3: Reorg after snapshot Shallow reorg drops block with live podium rows rollback_after clears derived tables; re-ingest reproduces state (integration_stage2 reorg test).
A-4: Wrong selector / uint8 podium getter Calling wrong ABI reverts whole batch Selector unit tests; verify-podium-live-anvil.sh; batch must use podiumEpoch(uint256) only.
A-5: Stale block tag Using head instead of log block mis-attributes podium Plan always uses BlockId::Number(d.block_number); integration fixture asserts block_number on stored rows matches log.
A-6: Multicall3 missing / malicious Deploy absent or wrong contract Fallback sequential path; Anvil bootstrap script ensures canonical deploy; fail block if both paths fail.
A-7: Duplicate log_index / PK clash Derivative warbow rows use synthetic log_index Existing log_index * 10 + i scheme unchanged; integration asserts no PK violations under batch path.
A-8: Ingestion race / double-apply Retried block after crash Transactional per-block ingest unchanged; idempotent ON CONFLICT DO NOTHING on warbow inserts preserved.

Verification criteria (merge gate)

Layer Command Required
Indexer cd indexer && cargo clippy --all-targets -- -D warnings && cargo test PASS
Podium live bash scripts/verify-podium-live-anvil.sh PASS
Wallet profile bash scripts/verify-wallet-profile-anvil.sh PASS
RPC metrics bash scripts/verify-indexer-rpc-metrics.sh PASS
RPC benchmark BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh PASS; document active_arena warbow_score+podium_live reduction in MR
Docs Update design.md, invariants-and-business-logic.md, rpc-load-benchmark.md Included in MR

Out of scope

  • Batching chain_timer further (done in #307).
  • Batching eth_getBlockByNumber / eth_getLogs across blocks.
  • Frontend RPC changes (#301).
  • allowFailure: true per sub-call (explicitly not desired — reverts should not occur on valid historical reads).

Suggested labels

indexer, performance, arena-v2

Suggested weight

5

## Summary Batch **`warbow_score`** and **`arena_podium_live`** post-log `eth_call` side-effects during block ingestion into **one Multicall3 `aggregate3` per block** (at `BlockId::Number(block)`), mirroring the chain-timer head poller ([#307](https://gitlab.com/PlasticDigits/yieldomega/-/issues/307)). Today each `Buy` / WarBow log triggers **~9 sequential `eth_call`s**; per-block planning + dedupe should collapse that to **~1 counted RPC** per block (plus unchanged single-call paths where batching has no win). Parent / related: [#306](https://gitlab.com/PlasticDigits/yieldomega/-/issues/306) (RPC metrics) · [#307](https://gitlab.com/PlasticDigits/yieldomega/-/issues/307) (chain-timer Multicall3) · [#140](https://gitlab.com/PlasticDigits/yieldomega/-/issues/140) (transactional block ingest) · [#317](https://gitlab.com/PlasticDigits/yieldomega/-/issues/317) (ingest side-effects fail-closed) · [#273](https://gitlab.com/PlasticDigits/yieldomega/-/issues/273) (live podiums) · [#254](https://gitlab.com/PlasticDigits/yieldomega/-/issues/254) (WarBow BP snapshots). --- ## Current codebase ### Ingestion loop (`indexer/src/ingestion.rs`) - Walks blocks sequentially: `eth_getBlockByNumber` + `eth_getLogs` per block. - Decodes logs and persists inside **one Postgres transaction per block** (`INV-INDEXER-140`). - For each `TimeArena` log, after `persist_decoded_log_conn`, ingestion runs **two independent RPC paths**: 1. **`warbow_score::snapshot_warbow_players_after_log`** — `podiumEpoch(3)` + `battlePoints(player)` per affected player (2+ `eth_call`s per log). 2. **`arena_podium_live::snapshot_live_podium_after_log`** — sequential epoch lookups + `podium(cat)` snapshots (up to **7 `eth_call`s** on `ArenaBuy`). ### WarBow snapshots (`indexer/src/warbow_score.rs`) - `eth_call_u256` records **one** `RpcMethod::EthCall` per call with `RpcCaller::WarbowScore`. - Triggered on `ArenaBuy`, steal/revenge/flag events (not on `ArenaWarbowEpochScore` — already emitted on-chain). - Uses `encode_u8_call` / `encode_address_call` against `TimeArena` at **`d.block_number`**. ### Live podium projections (`indexer/src/arena_podium_live.rs`) - `snapshot_live_podium_after_log` branches on event type; **`ArenaBuy`** is the expensive path: - `lastBuyEpoch` → `podium(0)` → `podiumEpoch(1,2)` + `podium(1,2)` → `podiumEpoch(3)` → DB rollup for WarBow top-3. - `podium_at_block` records `RpcCaller::PodiumLive` per `eth_call`. - **`podiumEpoch(3)` is fetched twice** on buys (warbow path + podium path). ### Existing Multicall3 transport (`indexer/src/multicall.rs`) - `MulticallBatch` builder with **dedupe** on `(target, callData)`. - `aggregate3_at_block` — **`allowFailure: false`** for every sub-call; records **one** logical `eth_call` in `rpc_metrics`. - Used by **`chain_timer::poll_once_multicall`** ([#307](https://gitlab.com/PlasticDigits/yieldomega/-/issues/307)); falls back to sequential reads on failure. ### RPC metrics (`indexer/src/rpc_metrics.rs`) - `by_caller` / `by_method` are **raw call counts** in a 5-minute window, not per logical update. - Full-stack benchmark with bot swarm (2026-06-28): ~**809 calls/min** total; **`warbow_score` + `podium_live` ~1,100 `eth_call`s / 5 min** vs ~**180/min** for `chain_timer`. ### Policy (`docs/indexer/design.md`, `INV-INDEXER-317-INGEST-SIDE-EFFECTS`) - RPC snapshot failure **aborts the block transaction** — no warn-and-continue partial derived state. --- ## Why this is needed 1. **RPC amplification under load** — Bot swarm / busy arena produces many `Buy` logs per block; each buy currently adds ~9 sequential `eth_call`s on top of per-block `getBlock` + `getLogs`. This dominates indexer RPC during active play (see `docs/indexer/benchmarks/rpc-benchmark-fullstack-swarm-20260628T052907Z.json`). 2. **Duplicate reads** — Same-block `podiumEpoch` / epoch head reads are repeated across logs and across warbow vs podium paths; `MulticallBatch` dedupe eliminates redundant sub-calls. 3. **Provider quota / burst** — [#306](https://gitlab.com/PlasticDigits/yieldomega/-/issues/306) tracks `peak_calls_10s`; ingest-side `eth_call` bursts scale with tx count, not block time. 4. **Parity with chain-timer** — Head poller already batches ~30 contract reads into one Multicall3 call; ingest side-effects should follow the same pattern. --- ## Constraints and guardrails | Guardrail | Requirement | |-----------|-------------| | **`INV-INDEXER-140`** | Single Postgres transaction per block; DB writes for warbow/podium remain **per log in decode order** after RPC results are available. | | **`INV-INDEXER-317-INGEST-SIDE-EFFECTS`** | Multicall or sub-call failure **must abort** the block tx (same as today). No silent skip of derived tables. | | **`INV-INDEXER-273-*`** | `GET /v1/arena/podiums` live predictions must match pre-change behavior; `podium_prediction` semantics unchanged. | | **`INV-INDEXER-307-MULTICALL-BATCH`** | Reuse `multicall.rs`; **`allowFailure: false`** on all sub-calls — historical `eth_call`s at log block **should not revert** on healthy chain state; a revert indicates indexer bug or corrupt log context and must fail the block. | | **`INV-INDEXER-273-CHAIN-TIMER-SELECTORS`** | Continue using **`podiumEpoch(uint256)`** / **`podiumDeadline(uint256)`** array getters (not `uint8` overloads). | | **Onchain authority** | Indexer remains derived-only; batching must not change decoded values or business rules. | | **Multicall3 presence** | Anvil bootstrap via `scripts/lib/anvil_multicall3.sh` (already in stack). Production MegaETH ships canonical `0xcA11…CA11`. | | **Fallback** | If `aggregate3` fails (missing Multicall3, transport error), fall back to **existing sequential path** for that block (same as `chain_timer` `poll_once_sequential`) — preserves liveness; log at `debug`/`warn`. | | **Scope** | Indexer-only; no frontend or contract changes. | --- ## Relevant files | File | Role | |------|------| | `indexer/src/ingestion.rs` | Replace per-log dual RPC calls with per-block batch orchestration | | `indexer/src/warbow_score.rs` | Refactor to consume batched results (or move planning to new module) | | `indexer/src/arena_podium_live.rs` | Same — event-specific read planning | | `indexer/src/multicall.rs` | Add `push_address_arg`; possibly export decode helpers | | `indexer/src/chain_timer.rs` | `encode_u8_call`, `decode_podium_return`, selector constants | | `indexer/src/rpc_metrics.rs` | Attribution: one `eth_call` per `aggregate3` (consider `RpcCaller::Ingestion` vs split warbow/podium — document choice) | | `indexer/src/rpc_http.rs` | Transport (unchanged unless sticky provider helper needed) | | `indexer/tests/integration_stage2.rs` | Parity + rollback smoke | | `scripts/verify-podium-live-anvil.sh` | E2E live podium ingest | | `scripts/verify-wallet-profile-anvil.sh` | WarBow BP / wallet stats | | `scripts/benchmark-indexer-rpc-anvil.sh` | RPC regression (active-arena scenario) | | `docs/indexer/design.md` | Document new ingest batching + invariant IDs | | `docs/testing/invariants-and-business-logic.md` | Add `INV-INDEXER-329-*` (or next free ID) | | `docs/indexer/rpc-load-benchmark.md` | Refresh baseline notes | --- ## Recommended solution direction ### 1. New module: `indexer/src/arena_ingest_rpc.rs` (name TBD) - **`plan_block_rpc_batch(block, arena_logs: &[DecodedLog]) -> BlockRpcPlan`** - Walk logs in order; for each `TimeArena` log, append required sub-calls to a `MulticallBatch` (dedupe handles shared `podiumEpoch` reads). - Map each logical need (e.g. `battlePoints(player)`, `podium(cat, epoch)`) to a **stable result index** in the batch. - **`execute_block_rpc_batch(providers, block, batch, metrics) -> Vec<Bytes>`** - Single `aggregate3_at_block` with `allowFailure: false`. - On failure → sequential fallback executing the same plan (counts N `eth_call`s — acceptable degradation). ### 2. Per-event read sets (planning) | Event | Sub-calls to plan | |-------|-------------------| | `ArenaBuy` | `lastBuyEpoch`, `podiumEpoch(0..3)`, `podium(0..2)` at block, `battlePoints(buyer)` | | `ArenaWarbowSteal` / `Revenge` / `FlagClaimed` / `Guard` | `podiumEpoch(3)`, `battlePoints` per player | | `ArenaPodiumEpochRolled` | `podium(category)` | | `ArenaLastBuyEpochStarted` | `podium(0)` | | `ArenaWarbowEpochScore` | **No RPC** (on-chain score row; warbow path already skipped) | ### 3. Ingestion integration ``` for each block: decode all logs → persist rows in tx plan + execute ONE multicall (or fallback) BEFORE or interleaved with per-log DB side-effects apply results to warbow_score + arena_podium_live writers (pure DB + decode, no network) commit tx ``` Preferred: **plan from decoded logs before RPC**, execute **one** multicall, then run existing insert helpers with pre-fetched bytes (minimize behavior drift). ### 4. `MulticallBatch::push_address_arg` Mirror `warbow_score::encode_address_call` for `battlePoints(address)`. ### 5. Metrics - Record **one** `eth_call` per successful `aggregate3` (match #307). - Optional: structured log `ingest_multicall_subcalls=N deduped_from=M`. --- ## Acceptance criteria - [ ] **AC-1:** For a block with one or more `ArenaBuy` logs, ingest performs **at most one** Multicall3 `aggregate3` `eth_call` for all warbow + podium live reads at that block (excluding fallback path). - [ ] **AC-2:** `idx_warbow_epoch_score` and `idx_arena_podium_live` rows for a given test fixture are **byte-identical** to pre-change sequential ingest (same players, scores, epochs, block tags). - [ ] **AC-3:** `GET /v1/arena/podiums` and `GET /v1/arena/wallet/{addr}/stats` responses unchanged for `verify-podium-live-anvil.sh` and `verify-wallet-profile-anvil.sh`. - [ ] **AC-4:** `allowFailure: false` on all sub-calls; any sub-call failure fails the batch and **rolls back** the block transaction. - [ ] **AC-5:** Multicall3 unavailable → sequential fallback produces same DB state; ingest completes (Anvil without Multicall3 deploy test or mock). - [ ] **AC-6:** Reorg rollback still clears `idx_arena_podium_live` / warbow tables (`integration_stage2` rollback assertions pass). - [ ] **AC-7:** Active-arena RPC benchmark: `eth_call:warbow_score` + `eth_call:podium_live` per-minute counts drop **≥80%** vs pre-change baseline on same harness (`benchmark-indexer-rpc-anvil.sh`, active_arena scenario). - [ ] **AC-8:** New invariant IDs documented in `invariants-and-business-logic.md` + `design.md`. --- ## Test plan — functional paths | # | Path | Command / test | |---|------|----------------| | T-1 | Unit: batch planning for single `ArenaBuy` | `cargo test` in `arena_ingest_rpc` — assert sub-call count + dedupe of `podiumEpoch(3)` | | T-2 | Unit: multi-buy same block | Plan 3 buys → shared epoch reads deduped to 1 sub-call each | | T-3 | Unit: WarBow steal (2 players) | Plan includes 2× `battlePoints` + `podiumEpoch(3)` | | T-4 | Unit: `ArenaPodiumEpochRolled` | Single `podium(cat)` sub-call | | T-5 | Unit: `ArenaLastBuyEpochStarted` | Single `podium(0)` | | T-6 | Unit: `ArenaWarbowEpochScore` | Plan empty RPC (no sub-calls) | | T-7 | Integration: persist all events + rollback | `integration_stage2.rs::postgres_stage2_persist_all_events_and_rollback_after` | | T-8 | Integration: live podiums smoke | `integration_stage2.rs::arena_podiums_live_predictions_smoke` | | T-9 | Anvil E2E: podium live | `bash scripts/verify-podium-live-anvil.sh` | | T-10 | Anvil E2E: wallet profile / WarBow | `bash scripts/verify-wallet-profile-anvil.sh` | | T-11 | RPC metrics smoke | `bash scripts/verify-indexer-rpc-metrics.sh` | | T-12 | RPC benchmark regression | `BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh` — compare active_arena `eth_call` totals | | T-13 | Full stack swarm (manual) | `START_BOT_SWARM=1 bash scripts/start-qa-local-full-stack.sh --live-sale` + sample `/v1/status/ops` `rpc_metrics` | | T-14 | Clippy + unit | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` | --- ## Test plan — attack, hack, and abuse vectors | Vector | Threat | Test / verification | |--------|--------|---------------------| | **A-1: RPC griefing / rate limit** | Provider returns 429 mid-block | Ingest aborts block tx; retries block; no partial `idx_arena_podium_live` rows (`INV-INDEXER-317`). Simulate with mock provider or timeout. | | **A-2: Malicious log ordering** | Attacker stuffs many `Buy` logs in one block to amplify RPC | Per-block single multicall caps RPC at O(unique reads), not O(logs); benchmark block with N buys shows flat `eth_call` count. | | **A-3: Reorg after snapshot** | Shallow reorg drops block with live podium rows | `rollback_after` clears derived tables; re-ingest reproduces state (`integration_stage2` reorg test). | | **A-4: Wrong selector / uint8 podium getter** | Calling wrong ABI reverts whole batch | Selector unit tests; `verify-podium-live-anvil.sh`; batch must use `podiumEpoch(uint256)` only. | | **A-5: Stale block tag** | Using head instead of log block mis-attributes podium | Plan always uses `BlockId::Number(d.block_number)`; integration fixture asserts `block_number` on stored rows matches log. | | **A-6: Multicall3 missing / malicious** | Deploy absent or wrong contract | Fallback sequential path; Anvil bootstrap script ensures canonical deploy; fail block if both paths fail. | | **A-7: Duplicate log_index / PK clash** | Derivative warbow rows use synthetic `log_index` | Existing `log_index * 10 + i` scheme unchanged; integration asserts no PK violations under batch path. | | **A-8: Ingestion race / double-apply** | Retried block after crash | Transactional per-block ingest unchanged; idempotent `ON CONFLICT DO NOTHING` on warbow inserts preserved. | --- ## Verification criteria (merge gate) | Layer | Command | Required | |-------|---------|----------| | Indexer | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` | PASS | | Podium live | `bash scripts/verify-podium-live-anvil.sh` | PASS | | Wallet profile | `bash scripts/verify-wallet-profile-anvil.sh` | PASS | | RPC metrics | `bash scripts/verify-indexer-rpc-metrics.sh` | PASS | | RPC benchmark | `BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh` | PASS; document active_arena `warbow_score`+`podium_live` reduction in MR | | Docs | Update `design.md`, `invariants-and-business-logic.md`, `rpc-load-benchmark.md` | Included in MR | --- ## Out of scope - Batching `chain_timer` further (done in #307). - Batching `eth_getBlockByNumber` / `eth_getLogs` across blocks. - Frontend RPC changes ([#301](https://gitlab.com/PlasticDigits/yieldomega/-/issues/301)). - `allowFailure: true` per sub-call (explicitly **not** desired — reverts should not occur on valid historical reads). --- ## Suggested labels `indexer`, `performance`, `arena-v2` ## Suggested weight 5
PlasticDigits commented 2026-06-28 05:40:32 +00:00 (Migrated from gitlab.com)

marked as related to #306

marked as related to #306
PlasticDigits commented 2026-06-28 05:40:33 +00:00 (Migrated from gitlab.com)

marked as related to #307

marked as related to #307
PlasticDigits commented 2026-06-28 05:40:33 +00:00 (Migrated from gitlab.com)

marked as related to #140

marked as related to #140
PlasticDigits commented 2026-06-28 05:40:34 +00:00 (Migrated from gitlab.com)

marked as related to #317

marked as related to #317
PlasticDigits commented 2026-06-28 05:40:35 +00:00 (Migrated from gitlab.com)

marked as related to #273

marked as related to #273
PlasticDigits commented 2026-06-28 05:40:35 +00:00 (Migrated from gitlab.com)

marked as related to #254

marked as related to #254
PlasticDigits commented 2026-06-28 06:08:52 +00:00 (Migrated from gitlab.com)

mentioned in merge request !530

mentioned in merge request !530
PlasticDigits commented 2026-06-28 06:15:01 +00:00 (Migrated from gitlab.com)

mentioned in commit e1be645b69

mentioned in commit e1be645b69e7cdf54ea858e4ce785b60a209fc66
PlasticDigits commented 2026-06-28 06:45:42 +00:00 (Migrated from gitlab.com)

Verification — GitLab #356 (per-block Multicall3 ingest batching)

Verified on main at 2026-06-28. Implementation shipped in arena_ingest_rpc.rs; no repo changes required from this verify pass.

Acceptance criteria

ID Result How verified
AC-1 PASS cargo test arena_ingest_rpc — single ArenaBuy plans deduped batch; active-arena benchmark last sample shows eth_call:ingestion (not per-log warbow_score/podium_live)
AC-2 PASS integration_stage2.rs (postgres_stage2_persist_all_events_and_rollback_after + arena_podiums_live_predictions_smoke) — 7/7 integration tests green
AC-3 PASS bash scripts/verify-podium-live-anvil.sh — OK (read_block=25, live_rows=12); bash scripts/verify-wallet-profile-anvil.sh — OK
AC-4 PASS multicall::tests::decode_aggregate3_results_rejects_failed_subcall; ingest uses allowFailure: false via aggregate3_at_block; rollback clears derived tables in integration_stage2
AC-5 PASS INV-INDEXER-356-FALLBACK — execute_sequential_batch in arena_ingest_rpc.rs; manual Anvil-without-Multicall3 smoke confirmed chain_timer + ingest sequential eth_calls (no aggregate3); DB writers identical to batch path (parity tests AC-2)
AC-6 PASS postgres_stage2_persist_all_events_and_rollback_after — reorg rollback clears idx_arena_podium_live / warbow tables
AC-7 PASS BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh — active_arena last sample: eth_call:warbow_score + eth_call:podium_live = 0 vs pre-change baseline 20260615T021945Z 297 (100% reduction, ≥80% target)
AC-8 PASS INV-INDEXER-356-* in docs/testing/invariants-and-business-logic.md §356; docs/indexer/design.md §317; docs/indexer/rpc-load-benchmark.md post-#356 table

Test plan

# Result Command / notes
T-1…T-6 PASS cd indexer && cargo test arena_ingest_rpc — 7 unit tests (buy dedupe, multi-buy, steal, epoch rolled, last-buy, warbow score no-RPC)
T-7 PASS cargo test --test integration_stage2 postgres_stage2_persist_all_events_and_rollback_after
T-8 PASS arena_podiums_live_predictions_smoke (inside integration_stage2 + verify-podium-live-anvil.sh)
T-9 PASS bash scripts/verify-podium-live-anvil.sh
T-10 PASS bash scripts/verify-wallet-profile-anvil.sh
T-11 PASS bash scripts/verify-indexer-rpc-metrics.sh — schema=2.20.0, total_calls=57
T-12 PASS BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh
T-13 SKIP Full stack + bot swarm not run in Cloud verify VM (30+ min, frontend + swarm). Active-arena benchmark (T-12) samples /v1/status/ops rpc_metrics under sustained buys — same signal for ingest RPC attribution
T-14 PASS cd indexer && cargo clippy --all-targets -- -D warnings && cargo test — 89 unit + 7 integration tests

Merge gate

Layer Command Result
Indexer cargo clippy + cargo test PASS
Podium live verify-podium-live-anvil.sh PASS
Wallet profile verify-wallet-profile-anvil.sh PASS
RPC metrics verify-indexer-rpc-metrics.sh PASS
RPC benchmark benchmark-indexer-rpc-anvil.sh (120s) PASS
Docs / invariants INV-INDEXER-356-*, design.md, rpc-load-benchmark.md PASS (already on main)

Attack / abuse vectors (spot-check)

  • A-2 flat eth_call:ingestion per block under N buys — confirmed in benchmark (legacy callers → 0).
  • A-3 reorg rollback — integration_stage2.
  • A-4 podiumEpoch(uint256) selectors — unit tests + podium-live E2E.
  • A-6 sequential fallback path — code + no-Multicall3 smoke (see AC-5).

Closing — all automated acceptance items pass; T-13 deferred to operator manual QA per issue test plan label.

## Verification — GitLab #356 (per-block Multicall3 ingest batching) Verified on `main` at 2026-06-28. Implementation shipped in `arena_ingest_rpc.rs`; no repo changes required from this verify pass. ### Acceptance criteria | ID | Result | How verified | |----|--------|--------------| | **AC-1** | **PASS** | `cargo test arena_ingest_rpc` — single `ArenaBuy` plans deduped batch; active-arena benchmark last sample shows `eth_call:ingestion` (not per-log `warbow_score`/`podium_live`) | | **AC-2** | **PASS** | `integration_stage2.rs` (`postgres_stage2_persist_all_events_and_rollback_after` + `arena_podiums_live_predictions_smoke`) — 7/7 integration tests green | | **AC-3** | **PASS** | `bash scripts/verify-podium-live-anvil.sh` — OK (`read_block=25`, `live_rows=12`); `bash scripts/verify-wallet-profile-anvil.sh` — OK | | **AC-4** | **PASS** | `multicall::tests::decode_aggregate3_results_rejects_failed_subcall`; ingest uses `allowFailure: false` via `aggregate3_at_block`; rollback clears derived tables in `integration_stage2` | | **AC-5** | **PASS** | `INV-INDEXER-356-FALLBACK` — `execute_sequential_batch` in `arena_ingest_rpc.rs`; manual Anvil-without-Multicall3 smoke confirmed chain_timer + ingest sequential `eth_call`s (no `aggregate3`); DB writers identical to batch path (parity tests AC-2) | | **AC-6** | **PASS** | `postgres_stage2_persist_all_events_and_rollback_after` — reorg rollback clears `idx_arena_podium_live` / warbow tables | | **AC-7** | **PASS** | `BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh` — active_arena last sample: `eth_call:warbow_score` + `eth_call:podium_live` = **0** vs pre-change baseline `20260615T021945Z` **297** (**100%** reduction, ≥80% target) | | **AC-8** | **PASS** | `INV-INDEXER-356-*` in `docs/testing/invariants-and-business-logic.md` §356; `docs/indexer/design.md` §317; `docs/indexer/rpc-load-benchmark.md` post-#356 table | ### Test plan | # | Result | Command / notes | |---|--------|-----------------| | T-1…T-6 | **PASS** | `cd indexer && cargo test arena_ingest_rpc` — 7 unit tests (buy dedupe, multi-buy, steal, epoch rolled, last-buy, warbow score no-RPC) | | T-7 | **PASS** | `cargo test --test integration_stage2 postgres_stage2_persist_all_events_and_rollback_after` | | T-8 | **PASS** | `arena_podiums_live_predictions_smoke` (inside `integration_stage2` + `verify-podium-live-anvil.sh`) | | T-9 | **PASS** | `bash scripts/verify-podium-live-anvil.sh` | | T-10 | **PASS** | `bash scripts/verify-wallet-profile-anvil.sh` | | T-11 | **PASS** | `bash scripts/verify-indexer-rpc-metrics.sh` — schema=2.20.0, `total_calls=57` | | T-12 | **PASS** | `BENCHMARK_SCENARIO_SEC=120 bash scripts/benchmark-indexer-rpc-anvil.sh` | | T-13 | **SKIP** | Full stack + bot swarm not run in Cloud verify VM (30+ min, frontend + swarm). Active-arena benchmark (T-12) samples `/v1/status/ops` `rpc_metrics` under sustained buys — same signal for ingest RPC attribution | | T-14 | **PASS** | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` — 89 unit + 7 integration tests | ### Merge gate | Layer | Command | Result | |-------|---------|--------| | Indexer | `cargo clippy` + `cargo test` | **PASS** | | Podium live | `verify-podium-live-anvil.sh` | **PASS** | | Wallet profile | `verify-wallet-profile-anvil.sh` | **PASS** | | RPC metrics | `verify-indexer-rpc-metrics.sh` | **PASS** | | RPC benchmark | `benchmark-indexer-rpc-anvil.sh` (120s) | **PASS** | | Docs / invariants | `INV-INDEXER-356-*`, design.md, rpc-load-benchmark.md | **PASS** (already on main) | ### Attack / abuse vectors (spot-check) - **A-2** flat `eth_call:ingestion` per block under N buys — confirmed in benchmark (legacy callers → 0). - **A-3** reorg rollback — `integration_stage2`. - **A-4** `podiumEpoch(uint256)` selectors — unit tests + podium-live E2E. - **A-6** sequential fallback path — code + no-Multicall3 smoke (see AC-5). Closing — all automated acceptance items pass; T-13 deferred to operator manual QA per issue test plan label.
PlasticDigits (Migrated from gitlab.com) closed this issue 2026-06-28 06:45:43 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
cac-design-issue-544
renovate/configure
No results found.
Labels
Clear labels   
agent:fix_bugfix

   
agent:fix_conflicts

   
agent:fix_security

   
agent:gap_analysis

   
agent:implement

   
agent:open_issues

   
agents

   
agent:security_audit

   
agent:verify
  
area
timecurve

  
arena

   
arena-v2

   
audit

   
block:feature_review

   
block:security

   
bug

   
contracts

   
design

   
devops

   
docs
Documentation

  
documentation

   
enhancement

   
epic
Epic / umbrella issue

  
feature

   
frontend

   
indexer

   
low priority
Low priority, investigate other issues first

  
megaeth

   
operations

   
ops
Deploy and operations

  
performance

   
QA

   
qa checklist

   
referrals

   
removal
Legacy removal

  
security

   
smart-contracts

   
tech-debt

   
testing

   
time-arena
TimeArena v2 revamp

  
timecurve

   
visual-qa
No labels
agent:fix_bugfix
agent:fix_conflicts
agent:fix_security
agent:gap_analysis
agent:implement
agent:open_issues
agents
agent:security_audit
agent:verify
area
timecurve
arena
arena-v2
audit
block:feature_review
block:security
bug
contracts
design
devops
docs
documentation
enhancement
epic
feature
frontend
indexer
low priority
megaeth
operations
ops
performance
QA
qa checklist
referrals
removal
security
smart-contracts
tech-debt
testing
time-arena
timecurve
visual-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
code/yieldomega#356
No description provided.