Expose indexer ingest lag on arena timer and podium APIs #344

New issue

Closed

opened 2026-06-25 20:07:14 +00:00 by PlasticDigits · 5 comments

PlasticDigits commented 2026-06-25 20:07:14 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Parent

Transition Gap Analysis (#342) — P0 #2. Bundles H-I1 (lag visibility), M-I6.

Current codebase

• chain_timer.rs / chain_timer_poll.rs poll latest RPC head via Multicall3 for GET /v1/arena/timers and podium reads — epochs/deadlines can advance before log ingest catches up.
• Event tables advance only to chain_pointer in ingestion.rs.
• /v1/arena/timers and /v1/podiums expose read_block_number (RPC head) but not how far ingest has caught up.
• Public /v1/status omits chain_pointer / last_indexed_at_ms unless ops flag — clients cannot detect ingest lag.

Why needed

After an epoch roll, the UI can show new epoch/deadlines from the head poller before PodiumEpochRolled, live podium rows, or activity reflect the transition. Users need a subtle sync indicator and API consumers need indexed_through_block to avoid misleading “final” displays.

Constraints / guardrails

• Do not break existing API consumers — add fields, do not rename/remove.
• Ops-only fields on /v1/status stay gated; arena endpoints are the primary consumer surface.
• Align field names across timers and podiums responses.

Relevant files

• indexer/src/chain_timer.rs
• indexer/src/chain_timer_poll.rs
• indexer/src/api_arena.rs
• indexer/src/api.rs (/v1/status)
• indexer/src/ingestion.rs (chain pointer)
• frontend/src/lib/indexerApi.ts (types)

Recommended direction

1. Add indexed_head_block (or indexed_through_block) to GET /v1/arena/timers and GET /v1/podiums by reading current chain_pointer alongside head poller snapshot.
2. Optionally include chain_head_block alias if read_block_number naming is inconsistent.
3. Frontend (sibling UX issue): show subtle sync indicator when indexed_head_block < read_block_number.
4. Consider documenting lag semantics in docs/architecture/data-flows.md.

Acceptance criteria

• Timers and podiums JSON include indexed-through block number.
• Field populated correctly when ingest lags behind head poller (reproducible in Anvil test).
• TypeScript types updated in indexerApi.ts.
• No regression in existing verify scripts (verify-podium-timers-anvil.sh, verify-podium-live-anvil.sh).

Test plan

Path	Command
Indexer unit + integration	cd indexer && cargo clippy --all-targets -- -D warnings && cargo test
Integration	Extend integration_stage2.rs or add unit test for API field when pointer < head
Smoke	bash scripts/verify-podium-timers-anvil.sh
Frontend types	cd frontend && npm run typecheck

Attack / abuse test plan

• Stale pointer must not report indexed block higher than actual chain_pointer.
• Field must be monotonic non-decreasing per environment (no rewind without reorg handling).

Verification criteria

• curl timers response shows both RPC head and indexed-through blocks.
• Anvil scenario: roll on-chain, query API before ingest catches up — lag visible.

## Parent [Transition Gap Analysis (#342)](https://gitlab.com/PlasticDigits/yieldomega/-/issues/342) — P0 #2. Bundles **H-I1** (lag visibility), **M-I6**. ## Current codebase - `chain_timer.rs` / `chain_timer_poll.rs` poll latest RPC head via Multicall3 for `GET /v1/arena/timers` and podium reads — epochs/deadlines can advance before log ingest catches up. - Event tables advance only to `chain_pointer` in `ingestion.rs`. - `/v1/arena/timers` and `/v1/podiums` expose `read_block_number` (RPC head) but not how far ingest has caught up. - Public `/v1/status` omits `chain_pointer` / `last_indexed_at_ms` unless ops flag — clients cannot detect ingest lag. ## Why needed After an epoch roll, the UI can show new epoch/deadlines from the head poller before `PodiumEpochRolled`, live podium rows, or activity reflect the transition. Users need a subtle sync indicator and API consumers need `indexed_through_block` to avoid misleading “final” displays. ## Constraints / guardrails - Do not break existing API consumers — add fields, do not rename/remove. - Ops-only fields on `/v1/status` stay gated; arena endpoints are the primary consumer surface. - Align field names across timers and podiums responses. ## Relevant files - `indexer/src/chain_timer.rs` - `indexer/src/chain_timer_poll.rs` - `indexer/src/api_arena.rs` - `indexer/src/api.rs` (`/v1/status`) - `indexer/src/ingestion.rs` (chain pointer) - `frontend/src/lib/indexerApi.ts` (types) ## Recommended direction 1. Add `indexed_head_block` (or `indexed_through_block`) to `GET /v1/arena/timers` and `GET /v1/podiums` by reading current `chain_pointer` alongside head poller snapshot. 2. Optionally include `chain_head_block` alias if `read_block_number` naming is inconsistent. 3. Frontend (sibling UX issue): show subtle sync indicator when `indexed_head_block < read_block_number`. 4. Consider documenting lag semantics in `docs/architecture/data-flows.md`. ## Acceptance criteria - [ ] Timers and podiums JSON include indexed-through block number. - [ ] Field populated correctly when ingest lags behind head poller (reproducible in Anvil test). - [ ] TypeScript types updated in `indexerApi.ts`. - [ ] No regression in existing verify scripts (`verify-podium-timers-anvil.sh`, `verify-podium-live-anvil.sh`). ## Test plan | Path | Command | |------|---------| | Indexer unit + integration | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` | | Integration | Extend `integration_stage2.rs` or add unit test for API field when pointer < head | | Smoke | `bash scripts/verify-podium-timers-anvil.sh` | | Frontend types | `cd frontend && npm run typecheck` | ## Attack / abuse test plan - Stale pointer must not report indexed block higher than actual `chain_pointer`. - Field must be monotonic non-decreasing per environment (no rewind without reorg handling). ## Verification criteria - `curl` timers response shows both RPC head and indexed-through blocks. - Anvil scenario: roll on-chain, query API before ingest catches up — lag visible.

PlasticDigits commented 2026-06-25 20:07:48 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in issue #342

mentioned in issue #342

PlasticDigits commented 2026-06-25 20:55:47 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in merge request !517

mentioned in merge request !517

PlasticDigits commented 2026-06-25 21:53:30 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in commit 2b5d0240a5

mentioned in commit 2b5d0240a524dcc01d41a9fda55d3501635e2180

PlasticDigits commented 2026-06-25 22:01:36 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in merge request !519

mentioned in merge request !519

PlasticDigits commented 2026-06-26 06:43:41 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Verification — #344 Expose indexer ingest lag on arena timer and podium APIs

Result: PASS (all acceptance criteria met on main; no repo changes required).

Acceptance criteria

Criterion	Result	How verified
Timers and podiums JSON include indexed_through_block	PASS	Live Anvil stack (anvil :8547 + indexer :3102): curl /v1/arena/timers and /v1/arena/podiums both return read_block_number + indexed_through_block (e.g. both "12" at sync). x-schema-version: 2.20.0 (≥ 2.19.0).
Field correct when ingest lags behind head poller	PASS	integration_stage2.rs::api_arena_ingest_lag_fields_smoke — mocked head read_block_number=100, chain_pointer=50 → both /v1/arena/timers and /v1/arena/podiums return indexed_through_block="50"; asserts indexed_through_block ≤ read_block_number.
TypeScript types in indexerApi.ts	PASS	cd frontend && npm run typecheck — exit 0. ArenaTimersResponse and ArenaPodiumsResponse include optional indexed_through_block.
No regression in verify scripts	PASS	bash scripts/verify-podium-timers-anvil.sh — OK. bash scripts/verify-podium-live-anvil.sh — OK (read_block=25, live_rows=12).

Test plan (issue body)

Command	Result
cd indexer && cargo clippy --all-targets -- -D warnings && cargo test	PASS (82 unit + 7 integration; note: unset DATABASE_POOL_MAX if set in env — otherwise unrelated parses_and_clamps_high fails)
bash scripts/verify-podium-timers-anvil.sh	PASS
cd frontend && npm run typecheck	PASS

Attack / abuse checks

Check	Result	Evidence
Stale pointer never reports indexed block above actual chain_pointer	PASS	api_arena_ingest_lag_fields_smoke asserts indexed_through_block ≤ read_block_number; field sourced from Postgres chain_pointer in api_arena.rs::indexed_through_block.
Monotonic non-decreasing per environment	PASS (by design)	Ingestion advances chain_pointer only forward per block; reorg path resets via rollback_after (#351). No API path rewinds the field without a reorg.

Docs / invariants

• INV-INDEXER-344-INGEST-LAG documented in docs/testing/invariants-and-business-logic.md.
• Lag semantics in docs/architecture/data-flows.md § ingest lag vs head poller.
• Indexer README schema ≥ 2.19.0 notes indexed_through_block.

Manual verification notes

• curl on synced stack shows both block fields on timers and podiums.
• Lag visibility is reproducible via the integration smoke (preferred over racing live ingest).
• Public /v1/status correctly omits chain_pointer unless ops flag — arena endpoints are the consumer surface per issue constraints.

## Verification — #344 Expose indexer ingest lag on arena timer and podium APIs **Result: PASS** (all acceptance criteria met on `main`; no repo changes required). ### Acceptance criteria | Criterion | Result | How verified | |-----------|--------|--------------| | Timers and podiums JSON include `indexed_through_block` | **PASS** | Live Anvil stack (`anvil :8547` + indexer `:3102`): `curl /v1/arena/timers` and `/v1/arena/podiums` both return `read_block_number` + `indexed_through_block` (e.g. both `"12"` at sync). `x-schema-version: 2.20.0` (≥ 2.19.0). | | Field correct when ingest lags behind head poller | **PASS** | `integration_stage2.rs::api_arena_ingest_lag_fields_smoke` — mocked head `read_block_number=100`, `chain_pointer=50` → both `/v1/arena/timers` and `/v1/arena/podiums` return `indexed_through_block="50"`; asserts `indexed_through_block ≤ read_block_number`. | | TypeScript types in `indexerApi.ts` | **PASS** | `cd frontend && npm run typecheck` — exit 0. `ArenaTimersResponse` and `ArenaPodiumsResponse` include optional `indexed_through_block`. | | No regression in verify scripts | **PASS** | `bash scripts/verify-podium-timers-anvil.sh` — OK. `bash scripts/verify-podium-live-anvil.sh` — OK (`read_block=25`, `live_rows=12`). | ### Test plan (issue body) | Command | Result | |---------|--------| | `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` | **PASS** (82 unit + 7 integration; note: unset `DATABASE_POOL_MAX` if set in env — otherwise unrelated `parses_and_clamps_high` fails) | | `bash scripts/verify-podium-timers-anvil.sh` | **PASS** | | `cd frontend && npm run typecheck` | **PASS** | ### Attack / abuse checks | Check | Result | Evidence | |-------|--------|----------| | Stale pointer never reports indexed block above actual `chain_pointer` | **PASS** | `api_arena_ingest_lag_fields_smoke` asserts `indexed_through_block ≤ read_block_number`; field sourced from Postgres `chain_pointer` in `api_arena.rs::indexed_through_block`. | | Monotonic non-decreasing per environment | **PASS** (by design) | Ingestion advances `chain_pointer` only forward per block; reorg path resets via `rollback_after` ([#351](https://gitlab.com/PlasticDigits/yieldomega/-/issues/351)). No API path rewinds the field without a reorg. | ### Docs / invariants - **`INV-INDEXER-344-INGEST-LAG`** documented in `docs/testing/invariants-and-business-logic.md`. - Lag semantics in `docs/architecture/data-flows.md` § ingest lag vs head poller. - Indexer README schema **≥ 2.19.0** notes `indexed_through_block`. ### Manual verification notes - `curl` on synced stack shows both block fields on timers and podiums. - Lag visibility is reproducible via the integration smoke (preferred over racing live ingest). - Public `/v1/status` correctly omits `chain_pointer` unless ops flag — arena endpoints are the consumer surface per issue constraints.

PlasticDigits (Migrated from gitlab.com) closed this issue 2026-06-26 06:43:42 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

cac-design-issue-544

renovate/configure

No results found.

Labels

Clear labels   

agent:fix_bugfix

  

agent:fix_conflicts

  

agent:fix_security

  

agent:gap_analysis

  

agent:implement

  

agent:open_issues

  

agents

  

agent:security_audit

  

agent:verify

  

area

timecurve

  

arena

  

arena-v2

  

audit

  

block:feature_review

  

block:security

  

bug

  

contracts

  

design

  

devops

  

docs

Documentation

  

documentation

  

enhancement

  

epic

Epic / umbrella issue

  

feature

  

frontend

  

indexer

  

low priority

Low priority, investigate other issues first

  

megaeth

  

operations

  

ops

Deploy and operations

  

performance

  

QA

  

qa checklist

  

referrals

  

removal

Legacy removal

  

security

  

smart-contracts

  

tech-debt

  

testing

  

time-arena

TimeArena v2 revamp

  

timecurve

  

visual-qa

No labels

agent:fix_bugfix

agent:fix_conflicts

agent:fix_security

agent:gap_analysis

agent:implement

agent:open_issues

agents

agent:security_audit

agent:verify

area

timecurve

arena

arena-v2

audit

block:feature_review

block:security

bug

contracts

design

devops

docs

documentation

enhancement

epic

feature

frontend

indexer

low priority

megaeth

operations

ops

performance

QA

qa checklist

referrals

removal

security

smart-contracts

tech-debt

testing

time-arena

timecurve

visual-qa

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

code/yieldomega#344

No description provided.