code/cl8y-dex-terraclassic

Fork 0

Indexer: ingestion hardening — cursor on error, tx pagination, reorg recovery (C1–C3) #236

New issue

Closed

opened 2026-05-31 04:41:17 +00:00 by PlasticDigits · 9 comments

PlasticDigits commented

2026-05-31 04:41:17 +00:00

(Migrated from gitlab.com)

Reference

Gap analysis: gaps/GAP_1780200149.md — findings C1, C2, C3.

Current codebase

The Rust indexer polls Terra Classic LCD, parses txs per block, and persists swaps, candles, limit-order lifecycle, and trader aggregates to Postgres. Progress is tracked as last_indexed_height in indexer_state.

C1 — Cursor advances on processing failure: In indexer/src/indexer/poller.rs, after parser::process_block_txs returns Err, the poller only logs tracing::error! and still calls state::set_last_indexed_height(&pool, height). Failed blocks are never retried and are permanently skipped.

C2 — ≤100 txs/block, no pagination: indexer/src/lcd/mod.rs get_block_txs calls search_txs with page=1, limit=100 and no follow-up pages. Busy blocks silently truncate swaps and limit-order events.

C3 — No automatic reorg handling: The poller is forward-only. Recovery is documented manually in docs/runbooks/indexer-reorg-replay-dedup.md. Swap inserts dedupe on (tx_hash, pair_id) via ON CONFLICT DO NOTHING, but candles, positions, and aggregates use merge/upsert — replay without cleanup can leave inconsistent derived state.

Why this is needed

The indexer is the off-chain source of truth for charts, CG/CMC listings, portfolio, route solving, and trader analytics. Silent block skips, truncated busy blocks, and undetected reorgs produce permanent data loss or corruption that users and integrators cannot distinguish from correct state. This is the top off-chain blocker for trustworthy analytics and listings.

Constraints / guardrails

Do not advance last_indexed_height until block processing succeeds (or define an explicit, documented partial-commit policy — default: no advance on error).
Pagination must respect LCD rate limits; reuse existing endpoint failover/cooldown in LcdClient.
Reorg detection must compare block hash (or trusted canonical tip), not height alone.
Preserve idempotent swap dedup; any rewind must document impact on candles/trader aggregates.
Avoid breaking existing operator runbook without updating docs/runbooks/indexer-reorg-replay-dedup.md.
New behavior must be observable via structured logs/metrics (height, hash, page count, retry count).
No unbounded memory when paginating large blocks.

Relevant files

Area	Path
Poller loop	`indexer/src/indexer/poller.rs`
LCD tx search	`indexer/src/lcd/mod.rs`, `indexer/src/lcd/types.rs`
Block parser	`indexer/src/indexer/parser.rs` (and submodules)
Cursor state	`indexer/src/db/queries/state.rs`
Swap dedup	`indexer/src/db/queries/swap_events.rs`
Runbook	`docs/runbooks/indexer-reorg-replay-dedup.md`
Invariants	`docs/indexer-invariants.md`
Config	`indexer/src/config.rs` (`START_BLOCK`)

Recommended direction

C1: Only call set_last_indexed_height after successful process_block_txs. On error, retry with backoff; after N failures, halt catch-up and surface alert (do not skip). Consider a failed_blocks table or metric for operator visibility.
C2: Loop search_txs with pagination until pagination.total is exhausted or next page is empty. Add integration test with wiremock returning >100 txs for one height.
C3: Store (height, block_hash) checkpoint per indexed block. On each poll, verify parent hash chain; on mismatch, enter rewind mode: stop indexer, roll back cursor to fork point (and document SQL cleanup for derived tables), replay from known-good height. Ship an operator script (scripts/indexer-reorg-recover.sh or similar) automating the runbook where safe.

Acceptance criteria

Block processing error does not advance last_indexed_height.
Blocks with >100 txs ingest all txs (verified against mock LCD with 150+ txs).
Reorg detection identifies hash mismatch and prevents silent forward progress.
Documented automated or semi-automated recovery path replaces manual-only runbook steps where feasible.
Existing swap dedup behavior preserved on replay.
Integration tests cover: cursor-on-failure, multi-page tx fetch, hash mismatch detection.

Test plan — all paths

Path	Test
Happy path	Index N blocks; cursor equals latest; all swaps persisted
LCD fetch failure	Block fetch errors retry; cursor unchanged
Parser/DB failure	`process_block_txs` error; cursor unchanged; retry succeeds
0 txs in block	Cursor advances; no parser call needed
1–100 txs	Single page; all ingested
101–500 txs	Multi-page loop; count matches LCD total
Restart mid-catch-up	Resumes from last committed height
Replay same height	Swaps deduped; no duplicate rows
Reorg same height, new hash	Detection triggers; indexer stops or enters rewind
`START_BLOCK` on fresh DB	Starts at configured height

Run: cd indexer && cargo test --tests -j 1 -- --test-threads=1 (Postgres required).

Test plan — attack / abuse / failure vectors

Vector	Expected behavior
Malformed tx payloads in block	Block fails processing; cursor not advanced; retry or operator alert
LCD returns partial page / wrong total	Detect incomplete ingestion; do not advance
LCD pagination DoS (huge total)	Bounded page loop + timeout; fail safe
Deep reorg	Hash mismatch detected; no silent corruption
Replay attack (re-submit old canonical txs)	Dedup prevents duplicate swaps
DB unavailable mid-block	Transaction rollback; cursor unchanged

Verification criteria

Unit/integration tests pass for poller, LCD pagination, reorg detection.
Manual: inject parser error via test hook; confirm height does not advance.
Manual: wiremock block with 150 txs; DB swap count matches.
Runbook updated and cross-linked from docs/indexer-invariants.md.
No regression in existing 22 indexer integration test files.

## Reference Gap analysis: [`gaps/GAP_1780200149.md`](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/blob/main/gaps/GAP_1780200149.md) — findings **C1**, **C2**, **C3**. ## Current codebase The Rust indexer polls Terra Classic LCD, parses txs per block, and persists swaps, candles, limit-order lifecycle, and trader aggregates to Postgres. Progress is tracked as `last_indexed_height` in `indexer_state`. **C1 — Cursor advances on processing failure:** In `indexer/src/indexer/poller.rs`, after `parser::process_block_txs` returns `Err`, the poller only logs `tracing::error!` and still calls `state::set_last_indexed_height(&pool, height)`. Failed blocks are never retried and are permanently skipped. **C2 — ≤100 txs/block, no pagination:** `indexer/src/lcd/mod.rs` `get_block_txs` calls `search_txs` with `page=1, limit=100` and no follow-up pages. Busy blocks silently truncate swaps and limit-order events. **C3 — No automatic reorg handling:** The poller is forward-only. Recovery is documented manually in `docs/runbooks/indexer-reorg-replay-dedup.md`. Swap inserts dedupe on `(tx_hash, pair_id)` via `ON CONFLICT DO NOTHING`, but candles, positions, and aggregates use merge/upsert — replay without cleanup can leave inconsistent derived state. ## Why this is needed The indexer is the off-chain source of truth for charts, CG/CMC listings, portfolio, route solving, and trader analytics. Silent block skips, truncated busy blocks, and undetected reorgs produce **permanent data loss or corruption** that users and integrators cannot distinguish from correct state. This is the top off-chain blocker for trustworthy analytics and listings. ## Constraints / guardrails - Do **not** advance `last_indexed_height` until block processing succeeds (or define an explicit, documented partial-commit policy — default: no advance on error). - Pagination must respect LCD rate limits; reuse existing endpoint failover/cooldown in `LcdClient`. - Reorg detection must compare **block hash** (or trusted canonical tip), not height alone. - Preserve idempotent swap dedup; any rewind must document impact on candles/trader aggregates. - Avoid breaking existing operator runbook without updating `docs/runbooks/indexer-reorg-replay-dedup.md`. - New behavior must be observable via structured logs/metrics (height, hash, page count, retry count). - No unbounded memory when paginating large blocks. ## Relevant files | Area | Path | |------|------| | Poller loop | `indexer/src/indexer/poller.rs` | | LCD tx search | `indexer/src/lcd/mod.rs`, `indexer/src/lcd/types.rs` | | Block parser | `indexer/src/indexer/parser.rs` (and submodules) | | Cursor state | `indexer/src/db/queries/state.rs` | | Swap dedup | `indexer/src/db/queries/swap_events.rs` | | Runbook | `docs/runbooks/indexer-reorg-replay-dedup.md` | | Invariants | `docs/indexer-invariants.md` | | Config | `indexer/src/config.rs` (`START_BLOCK`) | ## Recommended direction 1. **C1:** Only call `set_last_indexed_height` after successful `process_block_txs`. On error, retry with backoff; after N failures, halt catch-up and surface alert (do not skip). Consider a `failed_blocks` table or metric for operator visibility. 2. **C2:** Loop `search_txs` with pagination until `pagination.total` is exhausted or next page is empty. Add integration test with wiremock returning >100 txs for one height. 3. **C3:** Store `(height, block_hash)` checkpoint per indexed block. On each poll, verify parent hash chain; on mismatch, enter rewind mode: stop indexer, roll back cursor to fork point (and document SQL cleanup for derived tables), replay from known-good height. Ship an operator script (`scripts/indexer-reorg-recover.sh` or similar) automating the runbook where safe. ## Acceptance criteria - [ ] Block processing error does **not** advance `last_indexed_height`. - [ ] Blocks with >100 txs ingest all txs (verified against mock LCD with 150+ txs). - [ ] Reorg detection identifies hash mismatch and prevents silent forward progress. - [ ] Documented automated or semi-automated recovery path replaces manual-only runbook steps where feasible. - [ ] Existing swap dedup behavior preserved on replay. - [ ] Integration tests cover: cursor-on-failure, multi-page tx fetch, hash mismatch detection. ## Test plan — all paths | Path | Test | |------|------| | Happy path | Index N blocks; cursor equals latest; all swaps persisted | | LCD fetch failure | Block fetch errors retry; cursor unchanged | | Parser/DB failure | `process_block_txs` error; cursor unchanged; retry succeeds | | 0 txs in block | Cursor advances; no parser call needed | | 1–100 txs | Single page; all ingested | | 101–500 txs | Multi-page loop; count matches LCD total | | Restart mid-catch-up | Resumes from last committed height | | Replay same height | Swaps deduped; no duplicate rows | | Reorg same height, new hash | Detection triggers; indexer stops or enters rewind | | `START_BLOCK` on fresh DB | Starts at configured height | Run: `cd indexer && cargo test --tests -j 1 -- --test-threads=1` (Postgres required). ## Test plan — attack / abuse / failure vectors | Vector | Expected behavior | |--------|-------------------| | Malformed tx payloads in block | Block fails processing; cursor not advanced; retry or operator alert | | LCD returns partial page / wrong total | Detect incomplete ingestion; do not advance | | LCD pagination DoS (huge total) | Bounded page loop + timeout; fail safe | | Deep reorg | Hash mismatch detected; no silent corruption | | Replay attack (re-submit old canonical txs) | Dedup prevents duplicate swaps | | DB unavailable mid-block | Transaction rollback; cursor unchanged | ## Verification criteria - [ ] Unit/integration tests pass for poller, LCD pagination, reorg detection. - [ ] Manual: inject parser error via test hook; confirm height does not advance. - [ ] Manual: wiremock block with 150 txs; DB swap count matches. - [ ] Runbook updated and cross-linked from `docs/indexer-invariants.md`. - [ ] No regression in existing 22 indexer integration test files.

PlasticDigits commented

2026-05-31 05:07:16 +00:00

(Migrated from gitlab.com)

mentioned in commit b3592e103b

mentioned in commit b3592e103b314c92ad025e3bbbdaab9bad652bac

PlasticDigits commented

2026-05-31 05:07:16 +00:00

(Migrated from gitlab.com)

mentioned in commit 1b9957a0b1

mentioned in commit 1b9957a0b18dfcd2a06009344e79f282acb0a103

PlasticDigits commented

2026-05-31 05:07:22 +00:00

(Migrated from gitlab.com)

Implementation complete (pushed to `main`)

Indexer ingestion hardening for gap findings C1–C3 (#236) is merged to main (b3592e1).

What changed

C1 — Cursor on error

last_indexed_height / last_indexed_block_hash commit only after successful block ingest (block_indexer.rs).
process_block_txs now propagates per-tx errors (no warn-and-skip).
Retries with backoff (BLOCK_PROCESS_MAX_RETRIES, BLOCK_PROCESS_RETRY_BACKOFF_MS); persistent failures recorded in indexer_failed_blocks.

C2 — Tx pagination

LcdClient::get_block_txs paginates until pagination.total is satisfied; rejects incomplete totals; bounded by BLOCK_TX_MAX_PAGES × BLOCK_TX_PAGE_LIMIT.

C3 — Reorg detection

Before each new height, canonical block hash at last committed height is compared to last_indexed_block_hash; mismatch halts the indexer.
Semi-automated recovery: ./scripts/indexer-reorg-recover.sh --height H [--apply].

Docs / agent playbooks

docs/indexer-invariants.md — C1–C3 rows
docs/runbooks/indexer-reorg-replay-dedup.md — updated recovery flow
skills/AGENTS_INDEXER_INGESTION_HARDENING.md — third-party agent playbook

Verification checklist (QA / agents)

cd indexer && cargo test --lib — passes (includes LCD pagination wiremock tests)
cd indexer && cargo test --test indexer_ingestion_hardening -j 1 -- --test-threads=1 — 4/4 pass (Postgres required)
Full suite: cargo test --tests -j 1 -- --test-threads=1 — no regressions in existing indexer integration tests
Migration 20260531120000_indexer_ingestion_hardening.sql applies cleanly (indexer_failed_blocks table)
C1 manual: induce parser/LCD failure at a height → confirm last_indexed_height unchanged and indexer_failed_blocks row exists
C2 manual: wiremock or staging block with 150+ txs → swap/event count matches LCD total
C3 manual: set stale last_indexed_block_hash in DB → indexer halts with reorg log (no forward progress)
./scripts/indexer-reorg-recover.sh --height N dry-run → ./scripts/indexer-reorg-recover.sh --height N --apply → indexer replays from N
Replay same indexed height → swap dedup unchanged (ON CONFLICT DO NOTHING)

Request

@qa-team / QA agents: please run the checklist above on a fresh deploy (LocalTerra or staging) and confirm ingestion behavior under failure and busy-block scenarios. Leave findings on this issue.

Issue remains open until QA sign-off.

## Implementation complete (pushed to `main`) Indexer ingestion hardening for gap findings **C1–C3** ([#236](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/236)) is merged to `main` (`b3592e1`). ### What changed **C1 — Cursor on error** - `last_indexed_height` / `last_indexed_block_hash` commit only after successful block ingest (`block_indexer.rs`). - `process_block_txs` now **propagates** per-tx errors (no warn-and-skip). - Retries with backoff (`BLOCK_PROCESS_MAX_RETRIES`, `BLOCK_PROCESS_RETRY_BACKOFF_MS`); persistent failures recorded in `indexer_failed_blocks`. **C2 — Tx pagination** - `LcdClient::get_block_txs` paginates until `pagination.total` is satisfied; rejects incomplete totals; bounded by `BLOCK_TX_MAX_PAGES` × `BLOCK_TX_PAGE_LIMIT`. **C3 — Reorg detection** - Before each new height, canonical block hash at last committed height is compared to `last_indexed_block_hash`; mismatch halts the indexer. - Semi-automated recovery: `./scripts/indexer-reorg-recover.sh --height H [--apply]`. ### Docs / agent playbooks - [`docs/indexer-invariants.md`](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/blob/main/docs/indexer-invariants.md) — C1–C3 rows - [`docs/runbooks/indexer-reorg-replay-dedup.md`](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/blob/main/docs/runbooks/indexer-reorg-replay-dedup.md) — updated recovery flow - [`skills/AGENTS_INDEXER_INGESTION_HARDENING.md`](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/blob/main/skills/AGENTS_INDEXER_INGESTION_HARDENING.md) — third-party agent playbook ### Verification checklist (QA / agents) - [ ] `cd indexer && cargo test --lib` — passes (includes LCD pagination wiremock tests) - [ ] `cd indexer && cargo test --test indexer_ingestion_hardening -j 1 -- --test-threads=1` — 4/4 pass (Postgres required) - [ ] Full suite: `cargo test --tests -j 1 -- --test-threads=1` — no regressions in existing indexer integration tests - [ ] Migration `20260531120000_indexer_ingestion_hardening.sql` applies cleanly (`indexer_failed_blocks` table) - [ ] **C1 manual:** induce parser/LCD failure at a height → confirm `last_indexed_height` unchanged and `indexer_failed_blocks` row exists - [ ] **C2 manual:** wiremock or staging block with 150+ txs → swap/event count matches LCD total - [ ] **C3 manual:** set stale `last_indexed_block_hash` in DB → indexer halts with reorg log (no forward progress) - [ ] `./scripts/indexer-reorg-recover.sh --height N` dry-run → `./scripts/indexer-reorg-recover.sh --height N --apply` → indexer replays from `N` - [ ] Replay same indexed height → swap dedup unchanged (`ON CONFLICT DO NOTHING`) ### Request **@qa-team / QA agents:** please run the checklist above on a fresh deploy (LocalTerra or staging) and confirm ingestion behavior under failure and busy-block scenarios. Leave findings on this issue. Issue remains **open** until QA sign-off.

PlasticDigits commented

2026-05-31 07:02:53 +00:00

(Migrated from gitlab.com)

QA verification complete — all criteria pass

Verified on branch main @ 841c321 using git worktree verify/issue-236 against LocalTerra + Postgres (dex_indexer / dex_indexer_test).

Environment

LocalTerra: healthy (docker compose)
Postgres: 127.0.0.1:5432 — cl8y_legal / dex_indexer
Indexer API: http://127.0.0.1:3001/health → {"status":"ok"}
Live cursor: last_indexed_height=18040, last_indexed_block_hash set

Automated tests (all pass)

Check	Result
`cargo test --lib`	73/73 pass (includes `lcd::tests::get_block_txs_multi_page`, `get_block_txs_rejects_incomplete_total`)
`cargo test --test indexer_ingestion_hardening -j 1 -- --test-threads=1`	6/6 pass
`cargo test --tests -j 1 -- --test-threads=1`	All 22 integration binaries pass (~105 tests, exit 0)

C1 — cursor_does_not_advance_on_parser_failure: height unchanged after max retries; indexer_failed_blocks row inserted.

C2 — multi_page_block_txs_ingested_count_matches_lcd_total: 155 txs across 2 pages (page_count=2).

C3 — reorg_detection_halts_on_hash_mismatch: BlockIndexError::ReorgDetected on hash mismatch.

Additional block-time invariants: missing_tx_timestamp_uses_block_header_time, invalid_tx_and_header_timestamp_fails_block (cursor unchanged).

Migration & schema

20260531120000_indexer_ingestion_hardening.sql applied in test DB
indexer_failed_blocks table present with expected columns

Operator recovery

./scripts/indexer-reorg-recover.sh --height 18040 dry-run: correct SQL preview (cursor → 18039, hash cleared, indexer_failed_blocks truncated)
Runbook cross-linked from docs/indexer-invariants.md (C1–C3 rows) and skills/AGENTS_INDEXER_INGESTION_HARDENING.md

Acceptance criteria mapping

Block processing error does not advance last_indexed_height (C1 test)
Blocks with >100 txs ingest all txs — wiremock 155 txs (C2 test)
Reorg detection via block hash mismatch halts progress (C3 test)
Semi-automated recovery script + updated runbook
Swap dedup (tx_hash, pair_id) ON CONFLICT DO NOTHING preserved
Integration tests cover cursor-on-failure, multi-page fetch, hash mismatch
No regression in existing 22 indexer integration test files

Manual verification checklist (for operators)

Induce parser/LCD failure → cursor unchanged + indexer_failed_blocks row (integration test)
Wiremock 150+ txs → count matches LCD total (155 txs, 2 pages)
Stale hash → reorg halt (integration test; live indexer not disturbed)
./scripts/indexer-reorg-recover.sh --height N dry-run produces expected SQL

No bugs found. No code changes required.

Closing issue — ingestion hardening C1–C3 verified.

## QA verification complete — all criteria pass Verified on branch `main` @ `841c321` using git worktree `verify/issue-236` against LocalTerra + Postgres (`dex_indexer` / `dex_indexer_test`). ### Environment - LocalTerra: healthy (docker compose) - Postgres: `127.0.0.1:5432` — `cl8y_legal` / `dex_indexer` - Indexer API: `http://127.0.0.1:3001/health` → `{"status":"ok"}` - Live cursor: `last_indexed_height=18040`, `last_indexed_block_hash` set ### Automated tests (all pass) | Check | Result | |-------|--------| | `cargo test --lib` | **73/73** pass (includes `lcd::tests::get_block_txs_multi_page`, `get_block_txs_rejects_incomplete_total`) | | `cargo test --test indexer_ingestion_hardening -j 1 -- --test-threads=1` | **6/6** pass | | `cargo test --tests -j 1 -- --test-threads=1` | **All 22 integration binaries pass** (~105 tests, exit 0) | **C1** — `cursor_does_not_advance_on_parser_failure`: height unchanged after max retries; `indexer_failed_blocks` row inserted. **C2** — `multi_page_block_txs_ingested_count_matches_lcd_total`: 155 txs across 2 pages (`page_count=2`). **C3** — `reorg_detection_halts_on_hash_mismatch`: `BlockIndexError::ReorgDetected` on hash mismatch. Additional block-time invariants: `missing_tx_timestamp_uses_block_header_time`, `invalid_tx_and_header_timestamp_fails_block` (cursor unchanged). ### Migration & schema - `20260531120000_indexer_ingestion_hardening.sql` applied in test DB - `indexer_failed_blocks` table present with expected columns ### Operator recovery - `./scripts/indexer-reorg-recover.sh --height 18040` dry-run: correct SQL preview (cursor → 18039, hash cleared, `indexer_failed_blocks` truncated) - Runbook cross-linked from `docs/indexer-invariants.md` (C1–C3 rows) and `skills/AGENTS_INDEXER_INGESTION_HARDENING.md` ### Acceptance criteria mapping - [x] Block processing error does **not** advance `last_indexed_height` (C1 test) - [x] Blocks with >100 txs ingest all txs — wiremock 155 txs (C2 test) - [x] Reorg detection via block hash mismatch halts progress (C3 test) - [x] Semi-automated recovery script + updated runbook - [x] Swap dedup `(tx_hash, pair_id) ON CONFLICT DO NOTHING` preserved - [x] Integration tests cover cursor-on-failure, multi-page fetch, hash mismatch - [x] No regression in existing 22 indexer integration test files ### Manual verification checklist (for operators) - [x] Induce parser/LCD failure → cursor unchanged + `indexer_failed_blocks` row (integration test) - [x] Wiremock 150+ txs → count matches LCD total (155 txs, 2 pages) - [x] Stale hash → reorg halt (integration test; live indexer not disturbed) - [x] `./scripts/indexer-reorg-recover.sh --height N` dry-run produces expected SQL **No bugs found. No code changes required.** Closing issue — ingestion hardening C1–C3 verified.

PlasticDigits (Migrated from gitlab.com) closed this issue

2026-05-31 07:02:56 +00:00

Brouie commented

2026-06-04 02:35:50 +00:00

(Migrated from gitlab.com)

mentioned in issue #292