Indexer: automate reorg recovery alerting and replay #362

Closed
opened 2026-06-12 05:05:43 +00:00 by PlasticDigits · 17 comments
PlasticDigits commented 2026-06-12 05:05:43 +00:00 (Migrated from gitlab.com)

Parent

Gap analysis follow-up from GitLab #361 — register item C3 (reorg handling).

Current codebase

  • The indexer poller stores last_indexed_height + last_indexed_block_hash and halts on hash mismatch before advancing (indexer/src/indexer/poller.rs, block_indexer::verify_checkpoint_unchanged).
  • Reorg during block processing also halts with cursor unchanged (BlockIndexError::ReorgDetected).
  • Semi-manual recovery exists: scripts/indexer-reorg-recover.sh + runbook docs/runbooks/indexer-reorg-replay-dedup.md.
  • Swap dedup is safe on replay (ON CONFLICT DO NOTHING on (tx_hash, pair_id, swap_index)); candles/positions/aggregates may need cleanup for heights ≥ fork point.

Why needed

Today recovery is detect-and-halt + operator runbook. Terra Classic reorgs are rare but possible; manual SQL/script steps increase downtime and error risk for derived tables (candles, trader aggregates, volume rollups).

Constraints / guardrails

  • Do not silently skip blocks or advance cursor on hash mismatch.
  • Prefer idempotent replay; document when derived-table SQL cleanup is required.
  • No hosted CI gate required (parent #361 rejected unified MR CI — scope is operator tooling / optional automation).
  • Keep compatibility with existing indexer_state schema and START_BLOCK bootstrap.

Relevant files

  • indexer/src/indexer/poller.rs
  • indexer/src/indexer/block_indexer.rs
  • indexer/src/db/queries/state.rs
  • scripts/indexer-reorg-recover.sh
  • docs/runbooks/indexer-reorg-replay-dedup.md
  • docs/templates/incident-dex-indexer.md

Pick one or combine (ADR if behavior changes):

  1. Operator alert: structured metric/log + webhook/PagerDuty hook when reorg halt fires (height, stored vs canonical hash).
  2. One-click replay job: extend indexer-reorg-recover.sh (or a make indexer-reorg-recover) to optionally truncate derived rows for height >= H with dry-run preview, then restart indexer.
  3. Limited auto-replay: auto-rewind cursor to last common ancestor when mismatch depth ≤ N blocks and derived cleanup script succeeds (feature-flagged, default off).

Document the chosen path in the runbook and incident template.

Acceptance criteria

  • Reorg halt emits an operator-actionable signal (not only tracing::error to stdout).
  • Documented recovery path from halt → healthy catch-up without manual guesswork on cursor height.
  • Dry-run mode shows SQL / row-impact preview before destructive steps.
  • Replay over already-indexed swaps does not duplicate swap rows.
  • Runbook updated with step-by-step for shallow vs deep reorg.

Test plan

Path Command / action
Indexer lib cd indexer && cargo test --lib
Integration make setup-indexer-postgres && make test-indexer-integration (poller/reorg tests if added)
Script dry-run ./scripts/indexer-reorg-recover.sh --height <H> (no --apply)
Manual LocalTerra Induce hash mismatch in dev DB + verify recovery script + indexer catch-up

Attack / abuse test plan

  • Recovery tooling must require explicit --apply (or equivalent) before mutating indexer_state / truncating tables.
  • Auto-replay (if implemented) must cap rewind depth and refuse when fork depth exceeds threshold.
  • No API endpoint that rewinds indexer state without auth (keep CLI/operator-only).

Verification criteria

  • Operator can recover from a simulated reorg halt using documented steps in <30 min on QA Postgres.
  • Post-recovery: last_indexed_height advances, API pair/candle endpoints consistent with LCD tip within expected lag.
## Parent Gap analysis follow-up from [GitLab #361](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/361) — register item **C3** (reorg handling). ## Current codebase - The indexer poller stores `last_indexed_height` + `last_indexed_block_hash` and **halts** on hash mismatch before advancing (`indexer/src/indexer/poller.rs`, `block_indexer::verify_checkpoint_unchanged`). - Reorg during block processing also halts with cursor unchanged (`BlockIndexError::ReorgDetected`). - Semi-manual recovery exists: `scripts/indexer-reorg-recover.sh` + runbook [`docs/runbooks/indexer-reorg-replay-dedup.md`](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/blob/main/docs/runbooks/indexer-reorg-replay-dedup.md). - Swap dedup is safe on replay (`ON CONFLICT DO NOTHING` on `(tx_hash, pair_id, swap_index)`); candles/positions/aggregates may need cleanup for heights ≥ fork point. ## Why needed Today recovery is **detect-and-halt + operator runbook**. Terra Classic reorgs are rare but possible; manual SQL/script steps increase downtime and error risk for derived tables (candles, trader aggregates, volume rollups). ## Constraints / guardrails - Do **not** silently skip blocks or advance cursor on hash mismatch. - Prefer idempotent replay; document when derived-table SQL cleanup is required. - No hosted CI gate required (parent #361 rejected unified MR CI — scope is operator tooling / optional automation). - Keep compatibility with existing `indexer_state` schema and `START_BLOCK` bootstrap. ## Relevant files - `indexer/src/indexer/poller.rs` - `indexer/src/indexer/block_indexer.rs` - `indexer/src/db/queries/state.rs` - `scripts/indexer-reorg-recover.sh` - `docs/runbooks/indexer-reorg-replay-dedup.md` - `docs/templates/incident-dex-indexer.md` ## Recommended direction Pick one or combine (ADR if behavior changes): 1. **Operator alert**: structured metric/log + webhook/PagerDuty hook when reorg halt fires (height, stored vs canonical hash). 2. **One-click replay job**: extend `indexer-reorg-recover.sh` (or a `make indexer-reorg-recover`) to optionally truncate derived rows for `height >= H` with dry-run preview, then restart indexer. 3. **Limited auto-replay**: auto-rewind cursor to last common ancestor when mismatch depth ≤ N blocks and derived cleanup script succeeds (feature-flagged, default off). Document the chosen path in the runbook and incident template. ## Acceptance criteria - [ ] Reorg halt emits an operator-actionable signal (not only `tracing::error` to stdout). - [ ] Documented recovery path from halt → healthy catch-up without manual guesswork on cursor height. - [ ] Dry-run mode shows SQL / row-impact preview before destructive steps. - [ ] Replay over already-indexed swaps does not duplicate swap rows. - [ ] Runbook updated with step-by-step for shallow vs deep reorg. ## Test plan | Path | Command / action | |------|------------------| | Indexer lib | `cd indexer && cargo test --lib` | | Integration | `make setup-indexer-postgres && make test-indexer-integration` (poller/reorg tests if added) | | Script dry-run | `./scripts/indexer-reorg-recover.sh --height <H>` (no `--apply`) | | Manual LocalTerra | Induce hash mismatch in dev DB + verify recovery script + indexer catch-up | ## Attack / abuse test plan - Recovery tooling must require explicit `--apply` (or equivalent) before mutating `indexer_state` / truncating tables. - Auto-replay (if implemented) must cap rewind depth and refuse when fork depth exceeds threshold. - No API endpoint that rewinds indexer state without auth (keep CLI/operator-only). ## Verification criteria - Operator can recover from a simulated reorg halt using documented steps in &lt;30 min on QA Postgres. - Post-recovery: `last_indexed_height` advances, API pair/candle endpoints consistent with LCD tip within expected lag.
PlasticDigits commented 2026-06-12 05:06:04 +00:00 (Migrated from gitlab.com)

mentioned in issue #361

mentioned in issue #361
PlasticDigits commented 2026-06-12 05:13:47 +00:00 (Migrated from gitlab.com)

mentioned in commit 9f6ae80be4

mentioned in commit 9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b
PlasticDigits commented 2026-06-12 05:14:08 +00:00 (Migrated from gitlab.com)

mentioned in merge request !872

mentioned in merge request !872
ghost1 commented 2026-06-12 05:18:17 +00:00 (Migrated from gitlab.com)

mentioned in commit c5c2ac34cb

mentioned in commit c5c2ac34cbe84f04dc4ec1339984ae199507bdc2
PlasticDigits commented 2026-06-12 07:55:15 +00:00 (Migrated from gitlab.com)

mentioned in commit ee25e50e23

mentioned in commit ee25e50e23ac2a6d86562bc30882bd45c4b3c74f
PlasticDigits commented 2026-06-12 08:18:13 +00:00 (Migrated from gitlab.com)

mentioned in commit 8a92d56452

mentioned in commit 8a92d56452d1a75a36685fe839407905e4d6348c
PlasticDigits commented 2026-06-12 08:18:19 +00:00 (Migrated from gitlab.com)

mentioned in merge request !883

mentioned in merge request !883
ghost1 commented 2026-06-12 08:21:22 +00:00 (Migrated from gitlab.com)

mentioned in commit ae66cdddca

mentioned in commit ae66cdddca09d5c9a9ca1d93dc5af1f304ecb097
PlasticDigits commented 2026-06-12 08:34:47 +00:00 (Migrated from gitlab.com)

mentioned in merge request !882

mentioned in merge request !882
PlasticDigits commented 2026-06-12 11:14:02 +00:00 (Migrated from gitlab.com)

mentioned in commit 0cf04e530e

mentioned in commit 0cf04e530e17ab1c96866a199dec962d92b01c45
PlasticDigits commented 2026-06-12 12:03:54 +00:00 (Migrated from gitlab.com)

mentioned in commit f784f944fc

mentioned in commit f784f944fc0bc4b6c9965d2963f33a330f07a46e
PlasticDigits commented 2026-06-12 12:04:17 +00:00 (Migrated from gitlab.com)

mentioned in merge request !889

mentioned in merge request !889
PlasticDigits commented 2026-06-13 02:44:37 +00:00 (Migrated from gitlab.com)

mentioned in commit aa6f387f9f

mentioned in commit aa6f387f9f4570042c3f7ddb6c62df5de1c884fb
PlasticDigits commented 2026-06-13 02:48:28 +00:00 (Migrated from gitlab.com)

mentioned in commit 52748e1f1d

mentioned in commit 52748e1f1d0e624e831db2d09da07fe0ea27058e
PlasticDigits commented 2026-06-13 02:56:17 +00:00 (Migrated from gitlab.com)

Verification report — #362 (Indexer: automate reorg recovery alerting and replay)

Branch / commit: main @ 69c0aa1
Verifier: Cloud Agent QA
Outcome: FAIL — acceptance criteria not met; issue remains open for implementation.

Acceptance criteria

Criterion Result Evidence
Reorg halt emits operator-actionable signal (not only tracing::error to stdout) FAIL indexer/src/indexer/poller.rs logs tracing::error! with structured fields (last_indexed, stored_hash, canonical_hash) and returns Err. No Prometheus counter, webhook/PagerDuty hook, or dedicated alert env var. indexer/src/config.rs has no reorg-alert settings. docs/operator-secrets.md still documents tracing-only observability (#200).
Documented recovery path from halt → healthy catch-up without manual guesswork on cursor height FAIL Baseline #236 runbook exists (docs/runbooks/indexer-reorg-replay-dedup.md) with semi-automated + deep-reorg sections, but no #362 enhancements: no fork-height discovery steps, no make indexer-reorg-recover target, no incident-template alert playbook. Operator must still infer fork point H manually.
Dry-run mode shows SQL / row-impact preview before destructive steps FAIL ./scripts/indexer-reorg-recover.sh --height 1234567 (no --apply) prints cursor-reset SQL only. No SELECT COUNT(*) preview for candles, positions, trader aggregates, or volume rollups at height >= H.
Replay over already-indexed swaps does not duplicate swap rows PASS swap_events unique on (tx_hash, pair_id, swap_index) with ON CONFLICT DO NOTHING (indexer/src/db/queries/swap_events.rs); parser short-circuits via trade_exists (indexer/src/indexer/parser.rs:383). Migration 20260605000000_swap_events_per_tx_pair_swap_index.sql.
Runbook updated with step-by-step for shallow vs deep reorg FAIL Runbook has "Semi-automated recovery" and "Manual recovery (deep reorg)" headings but lacks explicit shallow-vs-deep decision tree, derived-table cleanup SQL examples, or post-recovery validation checklist required by #362.

Test plan

Path Result Command / notes
Indexer lib PASS cd indexer && cargo test --lib → 135 passed
Integration PASS make setup-indexer-postgres && make test-indexer-integration → all suites passed
Reorg integration PASS cargo test --test indexer_ingestion_hardening reorg → reorg_detection_halts_on_hash_mismatch ok
Script dry-run (SQL preview) PASS (partial) ./scripts/indexer-reorg-recover.sh --height 1234567 → SQL preview, no DB mutation (confirmed height stayed 700 on dex_indexer_test after dry-run)
Script dry-run (row-impact preview) FAIL No row counts emitted (see above)
Manual LocalTerra reorg simulation + catch-up FAIL make has-localterra → not running; full E2E recovery not executed (~15 min provision skipped per verify scope; blocked without chain)
QA Postgres simulated recovery (<30 min) FAIL Cursor-reset dry-run verified; end-to-end halt → --apply → indexer catch-up → API/LCD consistency not run (no indexer process + no LocalTerra LCD)

Attack / abuse guardrails

Check Result Notes
Destructive steps require explicit --apply PASS Default is dry-run; --apply required to execute SQL
Auto-replay depth cap (if implemented) N/A No auto-replay feature present
No unauthenticated API to rewind indexer state PASS Recovery is CLI/operator-only (scripts/indexer-reorg-recover.sh)

What exists today (#236 baseline)

  • Reorg hash guard halts poller (verify_checkpoint_unchanged, BlockIndexError::ReorgDetected).
  • Semi-manual recovery script + runbook from #236.
  • Swap dedup safe on replay.
  1. Add operator alert path: structured log field set + optional webhook URL env (or reintroduce METRICS_BIND counter indexer_reorg_halt_total{height=…}).
  2. Extend indexer-reorg-recover.sh with --preview-derived row counts and optional derived-table cleanup SQL; add make indexer-reorg-recover.
  3. Expand runbook with shallow (≤N blocks, script-only) vs deep (snapshot restore / manual SQL) decision tree and post-recovery validation (last_indexed_height advancing, pair/candle API vs LCD tip).
  4. Update docs/templates/incident-dex-indexer.md with reorg-halt triage checklist.
## Verification report — #362 (Indexer: automate reorg recovery alerting and replay) **Branch / commit:** `main` @ `69c0aa1` **Verifier:** Cloud Agent QA **Outcome:** **FAIL** — acceptance criteria not met; issue remains open for implementation. ### Acceptance criteria | Criterion | Result | Evidence | |-----------|--------|----------| | Reorg halt emits operator-actionable signal (not only `tracing::error` to stdout) | **FAIL** | `indexer/src/indexer/poller.rs` logs `tracing::error!` with structured fields (`last_indexed`, `stored_hash`, `canonical_hash`) and returns `Err`. No Prometheus counter, webhook/PagerDuty hook, or dedicated alert env var. `indexer/src/config.rs` has no reorg-alert settings. `docs/operator-secrets.md` still documents tracing-only observability (#200). | | Documented recovery path from halt → healthy catch-up without manual guesswork on cursor height | **FAIL** | Baseline #236 runbook exists (`docs/runbooks/indexer-reorg-replay-dedup.md`) with semi-automated + deep-reorg sections, but no #362 enhancements: no fork-height discovery steps, no `make indexer-reorg-recover` target, no incident-template alert playbook. Operator must still infer fork point `H` manually. | | Dry-run mode shows SQL / **row-impact preview** before destructive steps | **FAIL** | `./scripts/indexer-reorg-recover.sh --height 1234567` (no `--apply`) prints cursor-reset SQL only. No `SELECT COUNT(*)` preview for candles, positions, trader aggregates, or volume rollups at `height >= H`. | | Replay over already-indexed swaps does not duplicate swap rows | **PASS** | `swap_events` unique on `(tx_hash, pair_id, swap_index)` with `ON CONFLICT DO NOTHING` (`indexer/src/db/queries/swap_events.rs`); parser short-circuits via `trade_exists` (`indexer/src/indexer/parser.rs:383`). Migration `20260605000000_swap_events_per_tx_pair_swap_index.sql`. | | Runbook updated with step-by-step for **shallow vs deep reorg** | **FAIL** | Runbook has "Semi-automated recovery" and "Manual recovery (deep reorg)" headings but lacks explicit shallow-vs-deep decision tree, derived-table cleanup SQL examples, or post-recovery validation checklist required by #362. | ### Test plan | Path | Result | Command / notes | |------|--------|-----------------| | Indexer lib | **PASS** | `cd indexer && cargo test --lib` → 135 passed | | Integration | **PASS** | `make setup-indexer-postgres && make test-indexer-integration` → all suites passed | | Reorg integration | **PASS** | `cargo test --test indexer_ingestion_hardening reorg` → `reorg_detection_halts_on_hash_mismatch` ok | | Script dry-run (SQL preview) | **PASS** (partial) | `./scripts/indexer-reorg-recover.sh --height 1234567` → SQL preview, no DB mutation (confirmed height stayed `700` on `dex_indexer_test` after dry-run) | | Script dry-run (row-impact preview) | **FAIL** | No row counts emitted (see above) | | Manual LocalTerra reorg simulation + catch-up | **FAIL** | `make has-localterra` → not running; full E2E recovery not executed (~15 min provision skipped per verify scope; blocked without chain) | | QA Postgres simulated recovery (<30 min) | **FAIL** | Cursor-reset dry-run verified; end-to-end halt → `--apply` → indexer catch-up → API/LCD consistency not run (no indexer process + no LocalTerra LCD) | ### Attack / abuse guardrails | Check | Result | Notes | |-------|--------|-------| | Destructive steps require explicit `--apply` | **PASS** | Default is dry-run; `--apply` required to execute SQL | | Auto-replay depth cap (if implemented) | **N/A** | No auto-replay feature present | | No unauthenticated API to rewind indexer state | **PASS** | Recovery is CLI/operator-only (`scripts/indexer-reorg-recover.sh`) | ### What exists today (#236 baseline) - Reorg hash guard halts poller (`verify_checkpoint_unchanged`, `BlockIndexError::ReorgDetected`). - Semi-manual recovery script + runbook from #236. - Swap dedup safe on replay. ### Recommended follow-ups for implementer 1. Add operator alert path: structured log field set + optional webhook URL env (or reintroduce `METRICS_BIND` counter `indexer_reorg_halt_total{height=…}`). 2. Extend `indexer-reorg-recover.sh` with `--preview-derived` row counts and optional derived-table cleanup SQL; add `make indexer-reorg-recover`. 3. Expand runbook with shallow (≤N blocks, script-only) vs deep (snapshot restore / manual SQL) decision tree and post-recovery validation (`last_indexed_height` advancing, pair/candle API vs LCD tip). 4. Update `docs/templates/incident-dex-indexer.md` with reorg-halt triage checklist.
Brouie commented 2026-06-13 03:07:39 +00:00 (Migrated from gitlab.com)

Verified on my side on main 87b6f9a. All five ACs covered:

  • Operator-actionable signal (not just stderr): on reorg halt the indexer emits a structured tracing event (event=indexer_reorg_halt with height, stored vs canonical hash, recovery_fork_height, recovery_command, runbook), a machine-parseable INDEXER_REORG_HALT JSON line, and optionally POSTs to REORG_ALERT_WEBHOOK_URL. The webhook delivery integration test passes.
  • Recovery path without cursor guesswork: the alert carries the exact recovery_command, and the runbook documents halt→healthy end to end.
  • Dry-run preview before anything destructive: indexer-reorg-recover.sh defaults to a dry run — shows the current cursor, a per-table row-impact preview, and the cursor-reset SQL, and refuses to mutate without --apply. DATABASE_URL is redacted in the output.
  • Replay doesn't duplicate swaps: cursor-only replay keeps swap_events (ON CONFLICT dedup), and --cleanup-derived is the explicit path for a true reorg where canonical txs differ.
  • Runbook covers shallow vs deep: separate sections for 1–few blocks (cleanup-derived steps) and deep/uncertain fork (snapshot restore), plus a verification table.

Ran: test-indexer-reorg-recover.sh dry-run smoke (PASS), a live dry-run preview, lib tests (recovery_command_matches_fork_height, webhook_payload_serializes_event_type), and the reorg_alert_webhook integration test — all green.

One note: I didn't run a live --apply replay this pass — recent blocks are empty so it wouldn't even exercise swap dedup, and the dedup itself was already proven in an earlier replay (swap_events held at 119→119, 0 dups) with the ON CONFLICT invariant unchanged. Happy to run a live cursor-rewind replay if you want fresh evidence.

Good to close from my side. @PlasticDigits

Verified on my side on main 87b6f9a. All five ACs covered: - Operator-actionable signal (not just stderr): on reorg halt the indexer emits a structured tracing event (event=indexer_reorg_halt with height, stored vs canonical hash, recovery_fork_height, recovery_command, runbook), a machine-parseable INDEXER_REORG_HALT JSON line, and optionally POSTs to REORG_ALERT_WEBHOOK_URL. The webhook delivery integration test passes. - Recovery path without cursor guesswork: the alert carries the exact recovery_command, and the runbook documents halt→healthy end to end. - Dry-run preview before anything destructive: indexer-reorg-recover.sh defaults to a dry run — shows the current cursor, a per-table row-impact preview, and the cursor-reset SQL, and refuses to mutate without --apply. DATABASE_URL is redacted in the output. - Replay doesn't duplicate swaps: cursor-only replay keeps swap_events (ON CONFLICT dedup), and --cleanup-derived is the explicit path for a true reorg where canonical txs differ. - Runbook covers shallow vs deep: separate sections for 1–few blocks (cleanup-derived steps) and deep/uncertain fork (snapshot restore), plus a verification table. Ran: test-indexer-reorg-recover.sh dry-run smoke (PASS), a live dry-run preview, lib tests (recovery_command_matches_fork_height, webhook_payload_serializes_event_type), and the reorg_alert_webhook integration test — all green. One note: I didn't run a live --apply replay this pass — recent blocks are empty so it wouldn't even exercise swap dedup, and the dedup itself was already proven in an earlier replay (swap_events held at 119→119, 0 dups) with the ON CONFLICT invariant unchanged. Happy to run a live cursor-rewind replay if you want fresh evidence. Good to close from my side. @PlasticDigits
PlasticDigits (Migrated from gitlab.com) closed this issue 2026-06-13 03:50:16 +00:00
Brouie commented 2026-06-29 14:55:23 +00:00 (Migrated from gitlab.com)

mentioned in issue #337

mentioned in issue #337
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
code/cl8y-dex-terraclassic#362
No description provided.