Indexer: automate reorg recovery alerting and replay #362
Labels
No labels
agent:fix_bugfix
agent:fix_conflicts
agent:fix_security
agent:gap_analysis
agent:implement
agent:implement
agent:implement
agent:open_issues
agent:ready
agent:research
agent:security_audit
agent:verify
architecture
backend
blocker:hybrid
blocker:launch
blocker:limit-orders
blocker:v2
block:log_only
block:security
bug
ci
contracts
correctness
deploy
dev
devops
docs
documentation
duplicate
e2e
enhancement
epic
feature
frontend
functional-completion
gas
good first issue
governance
help wanted
high-risk
hooks
hybrid
indexer
infra
infrastructure
integrators
invalid
launch-blocker
limit-orders
localnet
localterra
low priority
missing-implementation
needs-design
ops
performance
priority
high
priority
medium
product
qa
QA
question
ready
ready
research
scripts
security
security-hardening
smartcontracts
tech-debt
testing
ux
UX
v2
verification
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
code/cl8y-dex-terraclassic#362
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Parent
Gap analysis follow-up from GitLab #361 — register item C3 (reorg handling).
Current codebase
last_indexed_height+last_indexed_block_hashand halts on hash mismatch before advancing (indexer/src/indexer/poller.rs,block_indexer::verify_checkpoint_unchanged).BlockIndexError::ReorgDetected).scripts/indexer-reorg-recover.sh+ runbookdocs/runbooks/indexer-reorg-replay-dedup.md.ON CONFLICT DO NOTHINGon(tx_hash, pair_id, swap_index)); candles/positions/aggregates may need cleanup for heights ≥ fork point.Why needed
Today recovery is detect-and-halt + operator runbook. Terra Classic reorgs are rare but possible; manual SQL/script steps increase downtime and error risk for derived tables (candles, trader aggregates, volume rollups).
Constraints / guardrails
indexer_stateschema andSTART_BLOCKbootstrap.Relevant files
indexer/src/indexer/poller.rsindexer/src/indexer/block_indexer.rsindexer/src/db/queries/state.rsscripts/indexer-reorg-recover.shdocs/runbooks/indexer-reorg-replay-dedup.mddocs/templates/incident-dex-indexer.mdRecommended direction
Pick one or combine (ADR if behavior changes):
indexer-reorg-recover.sh(or amake indexer-reorg-recover) to optionally truncate derived rows forheight >= Hwith dry-run preview, then restart indexer.Document the chosen path in the runbook and incident template.
Acceptance criteria
tracing::errorto stdout).Test plan
cd indexer && cargo test --libmake setup-indexer-postgres && make test-indexer-integration(poller/reorg tests if added)./scripts/indexer-reorg-recover.sh --height <H>(no--apply)Attack / abuse test plan
--apply(or equivalent) before mutatingindexer_state/ truncating tables.Verification criteria
last_indexed_heightadvances, API pair/candle endpoints consistent with LCD tip within expected lag.mentioned in issue #361
mentioned in commit
9f6ae80be4mentioned in merge request !872
mentioned in commit
c5c2ac34cbmentioned in commit
ee25e50e23mentioned in commit
8a92d56452mentioned in merge request !883
mentioned in commit
ae66cdddcamentioned in merge request !882
mentioned in commit
0cf04e530ementioned in commit
f784f944fcmentioned in merge request !889
mentioned in commit
aa6f387f9fmentioned in commit
52748e1f1dVerification report — #362 (Indexer: automate reorg recovery alerting and replay)
Branch / commit:
main@69c0aa1Verifier: Cloud Agent QA
Outcome: FAIL — acceptance criteria not met; issue remains open for implementation.
Acceptance criteria
tracing::errorto stdout)indexer/src/indexer/poller.rslogstracing::error!with structured fields (last_indexed,stored_hash,canonical_hash) and returnsErr. No Prometheus counter, webhook/PagerDuty hook, or dedicated alert env var.indexer/src/config.rshas no reorg-alert settings.docs/operator-secrets.mdstill documents tracing-only observability (#200).docs/runbooks/indexer-reorg-replay-dedup.md) with semi-automated + deep-reorg sections, but no #362 enhancements: no fork-height discovery steps, nomake indexer-reorg-recovertarget, no incident-template alert playbook. Operator must still infer fork pointHmanually../scripts/indexer-reorg-recover.sh --height 1234567(no--apply) prints cursor-reset SQL only. NoSELECT COUNT(*)preview for candles, positions, trader aggregates, or volume rollups atheight >= H.swap_eventsunique on(tx_hash, pair_id, swap_index)withON CONFLICT DO NOTHING(indexer/src/db/queries/swap_events.rs); parser short-circuits viatrade_exists(indexer/src/indexer/parser.rs:383). Migration20260605000000_swap_events_per_tx_pair_swap_index.sql.Test plan
cd indexer && cargo test --lib→ 135 passedmake setup-indexer-postgres && make test-indexer-integration→ all suites passedcargo test --test indexer_ingestion_hardening reorg→reorg_detection_halts_on_hash_mismatchok./scripts/indexer-reorg-recover.sh --height 1234567→ SQL preview, no DB mutation (confirmed height stayed700ondex_indexer_testafter dry-run)make has-localterra→ not running; full E2E recovery not executed (~15 min provision skipped per verify scope; blocked without chain)--apply→ indexer catch-up → API/LCD consistency not run (no indexer process + no LocalTerra LCD)Attack / abuse guardrails
--apply--applyrequired to execute SQLscripts/indexer-reorg-recover.sh)What exists today (#236 baseline)
verify_checkpoint_unchanged,BlockIndexError::ReorgDetected).Recommended follow-ups for implementer
METRICS_BINDcounterindexer_reorg_halt_total{height=…}).indexer-reorg-recover.shwith--preview-derivedrow counts and optional derived-table cleanup SQL; addmake indexer-reorg-recover.last_indexed_heightadvancing, pair/candle API vs LCD tip).docs/templates/incident-dex-indexer.mdwith reorg-halt triage checklist.Verified on my side on main
87b6f9a. All five ACs covered:Ran: test-indexer-reorg-recover.sh dry-run smoke (PASS), a live dry-run preview, lib tests (recovery_command_matches_fork_height, webhook_payload_serializes_event_type), and the reorg_alert_webhook integration test — all green.
One note: I didn't run a live --apply replay this pass — recent blocks are empty so it wouldn't even exercise swap dedup, and the dedup itself was already proven in an earlier replay (swap_events held at 119→119, 0 dups) with the ON CONFLICT invariant unchanged. Happy to run a live cursor-rewind replay if you want fresh evidence.
Good to close from my side. @PlasticDigits
mentioned in issue #337