Indexer: automate reorg recovery alerting and replay (#362) #872
No reviewers
Labels
No labels
agent:fix_bugfix
agent:fix_conflicts
agent:fix_security
agent:gap_analysis
agent:implement
agent:implement
agent:implement
agent:open_issues
agent:ready
agent:research
agent:security_audit
agent:verify
architecture
backend
blocker:hybrid
blocker:launch
blocker:limit-orders
blocker:v2
block:log_only
block:security
bug
ci
contracts
correctness
deploy
dev
devops
docs
documentation
duplicate
e2e
enhancement
epic
feature
frontend
functional-completion
gas
good first issue
governance
help wanted
high-risk
hooks
hybrid
indexer
infra
infrastructure
integrators
invalid
launch-blocker
limit-orders
localnet
localterra
low priority
missing-implementation
needs-design
ops
performance
priority
high
priority
medium
product
qa
QA
question
ready
ready
research
scripts
security
security-hardening
smartcontracts
tech-debt
testing
ux
UX
v2
verification
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
code/cl8y-dex-terraclassic!872
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "issue-362-reorg-recovery-alerting"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Implements GitLab #362 (gap C3 follow-up from #361):
reorg_alertmodule emits structuredindexer_reorg_halttracing (targetindexer.reorg_halt) with height, hashes, and recovery command. OptionalREORG_ALERT_WEBHOOK_URLPOSTs JSON for PagerDuty/Slack/custom hooks.scripts/indexer-reorg-recover.shwith read-only row-impact preview,--cleanup-derivedfor fork replay deletes, andmake indexer-reorg-recoverwrapper. Destructive steps still require explicit--apply.Acceptance checklist
cd indexer && cargo test --test reorg_alert_webhook; log fields inreorg_alert.rsdocs/runbooks/indexer-reorg-replay-dedup.md§ Shallow/Deep + halt checklist./scripts/indexer-reorg-recover.sh --height 100;bash scripts/test-indexer-reorg-recover.shcd indexer && cargo test --test indexer_ingestion_hardening swap_replay_does_not_duplicate_rowsdocs/runbooks/indexer-reorg-replay-dedup.mdVerification checklist (third parties)
cd indexer && cargo test --libmake setup-indexer-postgres && cd indexer && cargo test --test indexer_ingestion_hardening --test reorg_alert_webhook -j 1 -- --test-threads=1bash scripts/test-indexer-reorg-recover.sh./scripts/indexer-reorg-recover.sh --height 100— confirm DRY RUN + row counts + SQL preview (no--apply)REORG_ALERT_WEBHOOK_URLto a test hook; induce hash mismatch in dev; confirm webhook JSON payloadTest plan
Manual LocalTerra reorg simulation: SKIP (requires operator QA Postgres + induced mismatch; script dry-run validated against live Postgres).
Follow-ups
scripts/test-indexer-reorg-recover.shintomake test-indexer-integrationor CI when Postgres job exists.Related: #361, #236
changed the description
changed the description
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issues.
Reviewed by Cursor Bugbot for commit
9f6ae80be4. Configure here.Webhook may abort on halt
Medium Severity
Reorg webhook delivery uses
tokio::spawnwithout waiting. When the poller returns an error after a halt,mainexits viatokio::select!on the indexer task, so the runtime can drop before the POST finishes and optionalREORG_ALERT_WEBHOOK_URLalerts may never arrive.Reviewed by Cursor Bugbot for commit
9f6ae80be4. Configure here.Cursor reset before derived cleanup
Medium Severity
With
--applyand--cleanup-derived, cursor reset runs in a committed transaction before derived cleanup in a second transaction. If derived SQL fails, the indexer cursor is already rewound while fork-height rows may remain.Reviewed by Cursor Bugbot for commit
9f6ae80be4. Configure here.Wrong candles table name
High Severity
With
--cleanup-derived, the script queries and deletes fromohlcv_candles, but migrations rename that table tocandles, which the indexer uses. Derived cleanup fails at runtime while cursor reset may already have been applied in a separate step.Reviewed by Cursor Bugbot for commit
9f6ae80be4. Configure here.Security review — MR !872
Commit reviewed:
9f6ae80be488e0c4d8a2a8438fcd8ad1f536456bScope:
indexer/src/indexer/reorg_alert.rs,poller.rs(halt alerting),scripts/indexer-reorg-recover.sh/ Makefile wrapper, related tests and operator docs.Method: Diff-focused review; traced attacker-controlled inputs to sinks (webhook POST, SQL in recovery script, structured logging). Checked prior MR security threads (none found).
Areas examined
REORG_ALERT_WEBHOOK_URL→reqwestPOSTemit_reorg_haltpayload (hashes, recovery command)i64height, LCD/stored hashes). JSON viaserde; height validated in recovery script (^[0-9]+$). No injection path to webhook or logs beyond normal structured fields.indexer-reorg-recover.shSQL--heightstrictly validated as positive integer before interpolation; table names hardcoded. Destructive--cleanup-derivedrequires explicit--apply(unchanged operator trust model).DATABASE_URLmasked in script output; webhook carries operational metadata only.Outcome:
FINDINGS: 0medium+Security review: no medium+ findings on this diff.
No inline threads (nothing to anchor).
changed this line in version 2 of the diff
changed this line in version 2 of the diff
changed this line in version 2 of the diff
added 1 commit
c5c2ac34- fix(indexer): reorg recovery script and webhook delivery (#362)Compare with previous version
changed the description
resolved all threads
mentioned in commit
ee25e50e23