Indexer: automate reorg recovery alerting and replay (#362) #872

Merged
PlasticDigits merged 2 commits from issue-362-reorg-recovery-alerting into main 2026-06-12 07:55:14 +00:00
PlasticDigits commented 2026-06-12 05:14:05 +00:00 (Migrated from gitlab.com)

Summary

Implements GitLab #362 (gap C3 follow-up from #361):

  • Reorg halt alerting: reorg_alert module emits structured indexer_reorg_halt tracing (target indexer.reorg_halt) with height, hashes, and recovery command. Optional REORG_ALERT_WEBHOOK_URL POSTs JSON for PagerDuty/Slack/custom hooks.
  • Recovery automation: Extended scripts/indexer-reorg-recover.sh with read-only row-impact preview, --cleanup-derived for fork replay deletes, and make indexer-reorg-recover wrapper. Destructive steps still require explicit --apply.
  • Docs: Runbook shallow vs deep reorg paths, incident template triage, operator-secrets, indexer invariants, agent skill cross-links.

Acceptance checklist

Criterion Verification Result
Reorg halt emits operator-actionable signal (not only stdout error) cd indexer && cargo test --test reorg_alert_webhook; log fields in reorg_alert.rs PASS
Documented recovery path halt → healthy catch-up docs/runbooks/indexer-reorg-replay-dedup.md § Shallow/Deep + halt checklist PASS
Dry-run shows SQL / row-impact preview ./scripts/indexer-reorg-recover.sh --height 100; bash scripts/test-indexer-reorg-recover.sh PASS
Replay over already-indexed swaps does not duplicate rows cd indexer && cargo test --test indexer_ingestion_hardening swap_replay_does_not_duplicate_rows PASS
Runbook updated shallow vs deep reorg docs/runbooks/indexer-reorg-replay-dedup.md PASS

Verification checklist (third parties)

  1. cd indexer && cargo test --lib
  2. make setup-indexer-postgres && cd indexer && cargo test --test indexer_ingestion_hardening --test reorg_alert_webhook -j 1 -- --test-threads=1
  3. bash scripts/test-indexer-reorg-recover.sh
  4. ./scripts/indexer-reorg-recover.sh --height 100 — confirm DRY RUN + row counts + SQL preview (no --apply)
  5. (Optional) Set REORG_ALERT_WEBHOOK_URL to a test hook; induce hash mismatch in dev; confirm webhook JSON payload

Test plan

cd indexer && cargo test --lib
make setup-indexer-postgres
cd indexer && cargo test --test indexer_ingestion_hardening --test reorg_alert_webhook -j 1 -- --test-threads=1
bash scripts/test-indexer-reorg-recover.sh

Manual LocalTerra reorg simulation: SKIP (requires operator QA Postgres + induced mismatch; script dry-run validated against live Postgres).

Follow-ups

  • Optional: wire scripts/test-indexer-reorg-recover.sh into make test-indexer-integration or CI when Postgres job exists.
  • Trader position / rollup full rebuild helper for deep reorg (currently documented as snapshot restore path).

Related: #361, #236


Note

Medium Risk
Recovery --cleanup-derived --apply deletes indexed rows at/above the fork height and can leave trader rollups inconsistent without snapshot restore; alerting adds an outbound HTTP dependency when webhook URL is set.

Overview
Adds operator-facing reorg halt signals and safer recovery tooling on top of the existing hash guard (C3).

When the poller detects a hash mismatch, it now calls emit_reorg_halt: structured tracing with event=indexer_reorg_halt (target indexer.reorg_halt), fork height, hashes, recovery command, and runbook path. Optional REORG_ALERT_WEBHOOK_URL POSTs the same JSON before exit.

scripts/indexer-reorg-recover.sh gains current cursor display, read-only row-impact previews, --cleanup-derived (transactional deletes for block-height tables + affected candles), and a make indexer-reorg-recover wrapper (HEIGHT, APPLY, CLEANUP). Destructive work still requires --apply.

Runbooks, incident template, operator-secrets, and invariants document shallow vs deep reorg paths, alerting wiring, and halt → catch-up checklist. Tests cover webhook delivery (wiremock), recovery script dry-run smoke, and swap replay dedup.

Reviewed by Cursor Bugbot for commit c5c2ac34cb. Bugbot is set up for automated code reviews on this repo. Configure here.

## Summary Implements GitLab #362 (gap C3 follow-up from #361): - **Reorg halt alerting:** `reorg_alert` module emits structured `indexer_reorg_halt` tracing (target `indexer.reorg_halt`) with height, hashes, and recovery command. Optional `REORG_ALERT_WEBHOOK_URL` POSTs JSON for PagerDuty/Slack/custom hooks. - **Recovery automation:** Extended `scripts/indexer-reorg-recover.sh` with read-only row-impact preview, `--cleanup-derived` for fork replay deletes, and `make indexer-reorg-recover` wrapper. Destructive steps still require explicit `--apply`. - **Docs:** Runbook shallow vs deep reorg paths, incident template triage, operator-secrets, indexer invariants, agent skill cross-links. ## Acceptance checklist | Criterion | Verification | Result | |-----------|--------------|--------| | Reorg halt emits operator-actionable signal (not only stdout error) | `cd indexer && cargo test --test reorg_alert_webhook`; log fields in `reorg_alert.rs` | PASS | | Documented recovery path halt → healthy catch-up | `docs/runbooks/indexer-reorg-replay-dedup.md` § Shallow/Deep + halt checklist | PASS | | Dry-run shows SQL / row-impact preview | `./scripts/indexer-reorg-recover.sh --height 100`; `bash scripts/test-indexer-reorg-recover.sh` | PASS | | Replay over already-indexed swaps does not duplicate rows | `cd indexer && cargo test --test indexer_ingestion_hardening swap_replay_does_not_duplicate_rows` | PASS | | Runbook updated shallow vs deep reorg | `docs/runbooks/indexer-reorg-replay-dedup.md` | PASS | ## Verification checklist (third parties) 1. `cd indexer && cargo test --lib` 2. `make setup-indexer-postgres && cd indexer && cargo test --test indexer_ingestion_hardening --test reorg_alert_webhook -j 1 -- --test-threads=1` 3. `bash scripts/test-indexer-reorg-recover.sh` 4. `./scripts/indexer-reorg-recover.sh --height 100` — confirm DRY RUN + row counts + SQL preview (no `--apply`) 5. (Optional) Set `REORG_ALERT_WEBHOOK_URL` to a test hook; induce hash mismatch in dev; confirm webhook JSON payload ## Test plan ```bash cd indexer && cargo test --lib make setup-indexer-postgres cd indexer && cargo test --test indexer_ingestion_hardening --test reorg_alert_webhook -j 1 -- --test-threads=1 bash scripts/test-indexer-reorg-recover.sh ``` Manual LocalTerra reorg simulation: SKIP (requires operator QA Postgres + induced mismatch; script dry-run validated against live Postgres). ## Follow-ups - Optional: wire `scripts/test-indexer-reorg-recover.sh` into `make test-indexer-integration` or CI when Postgres job exists. - Trader position / rollup full rebuild helper for deep reorg (currently documented as snapshot restore path). Related: #361, #236 <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Recovery `--cleanup-derived --apply` deletes indexed rows at/above the fork height and can leave trader rollups inconsistent without snapshot restore; alerting adds an outbound HTTP dependency when webhook URL is set. > > **Overview** > Adds **operator-facing reorg halt signals** and **safer recovery tooling** on top of the existing hash guard (C3). > > When the poller detects a hash mismatch, it now calls **`emit_reorg_halt`**: structured `tracing` with `event=indexer_reorg_halt` (target `indexer.reorg_halt`), fork height, hashes, recovery command, and runbook path. Optional **`REORG_ALERT_WEBHOOK_URL`** POSTs the same JSON before exit. > > **`scripts/indexer-reorg-recover.sh`** gains current cursor display, read-only **row-impact previews**, **`--cleanup-derived`** (transactional deletes for block-height tables + affected candles), and a **`make indexer-reorg-recover`** wrapper (`HEIGHT`, `APPLY`, `CLEANUP`). Destructive work still requires **`--apply`**. > > Runbooks, incident template, operator-secrets, and invariants document shallow vs deep reorg paths, alerting wiring, and halt → catch-up checklist. Tests cover webhook delivery (wiremock), recovery script dry-run smoke, and swap replay dedup. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit c5c2ac34cbe84f04dc4ec1339984ae199507bdc2. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
ghost1 commented 2026-06-12 05:14:14 +00:00 (Migrated from gitlab.com)

changed the description

changed the description
ghost1 commented 2026-06-12 05:14:22 +00:00 (Migrated from gitlab.com)

changed the description

changed the description
ghost1 commented 2026-06-12 05:15:25 +00:00 (Migrated from gitlab.com)

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

Fix All in Cursor

Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issues.

Reviewed by Cursor Bugbot for commit 9f6ae80be4. Configure here.

<!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues. <!-- BUGBOT_FIX_ALL --> <a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aTpiYjFlNGE0MC02YjU4LTQ5ZjYtOWIxMC1iMzZmMGU5ODIwNTAiLCJlbmNyeXB0aW9uS2V5IjoidEtiRTNDZl9aMnBzSWlVcEZGUXJ6N09CNlNlMnJvVnV5OTFYQnV6WjVwVSIsImJyYW5jaCI6Imlzc3VlLTM2Mi1yZW9yZy1yZWNvdmVyeS1hbGVydGluZyIsInJlcG9Pd25lciI6IlBsYXN0aWNEaWdpdHMiLCJyZXBvTmFtZSI6ImNsOHktZGV4LXRlcnJhY2xhc3NpYyJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <!-- /BUGBOT_FIX_ALL --> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> <sup>Bugbot Autofix is [ON](https://www.cursor.com/dashboard/bugbot). A cloud agent has been kicked off to fix the reported issues. <!-- BUGBOT_AUTOFIX_AGENT_LINK --></sup> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
ghost1 commented 2026-06-12 05:15:26 +00:00 (Migrated from gitlab.com)

Webhook may abort on halt

Medium Severity

Reorg webhook delivery uses tokio::spawn without waiting. When the poller returns an error after a halt, main exits via tokio::select! on the indexer task, so the runtime can drop before the POST finishes and optional REORG_ALERT_WEBHOOK_URL alerts may never arrive.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 9f6ae80be4. Configure here.

### Webhook may abort on halt **Medium Severity** <!-- DESCRIPTION START --> Reorg webhook delivery uses `tokio::spawn` without waiting. When the poller returns an error after a halt, `main` exits via `tokio::select!` on the indexer task, so the runtime can drop before the POST finishes and optional `REORG_ALERT_WEBHOOK_URL` alerts may never arrive. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 74cbf31f-f412-4f6e-a86c-39fce1ab94d6 --> <!-- LOCATIONS START indexer/src/indexer/reorg_alert.rs#L56-L66 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjQ1NTk0YTVmLWVkMzctNGI3MC04MWU3LWVmZGM4ODkzNDhjNiIsImVuY3J5cHRpb25LZXkiOiJfTG5LeVQwUklvWEZqVF94YUNUbG1aZWR2MEJQZUhRZzlNNXlaaWZYdm1VIiwiYnJhbmNoIjoiaXNzdWUtMzYyLXJlb3JnLXJlY292ZXJ5LWFsZXJ0aW5nIiwicmVwb093bmVyIjoiUGxhc3RpY0RpZ2l0cyIsInJlcG9OYW1lIjoiY2w4eS1kZXgtdGVycmFjbGFzc2ljIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjQ1NTk0YTVmLWVkMzctNGI3MC04MWU3LWVmZGM4ODkzNDhjNiIsImVuY3J5cHRpb25LZXkiOiJfTG5LeVQwUklvWEZqVF94YUNUbG1aZWR2MEJQZUhRZzlNNXlaaWZYdm1VIiwiYnJhbmNoIjoiaXNzdWUtMzYyLXJlb3JnLXJlY292ZXJ5LWFsZXJ0aW5nIiwicmVwb093bmVyIjoiUGxhc3RpY0RpZ2l0cyIsInJlcG9OYW1lIjoiY2w4eS1kZXgtdGVycmFjbGFzc2ljIiwicHJOdW1iZXIiOjE1NCwiY29tbWl0U2hhIjoiOWY2YWU4MGJlNDg4ZTBjNGQ4YTJhODQzOGZjZDhhZDFmNTM2NDU2YiIsInByb3ZpZGVyIjoiZ2l0bGFiIiwiaG9zdG5hbWUiOiJnaXRsYWIuY29tIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
ghost1 commented 2026-06-12 05:15:26 +00:00 (Migrated from gitlab.com)

Cursor reset before derived cleanup

Medium Severity

With --apply and --cleanup-derived, cursor reset runs in a committed transaction before derived cleanup in a second transaction. If derived SQL fails, the indexer cursor is already rewound while fork-height rows may remain.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 9f6ae80be4. Configure here.

### Cursor reset before derived cleanup **Medium Severity** <!-- DESCRIPTION START --> With `--apply` and `--cleanup-derived`, cursor reset runs in a committed transaction before derived cleanup in a second transaction. If derived SQL fails, the indexer cursor is already rewound while fork-height rows may remain. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 0708544a-a7d0-4b7c-81f4-9374209b93ae --> <!-- LOCATIONS START scripts/indexer-reorg-recover.sh#L163-L170 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmNiMDU1MTM0LTQ4YWUtNGFiYS1iMmExLTk0NDMzMDRhNjJkMSIsImVuY3J5cHRpb25LZXkiOiI1d2w5Y284Q3lYNHdsazhtUTduOVhvQ093Y2xmTmRCRE9qMm5La1FUdEpvIiwiYnJhbmNoIjoiaXNzdWUtMzYyLXJlb3JnLXJlY292ZXJ5LWFsZXJ0aW5nIiwicmVwb093bmVyIjoiUGxhc3RpY0RpZ2l0cyIsInJlcG9OYW1lIjoiY2w4eS1kZXgtdGVycmFjbGFzc2ljIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmNiMDU1MTM0LTQ4YWUtNGFiYS1iMmExLTk0NDMzMDRhNjJkMSIsImVuY3J5cHRpb25LZXkiOiI1d2w5Y284Q3lYNHdsazhtUTduOVhvQ093Y2xmTmRCRE9qMm5La1FUdEpvIiwiYnJhbmNoIjoiaXNzdWUtMzYyLXJlb3JnLXJlY292ZXJ5LWFsZXJ0aW5nIiwicmVwb093bmVyIjoiUGxhc3RpY0RpZ2l0cyIsInJlcG9OYW1lIjoiY2w4eS1kZXgtdGVycmFjbGFzc2ljIiwicHJOdW1iZXIiOjE1NCwiY29tbWl0U2hhIjoiOWY2YWU4MGJlNDg4ZTBjNGQ4YTJhODQzOGZjZDhhZDFmNTM2NDU2YiIsInByb3ZpZGVyIjoiZ2l0bGFiIiwiaG9zdG5hbWUiOiJnaXRsYWIuY29tIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
ghost1 commented 2026-06-12 05:15:27 +00:00 (Migrated from gitlab.com)

Wrong candles table name

High Severity

With --cleanup-derived, the script queries and deletes from ohlcv_candles, but migrations rename that table to candles, which the indexer uses. Derived cleanup fails at runtime while cursor reset may already have been applied in a separate step.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 9f6ae80be4. Configure here.

### Wrong candles table name **High Severity** <!-- DESCRIPTION START --> With `--cleanup-derived`, the script queries and deletes from `ohlcv_candles`, but migrations rename that table to `candles`, which the indexer uses. Derived cleanup fails at runtime while cursor reset may already have been applied in a separate step. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 404e9345-b9f7-4973-82c0-9bb75930b2bc --> <!-- LOCATIONS START scripts/indexer-reorg-recover.sh#L105-L131 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjhmMjVlZmMxLWY5ZmUtNDczYy04NWMwLTMwZTJkYzc1MmEwMSIsImVuY3J5cHRpb25LZXkiOiJuWXVmak92SG4zZ0s1SWNid3E2ZFNqeHc4R2g1bGxMR21pektrWmFNdUZVIiwiYnJhbmNoIjoiaXNzdWUtMzYyLXJlb3JnLXJlY292ZXJ5LWFsZXJ0aW5nIiwicmVwb093bmVyIjoiUGxhc3RpY0RpZ2l0cyIsInJlcG9OYW1lIjoiY2w4eS1kZXgtdGVycmFjbGFzc2ljIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjhmMjVlZmMxLWY5ZmUtNDczYy04NWMwLTMwZTJkYzc1MmEwMSIsImVuY3J5cHRpb25LZXkiOiJuWXVmak92SG4zZ0s1SWNid3E2ZFNqeHc4R2g1bGxMR21pektrWmFNdUZVIiwiYnJhbmNoIjoiaXNzdWUtMzYyLXJlb3JnLXJlY292ZXJ5LWFsZXJ0aW5nIiwicmVwb093bmVyIjoiUGxhc3RpY0RpZ2l0cyIsInJlcG9OYW1lIjoiY2w4eS1kZXgtdGVycmFjbGFzc2ljIiwicHJOdW1iZXIiOjE1NCwiY29tbWl0U2hhIjoiOWY2YWU4MGJlNDg4ZTBjNGQ4YTJhODQzOGZjZDhhZDFmNTM2NDU2YiIsInByb3ZpZGVyIjoiZ2l0bGFiIiwiaG9zdG5hbWUiOiJnaXRsYWIuY29tIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
PlasticDigits commented 2026-06-12 05:16:02 +00:00 (Migrated from gitlab.com)

Security review — MR !872

Commit reviewed: 9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b

Scope: indexer/src/indexer/reorg_alert.rs, poller.rs (halt alerting), scripts/indexer-reorg-recover.sh / Makefile wrapper, related tests and operator docs.

Method: Diff-focused review; traced attacker-controlled inputs to sinks (webhook POST, SQL in recovery script, structured logging). Checked prior MR security threads (none found).

Areas examined

Surface Assessment
REORG_ALERT_WEBHOOK_URL → reqwest POST Operator-only env var; no chain/LCD/API input reaches URL. 10s timeout; best-effort async delivery does not affect halt path. Redirect-following is inherent to reqwest but requires privileged env control — no remote unauthenticated SSRF path.
emit_reorg_halt payload (hashes, recovery command) Sourced from internal reorg guard (i64 height, LCD/stored hashes). JSON via serde; height validated in recovery script (^[0-9]+$). No injection path to webhook or logs beyond normal structured fields.
indexer-reorg-recover.sh SQL --height strictly validated as positive integer before interpolation; table names hardcoded. Destructive --cleanup-derived requires explicit --apply (unchanged operator trust model).
Authn/authz / new API surface None introduced.
Secret leakage DATABASE_URL masked in script output; webhook carries operational metadata only.

Outcome: FINDINGS: 0 medium+

Security review: no medium+ findings on this diff.

No inline threads (nothing to anchor).

## Security review — MR !872 **Commit reviewed:** `9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b` **Scope:** `indexer/src/indexer/reorg_alert.rs`, `poller.rs` (halt alerting), `scripts/indexer-reorg-recover.sh` / Makefile wrapper, related tests and operator docs. **Method:** Diff-focused review; traced attacker-controlled inputs to sinks (webhook POST, SQL in recovery script, structured logging). Checked prior MR security threads (none found). ### Areas examined | Surface | Assessment | |---------|------------| | `REORG_ALERT_WEBHOOK_URL` → `reqwest` POST | Operator-only env var; no chain/LCD/API input reaches URL. 10s timeout; best-effort async delivery does not affect halt path. Redirect-following is inherent to reqwest but requires privileged env control — no remote unauthenticated SSRF path. | | `emit_reorg_halt` payload (hashes, recovery command) | Sourced from internal reorg guard (`i64` height, LCD/stored hashes). JSON via `serde`; height validated in recovery script (`^[0-9]+$`). No injection path to webhook or logs beyond normal structured fields. | | `indexer-reorg-recover.sh` SQL | `--height` strictly validated as positive integer before interpolation; table names hardcoded. Destructive `--cleanup-derived` requires explicit `--apply` (unchanged operator trust model). | | Authn/authz / new API surface | None introduced. | | Secret leakage | `DATABASE_URL` masked in script output; webhook carries operational metadata only. | **Outcome:** `FINDINGS: 0` medium+ Security review: **no medium+ findings** on this diff. No inline threads (nothing to anchor).
ghost1 commented 2026-06-12 05:18:17 +00:00 (Migrated from gitlab.com)

changed this line in version 2 of the diff

changed this line in [version 2 of the diff](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/154/diffs?diff_id=1848248576&start_sha=9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b#532cf1f922ff5b00c4efe18990a89a0405ff5b87_66_59)
ghost1 commented 2026-06-12 05:18:18 +00:00 (Migrated from gitlab.com)

changed this line in version 2 of the diff

changed this line in [version 2 of the diff](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/154/diffs?diff_id=1848248576&start_sha=9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b#9fac0403e21b79892ce32b8dc8eb637927839e58_170_186)
ghost1 commented 2026-06-12 05:18:18 +00:00 (Migrated from gitlab.com)

changed this line in version 2 of the diff

changed this line in [version 2 of the diff](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/154/diffs?diff_id=1848248576&start_sha=9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b#9fac0403e21b79892ce32b8dc8eb637927839e58_131_131)
ghost1 commented 2026-06-12 05:18:18 +00:00 (Migrated from gitlab.com)

added 1 commit

  • c5c2ac34 - fix(indexer): reorg recovery script and webhook delivery (#362)

Compare with previous version

added 1 commit <ul><li>c5c2ac34 - fix(indexer): reorg recovery script and webhook delivery (#362)</li></ul> [Compare with previous version](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/154/diffs?diff_id=1848248576&start_sha=9f6ae80be488e0c4d8a2a8438fcd8ad1f536456b)
ghost1 commented 2026-06-12 05:18:31 +00:00 (Migrated from gitlab.com)

changed the description

changed the description
ghost1 commented 2026-06-12 05:18:49 +00:00 (Migrated from gitlab.com)

resolved all threads

resolved all threads
PlasticDigits (Migrated from gitlab.com) merged commit ee25e50e23 into main 2026-06-12 07:55:14 +00:00
PlasticDigits commented 2026-06-12 07:55:15 +00:00 (Migrated from gitlab.com)

mentioned in commit ee25e50e23

mentioned in commit ee25e50e23ac2a6d86562bc30882bd45c4b3c74f
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
code/cl8y-dex-terraclassic!872
No description provided.