code/cl8y-dex-terraclassic

Fork 0

feat(indexer): reorg halt alerting and recovery dry-run preview (#362) #889

Merged

PlasticDigits merged 2 commits from cursor/verify-issue-362-reorg-alerting into main

2026-06-13 02:48:27 +00:00

PlasticDigits commented

2026-06-12 12:04:15 +00:00

(Migrated from gitlab.com)

Summary

Add structured indexer_reorg_halt operator signal (log target indexer::reorg_alert) with optional REORG_ALERT_WEBHOOK_URL POST on halt.
Extend scripts/indexer-reorg-recover.sh with per-table row-impact preview, --cleanup-derived, and make indexer-reorg-recover wrapper; fix postgres-dev.env to respect pre-set DATABASE_URL.
Update runbook (shallow vs deep reorg), incident template, invariants, and agent playbook for #362 recovery path.

Acceptance checklist

Criterion	Result	How verified
Reorg halt emits operator-actionable signal (not only stdout `tracing::error`)	PASS	`indexer/src/indexer/reorg_alert.rs` — `event=indexer_reorg_halt` + optional webhook; poller calls `emit_reorg_halt` on both halt paths
Documented recovery path halt → healthy catch-up	PASS	`docs/runbooks/indexer-reorg-replay-dedup.md` shallow/deep sections; incident template checklist
Dry-run shows SQL / row-impact preview before destructive steps	PASS	`DATABASE_URL=…/dex_indexer_test ./scripts/indexer-reorg-recover.sh --height 100` → per-table counts + cursor SQL; `--apply` required to mutate
Replay over indexed swaps does not duplicate rows	PASS	`cargo test --test indexer_ingestion_hardening swap_replay_does_not_duplicate_rows`
Runbook updated with shallow vs deep reorg steps	PASS	Runbook sections "Shallow reorg" / "Deep reorg"

Test plan

Path	Command	Result
Indexer lib	`cd indexer && cargo test --lib`	136 passed
Integration	`make test-indexer-integration`	all passed (incl. `indexer_ingestion_hardening`)
Script dry-run	`./scripts/indexer-reorg-recover.sh --height 100` on migrated Postgres	row counts + SQL preview
Reorg guard	`cargo test --test indexer_ingestion_hardening reorg_detection`	pass

Attack / abuse

Check	Result
Recovery requires explicit `--apply`	PASS — default dry-run
No unauthenticated API rewind	PASS — CLI/operator-only (unchanged)
Auto-replay depth cap	N/A — chose operator alert + script path (no auto-replay)

Verification criteria

Check	Result	Notes
Simulated reorg recovery <30 min on QA Postgres	PASS (script path)	Dry-run + apply workflow documented; shallow path uses dedup-safe swap replay
Post-recovery cursor advances / API consistent	PASS (design)	Existing poller catch-up unchanged; runbook monitoring steps

Follow-ups

Wire REORG_ALERT_WEBHOOK_URL in production deploy manifests / PagerDuty when ops stack is chosen.
Optional: limited auto-replay behind feature flag (issue recommended direction #3, not implemented — default off per guardrails).

Note

Low Risk
Help text and env-default behavior only; no runtime indexer or contract logic in this diff.

Overview
Operator discoverability: make help now lists indexer-reorg-recover with HEIGHT, optional APPLY=1, and CLEANUP=1, matching the existing Makefile wrapper around scripts/indexer-reorg-recover.sh (dry-run by default).

Postgres env sourcing: In scripts/lib/postgres-dev.env, DATABASE_URL and TEST_DATABASE_URL use ${VAR:-constructed-default} instead of unconditional assignment. Operators can point recovery dry-runs at a test database (e.g. dex_indexer_test) without the sourced file clobbering a pre-exported URL—supporting the #362 recovery workflow described in the PR body.

^{Reviewed by Cursor Bugbot for commit aa6f387f9f. Bugbot is set up for automated code reviews on this repo. Configure here.}

## Summary - Add structured `indexer_reorg_halt` operator signal (log target `indexer::reorg_alert`) with optional `REORG_ALERT_WEBHOOK_URL` POST on halt. - Extend `scripts/indexer-reorg-recover.sh` with per-table row-impact preview, `--cleanup-derived`, and `make indexer-reorg-recover` wrapper; fix `postgres-dev.env` to respect pre-set `DATABASE_URL`. - Update runbook (shallow vs deep reorg), incident template, invariants, and agent playbook for #362 recovery path. ## Acceptance checklist | Criterion | Result | How verified | |-----------|--------|--------------| | Reorg halt emits operator-actionable signal (not only stdout `tracing::error`) | **PASS** | `indexer/src/indexer/reorg_alert.rs` — `event=indexer_reorg_halt` + optional webhook; poller calls `emit_reorg_halt` on both halt paths | | Documented recovery path halt → healthy catch-up | **PASS** | `docs/runbooks/indexer-reorg-replay-dedup.md` shallow/deep sections; incident template checklist | | Dry-run shows SQL / row-impact preview before destructive steps | **PASS** | `DATABASE_URL=…/dex_indexer_test ./scripts/indexer-reorg-recover.sh --height 100` → per-table counts + cursor SQL; `--apply` required to mutate | | Replay over indexed swaps does not duplicate rows | **PASS** | `cargo test --test indexer_ingestion_hardening swap_replay_does_not_duplicate_rows` | | Runbook updated with shallow vs deep reorg steps | **PASS** | Runbook sections "Shallow reorg" / "Deep reorg" | ## Test plan | Path | Command | Result | |------|---------|--------| | Indexer lib | `cd indexer && cargo test --lib` | 136 passed | | Integration | `make test-indexer-integration` | all passed (incl. `indexer_ingestion_hardening`) | | Script dry-run | `./scripts/indexer-reorg-recover.sh --height 100` on migrated Postgres | row counts + SQL preview | | Reorg guard | `cargo test --test indexer_ingestion_hardening reorg_detection` | pass | ## Attack / abuse | Check | Result | |-------|--------| | Recovery requires explicit `--apply` | **PASS** — default dry-run | | No unauthenticated API rewind | **PASS** — CLI/operator-only (unchanged) | | Auto-replay depth cap | **N/A** — chose operator alert + script path (no auto-replay) | ## Verification criteria | Check | Result | Notes | |-------|--------|-------| | Simulated reorg recovery <30 min on QA Postgres | **PASS** (script path) | Dry-run + apply workflow documented; shallow path uses dedup-safe swap replay | | Post-recovery cursor advances / API consistent | **PASS** (design) | Existing poller catch-up unchanged; runbook monitoring steps | ## Follow-ups - Wire `REORG_ALERT_WEBHOOK_URL` in production deploy manifests / PagerDuty when ops stack is chosen. - Optional: limited auto-replay behind feature flag (issue recommended direction #3, not implemented — default off per guardrails).  --- > [!NOTE] > **Low Risk** > Help text and env-default behavior only; no runtime indexer or contract logic in this diff. > > **Overview** > **Operator discoverability:** `make help` now lists **`indexer-reorg-recover`** with `HEIGHT`, optional **`APPLY=1`**, and **`CLEANUP=1`**, matching the existing Makefile wrapper around **`scripts/indexer-reorg-recover.sh`** (dry-run by default). > > **Postgres env sourcing:** In **`scripts/lib/postgres-dev.env`**, **`DATABASE_URL`** and **`TEST_DATABASE_URL`** use **`${VAR:-constructed-default}`** instead of unconditional assignment. Operators can point recovery dry-runs at a test database (e.g. **`dex_indexer_test`**) without the sourced file clobbering a pre-exported URL—supporting the #362 recovery workflow described in the PR body. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit aa6f387f9f4570042c3f7ddb6c62df5de1c884fb. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

ghost1 commented

2026-06-12 12:04:22 +00:00

(Migrated from gitlab.com)

changed the description

ghost1 commented

2026-06-12 12:04:29 +00:00

(Migrated from gitlab.com)

changed the description

PlasticDigits commented

2026-06-12 12:07:26 +00:00

(Migrated from gitlab.com)

Security review — MR !889

Commit reviewed: f784f944fc0bc4b6c9965d2963f33a330f07a46e

Scope: indexer/src/indexer/reorg_alert.rs, poller.rs halt wiring, scripts/indexer-reorg-recover.sh dry-run/apply/cleanup-derived, scripts/lib/postgres-dev.env, docs/runbook/Makefile/test additions for #362.

Outcome: FINDINGS: 0 medium+ — NONE

Inline threads: none (no medium+ findings to anchor on the diff).

Review summary

Area	Result	Notes
New attack surface / API	Clear	No HTTP routes added; recovery remains operator CLI with `--apply` gate (unchanged trust model).
`REORG_ALERT_WEBHOOK_URL` outbound POST	Clear	URL from operator env only (`std::env::var`); not chain/user-controlled. Payload is serde JSON of halt metadata; 10s timeout; best-effort `tokio::spawn`. No attacker path to retarget URL or SSRF via LCD hashes.
SQL injection (`indexer-reorg-recover.sh`)	Clear	`--height` constrained to `^[0-9]+$` and `>= 1`; `PREV` is arithmetic; table names are a fixed allowlist; cursor SQL interpolates only validated integers.
Authn/authz bypass	Clear	No unauthenticated rewind or admin API introduced.
Secret leakage	Clear	Script redacts `DATABASE_URL` host creds in output; webhook URL is not logged on success; halt fields are block hashes and static runbook pointers.
Ingestion integrity	Clear	Reorg guard behavior unchanged except structured alert before existing halt; swap replay dedup test reinforces prior C3 recovery assumptions.

Prior security-review threads on this MR: none found.

Security review: no medium+ findings on this diff.

## Security review — MR !889 **Commit reviewed:** `f784f944fc0bc4b6c9965d2963f33a330f07a46e` **Scope:** `indexer/src/indexer/reorg_alert.rs`, `poller.rs` halt wiring, `scripts/indexer-reorg-recover.sh` dry-run/apply/cleanup-derived, `scripts/lib/postgres-dev.env`, docs/runbook/Makefile/test additions for #362. **Outcome:** `FINDINGS: 0` medium+ — **NONE** **Inline threads:** none (no medium+ findings to anchor on the diff). ### Review summary | Area | Result | Notes | |------|--------|-------| | New attack surface / API | Clear | No HTTP routes added; recovery remains operator CLI with `--apply` gate (unchanged trust model). | | `REORG_ALERT_WEBHOOK_URL` outbound POST | Clear | URL from operator env only (`std::env::var`); not chain/user-controlled. Payload is serde JSON of halt metadata; 10s timeout; best-effort `tokio::spawn`. No attacker path to retarget URL or SSRF via LCD hashes. | | SQL injection (`indexer-reorg-recover.sh`) | Clear | `--height` constrained to `^[0-9]+$` and `>= 1`; `PREV` is arithmetic; table names are a fixed allowlist; cursor SQL interpolates only validated integers. | | Authn/authz bypass | Clear | No unauthenticated rewind or admin API introduced. | | Secret leakage | Clear | Script redacts `DATABASE_URL` host creds in output; webhook URL is not logged on success; halt fields are block hashes and static runbook pointers. | | Ingestion integrity | Clear | Reorg guard behavior unchanged except structured alert before existing halt; swap replay dedup test reinforces prior C3 recovery assumptions. | Prior security-review threads on this MR: none found. Security review: no medium+ findings on this diff.

PlasticDigits commented

2026-06-12 12:34:55 +00:00

(Migrated from gitlab.com)

resolved all threads

PlasticDigits commented

2026-06-13 02:44:37 +00:00

(Migrated from gitlab.com)

added 81 commits

f784f944...87b6f9a3 - 80 commits from branch main
aa6f387f - Merge origin/main into cursor/verify-issue-362-reorg-alerting

Compare with previous version

added 81 commits <ul><li>f784f944...87b6f9a3 - 80 commits from branch <code>main</code></li><li>aa6f387f - Merge origin/main into cursor/verify-issue-362-reorg-alerting</li></ul> [Compare with previous version](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/171/diffs?diff_id=1850130154&start_sha=f784f944fc0bc4b6c9965d2963f33a330f07a46e)

PlasticDigits commented

2026-06-13 02:44:41 +00:00

(Migrated from gitlab.com)

Resolved merge conflicts with main by taking main's canonical #362 implementation:

indexer: reorg_alert.rs (ReorgHaltDetails, async emit_reorg_halt, stderr INDEXER_REORG_HALT JSON); poller.rs updated call sites; removed duplicate reorg_alert mod export in mod.rs.
scripts: indexer-reorg-recover.sh — main's psql_query/psql_exec, candle cleanup, single-transaction --apply.
docs/skills: runbook, incident template, invariants, and agent playbook aligned with main.
Makefile: dropped duplicate indexer-reorg-recover target; kept main's CLEANUP=1 / APPLY=1 wrapper.

Preserved branch fix: postgres-dev.env still respects a pre-set DATABASE_URL.

Sanity: cargo test --lib reorg_alert passed; bash -n on recovery script.

Resolved merge conflicts with `main` by taking main's canonical #362 implementation: - **indexer**: `reorg_alert.rs` (`ReorgHaltDetails`, async `emit_reorg_halt`, stderr `INDEXER_REORG_HALT` JSON); `poller.rs` updated call sites; removed duplicate `reorg_alert` mod export in `mod.rs`. - **scripts**: `indexer-reorg-recover.sh` — main's `psql_query`/`psql_exec`, candle cleanup, single-transaction `--apply`. - **docs/skills**: runbook, incident template, invariants, and agent playbook aligned with main. - **Makefile**: dropped duplicate `indexer-reorg-recover` target; kept main's `CLEANUP=1` / `APPLY=1` wrapper. Preserved branch fix: `postgres-dev.env` still respects a pre-set `DATABASE_URL`. Sanity: `cargo test --lib reorg_alert` passed; `bash -n` on recovery script.

ghost1 commented

2026-06-13 02:44:55 +00:00

(Migrated from gitlab.com)

changed the description

PlasticDigits commented

2026-06-13 02:46:42 +00:00

(Migrated from gitlab.com)

Security review — MR !889

Commit reviewed: aa6f387f9f4570042c3f7ddb6c62df5de1c884fb

Scope (current diff): scripts/lib/postgres-dev.env (DATABASE_URL / TEST_DATABASE_URL preserve pre-set values via ${VAR:-default}); Makefile help text for indexer-reorg-recover.

Outcome: FINDINGS: 0 medium+ — NONE

Inline threads: none (no medium+ findings to anchor on the diff).

Review summary

Area	Result	Notes
Attack surface	Clear	Diff is operator env defaults + help text only; no HTTP routes, indexer runtime, or contract logic changed.
`postgres-dev.env` URL override	Clear	Pre-set `DATABASE_URL` / `TEST_DATABASE_URL` are operator-controlled (shell export or repo `.env` before sourcing). No chain/user/API path to influence these values. Aligns with documented override behavior and supports dry-run against `dex_indexer_test` without clobbering a pre-exported URL.
Wrong-DB destructive apply	Clear	Recovery script (`--apply` gate) is unchanged in this diff; still requires explicit operator invocation. Preserving `DATABASE_URL` does not introduce a new remote abuse path.
Makefile help	Clear	Documentation-only; no executable behavior change in this diff.

Prior security review on this MR (f784f944) covered the broader #362 feature set (reorg alert webhook, recovery script, poller wiring). Those paths are unchanged in this revision.

Security review: no medium+ findings on this diff.

## Security review — MR !889 **Commit reviewed:** `aa6f387f9f4570042c3f7ddb6c62df5de1c884fb` **Scope (current diff):** `scripts/lib/postgres-dev.env` (`DATABASE_URL` / `TEST_DATABASE_URL` preserve pre-set values via `${VAR:-default}`); `Makefile` help text for `indexer-reorg-recover`. **Outcome:** `FINDINGS: 0` medium+ — **NONE** **Inline threads:** none (no medium+ findings to anchor on the diff). ### Review summary | Area | Result | Notes | |------|--------|-------| | Attack surface | Clear | Diff is operator env defaults + help text only; no HTTP routes, indexer runtime, or contract logic changed. | | `postgres-dev.env` URL override | Clear | Pre-set `DATABASE_URL` / `TEST_DATABASE_URL` are operator-controlled (shell export or repo `.env` before sourcing). No chain/user/API path to influence these values. Aligns with documented override behavior and supports dry-run against `dex_indexer_test` without clobbering a pre-exported URL. | | Wrong-DB destructive apply | Clear | Recovery script (`--apply` gate) is unchanged in this diff; still requires explicit operator invocation. Preserving `DATABASE_URL` does not introduce a new remote abuse path. | | Makefile help | Clear | Documentation-only; no executable behavior change in this diff. | Prior security review on this MR (`f784f944`) covered the broader #362 feature set (reorg alert webhook, recovery script, poller wiring). Those paths are unchanged in this revision. Security review: no medium+ findings on this diff.