feat(indexer): reorg halt alerting and recovery preview (#362) #883

Merged

PlasticDigits merged 5 commits from verify/issue-362-reorg-recovery-alerting into main 2026-06-12 11:14:01 +00:00

Conversation 23 Commits 5 Files changed 14 +228 -39

PlasticDigits commented 2026-06-12 08:18:17 +00:00 (Migrated from gitlab.com)

Copy link

Summary

• Add reorg_alert module: structured INDEXER_REORG_HALT stderr JSON + tracing target indexer_reorg_halt, optional REORG_ALERT_WEBHOOK_URL POST on halt.
• Extend indexer-reorg-recover.sh dry-run with per-table row-impact counts; add make indexer-reorg-recover HEIGHT=H [APPLY=1].
• Treat cleared last_indexed_block_hash (empty string) as post-recovery skip for reorg guard.
• Update runbook (shallow vs deep reorg), incident template, operator-secrets, invariants, and agent playbook.
• Integration tests: swap replay dedup, simulated recovery catch-up.

Verification checklist

Acceptance item	Result	How verified
Reorg halt emits operator-actionable signal	PASS	reorg_alert.rs emits INDEXER_REORG_HALT JSON + target=indexer_reorg_halt; optional webhook via REORG_ALERT_WEBHOOK_URL
Documented recovery path halt → catch-up	PASS	Runbook shallow/deep tables; incident template triage step
Dry-run shows SQL / row-impact preview	PASS	./scripts/indexer-reorg-recover.sh --height 100 and make indexer-reorg-recover HEIGHT=100
Replay swaps do not duplicate	PASS	swap_replay_is_idempotent_on_conflict integration test
Runbook shallow vs deep reorg	PASS	docs/runbooks/indexer-reorg-replay-dedup.md sections updated
cd indexer && cargo test --lib	PASS	136 passed
make setup-indexer-postgres && make test-indexer-integration	PASS	Full integration suite + indexer_ingestion_hardening 8/8
Recovery requires --apply	PASS	Script defaults to dry-run; APPLY=1 for make target
No unauthenticated API rewind	PASS	CLI/operator script only
Simulated reorg recovery	PASS	reorg_recovery_rewind_allows_catch_up integration test

Follow-ups

• Wire REORG_ALERT_WEBHOOK_URL in production orchestration / PagerDuty.
• Optional: auto-delete derived rows for block_height >= H behind a second --purge-derived flag (out of scope for #362).

---

Note

Medium Risk
Touches ingestion reorg detection and operator recovery paths where mistakes could stall or mis-replay indexing; route API error behavior changes when all path candidates fail.

Overview
Reorg halt (#362): Adds reorg_alert with machine-parseable INDEXER_REORG_HALT stderr JSON, tracing target indexer_reorg_halt, and optional REORG_ALERT_WEBHOOK_URL wired through Config. The reorg guard treats an empty last_indexed_block_hash like a missing hash so post-recovery rewind can catch up without a false halt.

Operator recovery: make indexer-reorg-recover now requires HEIGHT, uses APPLY=1 / CLEANUP=1 (not any non-empty env), and the shell script uses shared Postgres helpers for dry-run/apply. Docs, incident template, and playbooks align on alert signals and Make usage; runbook cleanup wording uses candles instead of ohlcv_candles.

Tests: Integration coverage for swap replay idempotency and simulated rewind → re-index after hash mismatch.

Route solver: Concurrent best-execution returns a 502-style error when every evaluated path fails (not silent NOT_FOUND). DB hybrid sim no longer short-circuits to zero output on unusable reserves when mirror freshness is MissingReserves, allowing LCD fallback.

^{Reviewed by Cursor Bugbot for commit 2ac33b06e6. Bugbot is set up for automated code reviews on this repo. Configure here.}

## Summary - Add `reorg_alert` module: structured `INDEXER_REORG_HALT` stderr JSON + tracing target `indexer_reorg_halt`, optional `REORG_ALERT_WEBHOOK_URL` POST on halt. - Extend `indexer-reorg-recover.sh` dry-run with per-table row-impact counts; add `make indexer-reorg-recover HEIGHT=H [APPLY=1]`. - Treat cleared `last_indexed_block_hash` (empty string) as post-recovery skip for reorg guard. - Update runbook (shallow vs deep reorg), incident template, operator-secrets, invariants, and agent playbook. - Integration tests: swap replay dedup, simulated recovery catch-up. ## Verification checklist | Acceptance item | Result | How verified | |-----------------|--------|--------------| | Reorg halt emits operator-actionable signal | **PASS** | `reorg_alert.rs` emits `INDEXER_REORG_HALT` JSON + `target=indexer_reorg_halt`; optional webhook via `REORG_ALERT_WEBHOOK_URL` | | Documented recovery path halt → catch-up | **PASS** | Runbook shallow/deep tables; incident template triage step | | Dry-run shows SQL / row-impact preview | **PASS** | `./scripts/indexer-reorg-recover.sh --height 100` and `make indexer-reorg-recover HEIGHT=100` | | Replay swaps do not duplicate | **PASS** | `swap_replay_is_idempotent_on_conflict` integration test | | Runbook shallow vs deep reorg | **PASS** | `docs/runbooks/indexer-reorg-replay-dedup.md` sections updated | | `cd indexer && cargo test --lib` | **PASS** | 136 passed | | `make setup-indexer-postgres && make test-indexer-integration` | **PASS** | Full integration suite + `indexer_ingestion_hardening` 8/8 | | Recovery requires `--apply` | **PASS** | Script defaults to dry-run; `APPLY=1` for make target | | No unauthenticated API rewind | **PASS** | CLI/operator script only | | Simulated reorg recovery | **PASS** | `reorg_recovery_rewind_allows_catch_up` integration test | ## Follow-ups - Wire `REORG_ALERT_WEBHOOK_URL` in production orchestration / PagerDuty. - Optional: auto-delete derived rows for `block_height >= H` behind a second `--purge-derived` flag (out of scope for #362). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches ingestion reorg detection and operator recovery paths where mistakes could stall or mis-replay indexing; route API error behavior changes when all path candidates fail. > > **Overview** > **Reorg halt (#362):** Adds `reorg_alert` with machine-parseable **`INDEXER_REORG_HALT`** stderr JSON, tracing target **`indexer_reorg_halt`**, and optional **`REORG_ALERT_WEBHOOK_URL`** wired through `Config`. The reorg guard treats an **empty** `last_indexed_block_hash` like a missing hash so post-recovery rewind can catch up without a false halt. > > **Operator recovery:** `make indexer-reorg-recover` now **requires** `HEIGHT`, uses **`APPLY=1` / `CLEANUP=1`** (not any non-empty env), and the shell script uses shared Postgres helpers for dry-run/apply. Docs, incident template, and playbooks align on alert signals and Make usage; runbook cleanup wording uses **`candles`** instead of **`ohlcv_candles`**. > > **Tests:** Integration coverage for **swap replay idempotency** and **simulated rewind → re-index** after hash mismatch. > > **Route solver:** Concurrent best-execution returns a **502-style error** when every evaluated path fails (not silent NOT_FOUND). DB hybrid sim no longer short-circuits to zero output on unusable reserves when mirror freshness is **`MissingReserves`**, allowing LCD fallback. > > ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 2ac33b06e69f3d13840ab3dc4ba68b4e3979bf4c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).} <!-- /CURSOR_SUMMARY -->

ghost1 commented 2026-06-12 08:18:24 +00:00 (Migrated from gitlab.com)

Copy link

changed the description

changed the description

ghost1 commented 2026-06-12 08:18:34 +00:00 (Migrated from gitlab.com)

Copy link

changed the description

changed the description

ghost1 commented 2026-06-12 08:19:38 +00:00 (Migrated from gitlab.com)

Copy link

Stale Security Review comment

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
^{Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issue.}

^{Reviewed by Cursor Bugbot for commit 8a92d56452. Configure here.}

<details> <summary>Stale Security Review comment</summary> <!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.<!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> ^{Bugbot Autofix is [ON](https://www.cursor.com/dashboard/bugbot). A cloud agent has been kicked off to fix the reported issue. <!-- BUGBOT_AUTOFIX_AGENT_LINK -->} <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 8a92d56452d1a75a36685fe839407905e4d6348c. Configure [here](https://www.cursor.com/dashboard/bugbot).} </details>

ghost1 commented 2026-06-12 08:19:39 +00:00 (Migrated from gitlab.com)

Copy link

MAKE APPLY zero still applies

Medium Severity

The indexer-reorg-recover target passes --apply whenever APPLY is any non-empty value because $(if $(APPLY),--apply,) treats 0 or false as set. A dry-run like APPLY=0 mutates indexer_state instead of previewing SQL.

 

^{Reviewed by Cursor Bugbot for commit 8a92d56452. Configure here.}

### MAKE APPLY zero still applies **Medium Severity** <!-- DESCRIPTION START --> The `indexer-reorg-recover` target passes `--apply` whenever `APPLY` is any non-empty value because `$(if $(APPLY),--apply,)` treats `0` or `false` as set. A dry-run like `APPLY=0` mutates `indexer_state` instead of previewing SQL. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: de688a1a-b474-4ee4-8cc0-0d7bea3fba49 --> <!-- LOCATIONS START Makefile#L327-L328 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjE1MDY4ZjI4LTBiZmItNGM0ZS05ZTVjLWNkMGVmNGQ2MzdmMiIsImVuY3J5cHRpb25LZXkiOiI1cFB6bmc4eGRsaGs2U2lFQzRlNkljbG1RWGFocUhRSy1tRFJUT2hGWFRzIiwiYnJhbmNoIjoidmVyaWZ5L2lzc3VlLTM2Mi1yZW9yZy1yZWNvdmVyeS1hbGVydGluZyIsInJlcG9Pd25lciI6IlBsYXN0aWNEaWdpdHMiLCJyZXBvTmFtZSI6ImNsOHktZGV4LXRlcnJhY2xhc3NpYyJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjE1MDY4ZjI4LTBiZmItNGM0ZS05ZTVjLWNkMGVmNGQ2MzdmMiIsImVuY3J5cHRpb25LZXkiOiI1cFB6bmc4eGRsaGs2U2lFQzRlNkljbG1RWGFocUhRSy1tRFJUT2hGWFRzIiwiYnJhbmNoIjoidmVyaWZ5L2lzc3VlLTM2Mi1yZW9yZy1yZWNvdmVyeS1hbGVydGluZyIsInJlcG9Pd25lciI6IlBsYXN0aWNEaWdpdHMiLCJyZXBvTmFtZSI6ImNsOHktZGV4LXRlcnJhY2xhc3NpYyIsInByTnVtYmVyIjoxNjUsImNvbW1pdFNoYSI6IjhhOTJkNTY0NTJkMWE3NWEzNjY4NWZlODM5NDA3OTA1ZTRkNjM0OGMiLCJwcm92aWRlciI6ImdpdGxhYiIsImhvc3RuYW1lIjoiZ2l0bGFiLmNvbSJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 8a92d56452d1a75a36685fe839407905e4d6348c. Configure [here](https://www.cursor.com/dashboard/bugbot).}

ghost1 commented 2026-06-12 08:21:22 +00:00 (Migrated from gitlab.com)

Copy link

changed this line in version 2 of the diff

changed this line in [version 2 of the diff](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/165/diffs?diff_id=1848518585&start_sha=8a92d56452d1a75a36685fe839407905e4d6348c#836efb6e25a091dcb4ff8e1dbb2f0be6a5cbf14c_328_328)

ghost1 commented 2026-06-12 08:21:23 +00:00 (Migrated from gitlab.com)

Copy link

added 1 commit

• ae66cddd - fix: only pass --apply when APPLY=1 in indexer-reorg-recover

Compare with previous version

added 1 commit <ul><li>ae66cddd - fix: only pass --apply when APPLY=1 in indexer-reorg-recover</li></ul> [Compare with previous version](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/165/diffs?diff_id=1848518585&start_sha=8a92d56452d1a75a36685fe839407905e4d6348c)

PlasticDigits commented 2026-06-12 08:21:26 +00:00 (Migrated from gitlab.com)

Copy link

Security review — MR !883

Commit reviewed: 8a92d56452d1a75a36685fe839407905e4d6348c

Scope: reorg_alert halt signaling + optional REORG_ALERT_WEBHOOK_URL webhook; verify_checkpoint_unchanged empty-hash post-recovery skip; indexer-reorg-recover.sh row-impact preview / make indexer-reorg-recover; docs/tests.

Outcome: FINDINGS: 0 medium+

Inline threads: none (no medium+ issues on this diff)

Summary

Reviewed attacker-controlled input paths for the new/changed code:

Surface	Attacker input?	Controls	Verdict
REORG_ALERT_WEBHOOK_URL / post_webhook	No — operator env only; not API-exposed	Same trust boundary as LCD_URLS / DATABASE_URL; 5s timeout; JSON body is indexer-generated (height, public block hashes, static recovery_cmd)	No SSRF/exfil path from remote callers
verify_checkpoint_unchanged empty-hash skip	No — requires operator recovery script or DB write	No HTTP handler mutates indexer_state; skip is intentional post-rewind window documented in runbook	No unauthenticated reorg-guard bypass
indexer-reorg-recover.sh SQL	--height only	Validated ^[0-9]+$, >= 1; table names hardcoded	No injection
Halt logs / webhook payload	Block hashes from LCD	serde_json serialization; hashes are public chain data	No secret leakage

Recovery remains CLI/operator-only (no new API rewind). Swap replay idempotency unchanged (ON CONFLICT DO NOTHING).

Note: Bugbot's APPLY=0 Makefile footgun is an operator ergonomics issue (local make with DB access), not a remote authz or injection finding — out of scope for this security pass.

Security review: no medium+ findings on this diff.

## Security review — MR !883 **Commit reviewed:** `8a92d56452d1a75a36685fe839407905e4d6348c` **Scope:** `reorg_alert` halt signaling + optional `REORG_ALERT_WEBHOOK_URL` webhook; `verify_checkpoint_unchanged` empty-hash post-recovery skip; `indexer-reorg-recover.sh` row-impact preview / `make indexer-reorg-recover`; docs/tests. **Outcome:** `FINDINGS: 0` medium+ **Inline threads:** none (no medium+ issues on this diff) ### Summary Reviewed attacker-controlled input paths for the new/changed code: | Surface | Attacker input? | Controls | Verdict | |---------|-----------------|----------|---------| | `REORG_ALERT_WEBHOOK_URL` / `post_webhook` | No — operator env only; not API-exposed | Same trust boundary as `LCD_URLS` / `DATABASE_URL`; 5s timeout; JSON body is indexer-generated (`height`, public block hashes, static `recovery_cmd`) | No SSRF/exfil path from remote callers | | `verify_checkpoint_unchanged` empty-hash skip | No — requires operator recovery script or DB write | No HTTP handler mutates `indexer_state`; skip is intentional post-rewind window documented in runbook | No unauthenticated reorg-guard bypass | | `indexer-reorg-recover.sh` SQL | `--height` only | Validated `^[0-9]+$`, `>= 1`; table names hardcoded | No injection | | Halt logs / webhook payload | Block hashes from LCD | `serde_json` serialization; hashes are public chain data | No secret leakage | Recovery remains CLI/operator-only (no new API rewind). Swap replay idempotency unchanged (`ON CONFLICT DO NOTHING`). **Note:** Bugbot's `APPLY=0` Makefile footgun is an operator ergonomics issue (local `make` with DB access), not a remote authz or injection finding — out of scope for this security pass. Security review: no medium+ findings on this diff.

ghost1 commented 2026-06-12 08:21:34 +00:00 (Migrated from gitlab.com)

Copy link

changed the description

changed the description

ghost1 commented 2026-06-12 08:21:40 +00:00 (Migrated from gitlab.com)

Copy link

resolved all threads

resolved all threads

PlasticDigits commented 2026-06-12 08:35:43 +00:00 (Migrated from gitlab.com)

Copy link

added 32 commits

• ae66cddd...62050542 - 31 commits from branch main
• 52f5f10e - Merge origin/main into verify/issue-362-reorg-recovery-alerting

Compare with previous version

added 32 commits <ul><li>ae66cddd...62050542 - 31 commits from branch <code>main</code></li><li>52f5f10e - Merge origin/main into verify/issue-362-reorg-recovery-alerting</li></ul> [Compare with previous version](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/165/diffs?diff_id=1848548956&start_sha=ae66cdddca09d5c9a9ca1d93dc5af1f304ecb097)

PlasticDigits commented 2026-06-12 08:35:46 +00:00 (Migrated from gitlab.com)

Copy link

Resolved merge conflicts with main.

Code: Unified reorg_alert.rs — kept ReorgHaltDetails API + INDEXER_REORG_HALT stderr JSON from this branch, merged with main's webhook delivery and --cleanup-derived recovery path. poller.rs uses ReorgHaltDetails::new; Makefile passes APPLY=1 / CLEANUP=1 only when explicitly set.

Script: indexer-reorg-recover.sh combines main's row-impact preview + derived cleanup with compose-aware postgres-psql.sh.

Docs: Merged rate-limit secrets (#363) with reorg alerting docs (#362); runbook/incident template/invariants updated consistently.

Sanity: cargo test --lib reorg_alert, cargo test --test reorg_alert_webhook, scripts/test-indexer-reorg-recover.sh — all pass.

Resolved merge conflicts with `main`. **Code:** Unified `reorg_alert.rs` — kept `ReorgHaltDetails` API + `INDEXER_REORG_HALT` stderr JSON from this branch, merged with main's webhook delivery and `--cleanup-derived` recovery path. `poller.rs` uses `ReorgHaltDetails::new`; Makefile passes `APPLY=1` / `CLEANUP=1` only when explicitly set. **Script:** `indexer-reorg-recover.sh` combines main's row-impact preview + derived cleanup with compose-aware `postgres-psql.sh`. **Docs:** Merged rate-limit secrets (#363) with reorg alerting docs (#362); runbook/incident template/invariants updated consistently. Sanity: `cargo test --lib reorg_alert`, `cargo test --test reorg_alert_webhook`, `scripts/test-indexer-reorg-recover.sh` — all pass.

ghost1 commented 2026-06-12 08:36:03 +00:00 (Migrated from gitlab.com)

Copy link

changed the description

changed the description

PlasticDigits commented 2026-06-12 08:38:23 +00:00 (Migrated from gitlab.com)

Copy link

Security review — MR !883

Commit reviewed: 52f5f10efe22a00f7e571932b7739df67a824e06

Scope: Post-merge-conflict re-review of reorg_alert halt signaling (INDEXER_REORG_HALT stderr JSON, optional REORG_ALERT_WEBHOOK_URL webhook), verify_checkpoint_unchanged empty-hash post-recovery skip, indexer-reorg-recover.sh compose-aware psql + row-impact preview, make indexer-reorg-recover (APPLY=1 / CLEANUP=1 gating), integration tests, docs.

Outcome: FINDINGS: 0 medium+

Inline threads: none (no medium+ issues on this diff)

Summary

Re-traced attacker-controlled input paths on the updated diff (includes merge with main):

Surface	Attacker input?	Controls	Verdict
REORG_ALERT_WEBHOOK_URL / post_webhook	No — operator env only; not API-exposed	Same trust boundary as LCD_URLS / DATABASE_URL; 10s timeout; JSON body is indexer-generated (height, public block hashes, static recovery_cmd)	No SSRF/exfil path from remote callers
verify_checkpoint_unchanged empty-hash skip	No — requires operator recovery script or DB write	No HTTP handler mutates indexer_state; skip is intentional post-rewind window documented in runbook	No unauthenticated reorg-guard bypass
indexer-reorg-recover.sh SQL	--height only	Validated ^[0-9]+$, >= 1; table names hardcoded; PREV is arithmetic	No injection
Halt logs / webhook payload	Block hashes from LCD	serde_json serialization; hashes are public chain data	No secret leakage
make indexer-reorg-recover	Operator CLI	APPLY=1 / CLEANUP=1 exact-match gating (fixes prior APPLY=0 footgun)	Operator ergonomics only; not a remote authz issue

Recovery remains CLI/operator-only (no new API rewind). Swap replay idempotency unchanged (ON CONFLICT DO NOTHING).

Prior review on 8a92d564 still applies; merge-resolution changes (Makefile gating, compose psql helpers, stderr JSON/target rename) do not introduce new exploitable paths.

Security review: no medium+ findings on this diff.

## Security review — MR !883 **Commit reviewed:** `52f5f10efe22a00f7e571932b7739df67a824e06` **Scope:** Post-merge-conflict re-review of `reorg_alert` halt signaling (`INDEXER_REORG_HALT` stderr JSON, optional `REORG_ALERT_WEBHOOK_URL` webhook), `verify_checkpoint_unchanged` empty-hash post-recovery skip, `indexer-reorg-recover.sh` compose-aware psql + row-impact preview, `make indexer-reorg-recover` (`APPLY=1` / `CLEANUP=1` gating), integration tests, docs. **Outcome:** `FINDINGS: 0` medium+ **Inline threads:** none (no medium+ issues on this diff) ### Summary Re-traced attacker-controlled input paths on the updated diff (includes merge with `main`): | Surface | Attacker input? | Controls | Verdict | |---------|-----------------|----------|---------| | `REORG_ALERT_WEBHOOK_URL` / `post_webhook` | No — operator env only; not API-exposed | Same trust boundary as `LCD_URLS` / `DATABASE_URL`; 10s timeout; JSON body is indexer-generated (`height`, public block hashes, static `recovery_cmd`) | No SSRF/exfil path from remote callers | | `verify_checkpoint_unchanged` empty-hash skip | No — requires operator recovery script or DB write | No HTTP handler mutates `indexer_state`; skip is intentional post-rewind window documented in runbook | No unauthenticated reorg-guard bypass | | `indexer-reorg-recover.sh` SQL | `--height` only | Validated `^[0-9]+$`, `>= 1`; table names hardcoded; `PREV` is arithmetic | No injection | | Halt logs / webhook payload | Block hashes from LCD | `serde_json` serialization; hashes are public chain data | No secret leakage | | `make indexer-reorg-recover` | Operator CLI | `APPLY=1` / `CLEANUP=1` exact-match gating (fixes prior `APPLY=0` footgun) | Operator ergonomics only; not a remote authz issue | Recovery remains CLI/operator-only (no new API rewind). Swap replay idempotency unchanged (`ON CONFLICT DO NOTHING`). Prior review on `8a92d564` still applies; merge-resolution changes (Makefile gating, compose psql helpers, stderr JSON/target rename) do not introduce new exploitable paths. Security review: no medium+ findings on this diff.

PlasticDigits commented 2026-06-12 11:05:00 +00:00 (Migrated from gitlab.com)

Copy link

added 11 commits

• 52f5f10e...2f9c7967 - 10 commits from branch main
• ee7828f0 - Merge origin/main into verify/issue-362-reorg-recovery-alerting

Compare with previous version

added 11 commits <ul><li>52f5f10e...2f9c7967 - 10 commits from branch <code>main</code></li><li>ee7828f0 - Merge origin/main into verify/issue-362-reorg-recovery-alerting</li></ul> [Compare with previous version](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/165/diffs?diff_id=1848867448&start_sha=52f5f10efe22a00f7e571932b7739df67a824e06)

PlasticDigits commented 2026-06-12 11:05:03 +00:00 (Migrated from gitlab.com)

Copy link

Resolved merge conflicts with origin/main.

Conflict: docs/operator-secrets.md — kept the MR's REORG_ALERT_WEBHOOK_URL env row and adopted main's #363 rate-limit wording for RATE_LIMIT_RPS.

Merge commit: ee7828f. MR is mergeable again; not merged per conflict-fix workflow.

Resolved merge conflicts with `origin/main`. **Conflict:** `docs/operator-secrets.md` — kept the MR's `REORG_ALERT_WEBHOOK_URL` env row and adopted main's #363 rate-limit wording for `RATE_LIMIT_RPS`. Merge commit: `ee7828f`. MR is mergeable again; not merged per conflict-fix workflow.

ghost1 commented 2026-06-12 11:05:21 +00:00 (Migrated from gitlab.com)

Copy link

changed the description

changed the description

PlasticDigits commented 2026-06-12 11:07:07 +00:00 (Migrated from gitlab.com)

Copy link

Security review — MR !883

Commit reviewed: ee7828f0388ad031bd4e11dac3670500d7f85da4

Scope: reorg_alert halt signaling (INDEXER_REORG_HALT stderr JSON, tracing target indexer_reorg_halt, optional REORG_ALERT_WEBHOOK_URL webhook); verify_checkpoint_unchanged empty-hash post-recovery skip; indexer-reorg-recover.sh compose-aware psql helpers + row-impact preview; make indexer-reorg-recover (HEIGHT required, APPLY=1 / CLEANUP=1 exact-match gating); integration tests; docs.

Outcome: FINDINGS: 0 medium+

Inline threads: none (no medium+ issues on this diff)

Summary

Re-traced attacker-controlled input paths on the current diff (vs origin/main):

Surface	Attacker input?	Controls	Verdict
REORG_ALERT_WEBHOOK_URL / post_webhook	No — operator env only; not API-exposed	Same trust boundary as LCD_URLS / DATABASE_URL; 10s timeout; JSON body is indexer-generated (height, public block hashes, static recovery_command)	No SSRF/exfil path from remote callers
verify_checkpoint_unchanged empty-hash skip	No — requires operator recovery script or DB write	No HTTP handler mutates indexer_state; skip is intentional post-rewind window documented in runbook	No unauthenticated reorg-guard bypass
indexer-reorg-recover.sh SQL	--height only	Validated ^[0-9]+$, >= 1; table names hardcoded; PREV is arithmetic	No injection
Halt logs / webhook payload	Block hashes from LCD	serde_json serialization; hashes are public chain data	No secret leakage
make indexer-reorg-recover	Operator CLI	APPLY=1 / CLEANUP=1 exact-match gating (fixes prior APPLY=0 footgun)	Operator ergonomics only; not a remote authz issue

Recovery remains CLI/operator-only (no new API rewind). Swap replay idempotency unchanged (ON CONFLICT DO NOTHING).

Prior reviews on 8a92d564 and 52f5f10 still apply; latest merge commit adds no new exploitable surfaces beyond the scoped diff above.

Security review: no medium+ findings on this diff.

## Security review — MR !883 **Commit reviewed:** `ee7828f0388ad031bd4e11dac3670500d7f85da4` **Scope:** `reorg_alert` halt signaling (`INDEXER_REORG_HALT` stderr JSON, tracing target `indexer_reorg_halt`, optional `REORG_ALERT_WEBHOOK_URL` webhook); `verify_checkpoint_unchanged` empty-hash post-recovery skip; `indexer-reorg-recover.sh` compose-aware psql helpers + row-impact preview; `make indexer-reorg-recover` (`HEIGHT` required, `APPLY=1` / `CLEANUP=1` exact-match gating); integration tests; docs. **Outcome:** `FINDINGS: 0` medium+ **Inline threads:** none (no medium+ issues on this diff) ### Summary Re-traced attacker-controlled input paths on the current diff (vs `origin/main`): | Surface | Attacker input? | Controls | Verdict | |---------|-----------------|----------|---------| | `REORG_ALERT_WEBHOOK_URL` / `post_webhook` | No — operator env only; not API-exposed | Same trust boundary as `LCD_URLS` / `DATABASE_URL`; 10s timeout; JSON body is indexer-generated (`height`, public block hashes, static `recovery_command`) | No SSRF/exfil path from remote callers | | `verify_checkpoint_unchanged` empty-hash skip | No — requires operator recovery script or DB write | No HTTP handler mutates `indexer_state`; skip is intentional post-rewind window documented in runbook | No unauthenticated reorg-guard bypass | | `indexer-reorg-recover.sh` SQL | `--height` only | Validated `^[0-9]+$`, `>= 1`; table names hardcoded; `PREV` is arithmetic | No injection | | Halt logs / webhook payload | Block hashes from LCD | `serde_json` serialization; hashes are public chain data | No secret leakage | | `make indexer-reorg-recover` | Operator CLI | `APPLY=1` / `CLEANUP=1` exact-match gating (fixes prior `APPLY=0` footgun) | Operator ergonomics only; not a remote authz issue | Recovery remains CLI/operator-only (no new API rewind). Swap replay idempotency unchanged (`ON CONFLICT DO NOTHING`). Prior reviews on `8a92d564` and `52f5f10` still apply; latest merge commit adds no new exploitable surfaces beyond the scoped diff above. Security review: no medium+ findings on this diff.

ghost1 commented 2026-06-12 11:07:07 +00:00 (Migrated from gitlab.com)

Copy link

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issues.}

^{Reviewed by Cursor Bugbot for commit ee7828f038. Configure here.}

<!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues. <!-- BUGBOT_FIX_ALL --> <a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aTozNzRkNDY1ZS1jN2RjLTQ0MzYtOGY5NC04NjFlMjhmMDA5MTAiLCJlbmNyeXB0aW9uS2V5IjoiOE9ibGI2MTByR2lsWGtSTGRRR3RYZDhoYTNFckRnWTZXZlJ5TXlYYjF6NCIsImJyYW5jaCI6InZlcmlmeS9pc3N1ZS0zNjItcmVvcmctcmVjb3ZlcnktYWxlcnRpbmciLCJyZXBvT3duZXIiOiJQbGFzdGljRGlnaXRzIiwicmVwb05hbWUiOiJjbDh5LWRleC10ZXJyYWNsYXNzaWMifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <!-- /BUGBOT_FIX_ALL --> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> ^{Bugbot Autofix is [ON](https://www.cursor.com/dashboard/bugbot). A cloud agent has been kicked off to fix the reported issues. <!-- BUGBOT_AUTOFIX_AGENT_LINK -->} <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ee7828f0388ad031bd4e11dac3670500d7f85da4. Configure [here](https://www.cursor.com/dashboard/bugbot).}

ghost1 commented 2026-06-12 11:07:08 +00:00 (Migrated from gitlab.com)

Copy link

Upstream errors become not-found

Medium Severity

run_concurrent_candidate_evaluations now logs and skips Ok(Err(...)) from evaluate_candidate instead of aborting. When every evaluated path returns a gateway error (e.g. LCD/router 502), evals stays empty and solve_global_best_execution responds with 404 no viable route rather than propagating 502.

 

^{Reviewed by Cursor Bugbot for commit ee7828f038. Configure here.}

### Upstream errors become not-found **Medium Severity** <!-- DESCRIPTION START --> `run_concurrent_candidate_evaluations` now logs and skips `Ok(Err(...))` from `evaluate_candidate` instead of aborting. When every evaluated path returns a gateway error (e.g. LCD/router 502), `evals` stays empty and `solve_global_best_execution` responds with **404** `no viable route` rather than propagating **502**. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 3f07b1cc-4bba-49a2-ae5e-4f084a27817b --> <!-- LOCATIONS START indexer/src/api/best_execution.rs#L595-L631 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjZkZWJlYmQ0LWRiNDQtNGQ0ZC1iNDZhLWJmYjgxZDgzYjE1OSIsImVuY3J5cHRpb25LZXkiOiJXaGZWbVN1LXdWbnFfdWNzVlZVdkNoNExjUUNIUkpnclEzVk1UZEZZekxFIiwiYnJhbmNoIjoidmVyaWZ5L2lzc3VlLTM2Mi1yZW9yZy1yZWNvdmVyeS1hbGVydGluZyIsInJlcG9Pd25lciI6IlBsYXN0aWNEaWdpdHMiLCJyZXBvTmFtZSI6ImNsOHktZGV4LXRlcnJhY2xhc3NpYyJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjZkZWJlYmQ0LWRiNDQtNGQ0ZC1iNDZhLWJmYjgxZDgzYjE1OSIsImVuY3J5cHRpb25LZXkiOiJXaGZWbVN1LXdWbnFfdWNzVlZVdkNoNExjUUNIUkpnclEzVk1UZEZZekxFIiwiYnJhbmNoIjoidmVyaWZ5L2lzc3VlLTM2Mi1yZW9yZy1yZWNvdmVyeS1hbGVydGluZyIsInJlcG9Pd25lciI6IlBsYXN0aWNEaWdpdHMiLCJyZXBvTmFtZSI6ImNsOHktZGV4LXRlcnJhY2xhc3NpYyIsInByTnVtYmVyIjoxNjUsImNvbW1pdFNoYSI6ImVlNzgyOGYwMzg4YWQwMzFiZDRlMTFkYWMzNjcwNTAwZDdmODVkYTQiLCJwcm92aWRlciI6ImdpdGxhYiIsImhvc3RuYW1lIjoiZ2l0bGFiLmNvbSJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ee7828f0388ad031bd4e11dac3670500d7f85da4. Configure [here](https://www.cursor.com/dashboard/bugbot).}

ghost1 commented 2026-06-12 11:07:09 +00:00 (Migrated from gitlab.com)

Copy link

Missing mirror skips LCD fallback

High Severity

The new zero-reserve short-circuit in query_hybrid_sim_unified treats any mirror with reserve_0 == 0 or reserve_1 == 0 as zero hop output. preload_mirrors_for_pairs uses the same zero reserves for MirrorFreshness::MissingReserves placeholders when no snapshot row exists, so those hops now return Ok(0) instead of reaching the existing LCD fallback path.

 

^{Reviewed by Cursor Bugbot for commit ee7828f038. Configure here.}

### Missing mirror skips LCD fallback **High Severity** <!-- DESCRIPTION START --> The new zero-reserve short-circuit in `query_hybrid_sim_unified` treats any mirror with `reserve_0 == 0` or `reserve_1 == 0` as zero hop output. `preload_mirrors_for_pairs` uses the same zero reserves for `MirrorFreshness::MissingReserves` placeholders when no snapshot row exists, so those hops now return `Ok(0)` instead of reaching the existing LCD fallback path. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 81836cc4-58da-4ab7-ac43-1c93a739c622 --> <!-- LOCATIONS START indexer/src/api/hybrid_route_opt.rs#L246-L250 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjBhZjYwODg5LWM1ODYtNDNhNy05ODJmLTBkMGRmMGVhM2I0YiIsImVuY3J5cHRpb25LZXkiOiJXYi1zaFJCMzJkUkxZNGtUXzBkM1RJQlk5WFdqWWNxZXBWZ1hWQjlNZUJzIiwiYnJhbmNoIjoidmVyaWZ5L2lzc3VlLTM2Mi1yZW9yZy1yZWNvdmVyeS1hbGVydGluZyIsInJlcG9Pd25lciI6IlBsYXN0aWNEaWdpdHMiLCJyZXBvTmFtZSI6ImNsOHktZGV4LXRlcnJhY2xhc3NpYyJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjBhZjYwODg5LWM1ODYtNDNhNy05ODJmLTBkMGRmMGVhM2I0YiIsImVuY3J5cHRpb25LZXkiOiJXYi1zaFJCMzJkUkxZNGtUXzBkM1RJQlk5WFdqWWNxZXBWZ1hWQjlNZUJzIiwiYnJhbmNoIjoidmVyaWZ5L2lzc3VlLTM2Mi1yZW9yZy1yZWNvdmVyeS1hbGVydGluZyIsInJlcG9Pd25lciI6IlBsYXN0aWNEaWdpdHMiLCJyZXBvTmFtZSI6ImNsOHktZGV4LXRlcnJhY2xhc3NpYyIsInByTnVtYmVyIjoxNjUsImNvbW1pdFNoYSI6ImVlNzgyOGYwMzg4YWQwMzFiZDRlMTFkYWMzNjcwNTAwZDdmODVkYTQiLCJwcm92aWRlciI6ImdpdGxhYiIsImhvc3RuYW1lIjoiZ2l0bGFiLmNvbSJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> ^{Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ee7828f0388ad031bd4e11dac3670500d7f85da4. Configure [here](https://www.cursor.com/dashboard/bugbot).}

ghost1 commented 2026-06-12 11:10:53 +00:00 (Migrated from gitlab.com)

Copy link

added 1 commit

• 2ac33b06 - fix(indexer): restore LCD fallback and gateway errors on route solve

Compare with previous version

added 1 commit <ul><li>2ac33b06 - fix(indexer): restore LCD fallback and gateway errors on route solve</li></ul> [Compare with previous version](/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/165/diffs?diff_id=1848878957&start_sha=ee7828f0388ad031bd4e11dac3670500d7f85da4)

ghost1 commented 2026-06-12 11:11:10 +00:00 (Migrated from gitlab.com)

Copy link

changed the description

changed the description

PlasticDigits commented 2026-06-12 11:14:02 +00:00 (Migrated from gitlab.com)

Copy link

mentioned in commit 0cf04e530e

mentioned in commit 0cf04e530e17ab1c96866a199dec962d92b01c45

PlasticDigits (Migrated from gitlab.com) merged commit 0cf04e530e into main 2026-06-12 11:14:02 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

agent:fix_bugfix

  

agent:fix_conflicts

  

agent:fix_security

  

agent:gap_analysis

  

agent:implement

  

agent:implement

Queue implement gate

  

agent:implement

Queue implement gate

  

agent:open_issues

  

agent:ready

  

agent:research

  

agent:security_audit

  

agent:verify

  

architecture

Architecture

  

backend

API and services

  

blocker:hybrid

Blocks hybrid launch

  

blocker:launch

Blocks launch

  

blocker:limit-orders

Blocks limits launch

  

blocker:v2

Blocks v2

  

block:log_only

  

block:security

  

bug

  

ci

  

contracts

Smart contracts

  

correctness

  

deploy

  

dev

  

devops

  

docs

  

documentation

  

duplicate

  

e2e

  

enhancement

  

epic

Epic

  

feature

  

frontend

  

functional-completion

Functional

  

gas

  

good first issue

  

governance

  

help wanted

  

high-risk

  

hooks

  

hybrid

Hybrid swaps

  

indexer

Indexer

  

infra

CI/CD

  

infrastructure

  

integrators

  

invalid

  

launch-blocker

  

limit-orders

Limit orders

  

localnet

  

localterra

  

low priority

  

missing-implementation

Implementation

  

needs-design

Design decision required

  

ops

  

performance

  

priority

high

  

priority

medium

  

product

  

qa

  

QA

  

question

  

ready

Definition of Ready

  

ready

Definition of Ready

  

research

  

scripts

  

security

Security

  

security-hardening

Security hardening

  

smartcontracts

  

tech-debt

  

testing

Testing

  

ux

  

UX

  

v2

Pool-only swaps

  

verification

  

wontfix

No labels

agent:fix_bugfix

agent:fix_conflicts

agent:fix_security

agent:gap_analysis

agent:implement

agent:implement

agent:implement

agent:open_issues

agent:ready

agent:research

agent:security_audit

agent:verify

architecture

backend

blocker:hybrid

blocker:launch

blocker:limit-orders

blocker:v2

block:log_only

block:security

bug

ci

contracts

correctness

deploy

dev

devops

docs

documentation

duplicate

e2e

enhancement

epic

feature

frontend

functional-completion

gas

good first issue

governance

help wanted

high-risk

hooks

hybrid

indexer

infra

infrastructure

integrators

invalid

launch-blocker

limit-orders

localnet

localterra

low priority

missing-implementation

needs-design

ops

performance

priority

high

priority

medium

product

qa

QA

question

ready

ready

research

scripts

security

security-hardening

smartcontracts

tech-debt

testing

ux

UX

v2

verification

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

code/cl8y-dex-terraclassic!883

No description provided.