code/cl8y-dex-terraclassic

Fork 0

Security: no rollback/forward-fix decision tree covering frontend, indexer, contract, and chain dependency incident types [SEC-H09] #445

New issue

Closed

opened 2026-06-30 10:23:14 +00:00 by totdking · 8 comments

totdking commented

2026-06-30 10:23:14 +00:00

(Migrated from gitlab.com)

No description provided.

totdking commented

2026-06-30 10:24:05 +00:00

(Migrated from gitlab.com)

Summary

No runbook in docs/runbooks/ documents a rollback or forward-fix decision tree covering the four incident types required by the checklist. docs/runbooks/launch-checklist.md rollback section (2 lines) says only "pause via factory" and "CosmWasm upgrades are out of band." docs/runbooks/wasm-admin-migration.md covers migration steps but not rollback decisions. Neither document tells an operator: when to rollback vs forward-fix for a frontend bug, how to rollback an indexer to a previous binary and run down.sql, when a contract issue requires migration back vs emergency pause and manual recovery, or how to handle a chain dependency issue (chain upgrade, LCD provider outage, IBC-hooks vulnerability). The four incident types are entirely absent from the incident runbooks as a structured decision surface.

What Was Checked

docs/runbooks/launch-checklist.md lines 177-181: Rollback/incident section. Full text: "CosmWasm upgrades/migrations are out of band for this runbook; document admin keys and wasm migration policy separately. For live incidents: pause via factory if your governance policy allows; communicate hook/pause behavior per security model." No branching by incident type.
docs/runbooks/wasm-admin-migration.md read in full: no Rollback section, no decision criteria for when to rollback vs patch forward.
docs/runbooks/emergency-commands.md read in full: command cookbook for pause, unpause, blacklist, unblacklist. No rollback decision tree.
docs/runbooks/blacklist-decision.md: decision tree for when to apply blacklist. Not a deployment rollback decision tree.
Grep for "rollback" across all docs/runbooks/ files: launch-checklist.md (2-line section), blacklist-decision.md (false-positive rollback for unblacklist only), hook-registration.md (hook revert is atomic rollback -- unrelated), testing.md (down.sql reference). No rollback/forward-fix decision tree for the four required incident types.

Expected (per checklist)

A runbook section documents a rollback/forward-fix decision tree covering:

Frontend-only issue (e.g., broken build, wrong env address): decision criteria (hotfix vs revert to prior static build), rollback path (redeploy previous build artifact), recovery verification
Indexer issue (e.g., incorrect data, crashed binary): decision criteria (restart vs rollback binary), rollback path (restart indexer with prior release binary, run down.sql if migration is involved), recovery verification
Contract issue (e.g., logic bug discovered post-deploy): decision criteria (forward-fix migration vs emergency pause and wait vs migrate back to prior code ID), limitations (code ID must still be on-chain, admin must be retained), recovery steps
Chain dependency issue (e.g., chain upgrade breaks compatibility, IBC-hooks patch required): decision criteria (wait for chain upgrade vs pause all pairs vs update LCD provider), coordination path, recovery verification

Actual

No such decision tree exists. The launch-checklist.md rollback section is 2 lines and points to "pause via factory" for live incidents with no further branching by incident type.

Evidence

docs/runbooks/launch-checklist.md lines 177-181: full rollback section is 2 lines with no incident-type branching
No grep match for rollback decision tree content covering all four incident types across any runbook file

Suggested Fix

Add a Rollback and forward-fix decision tree section to docs/runbooks/launch-checklist.md (or a new docs/runbooks/rollback-decision.md) covering the four incident types with: symptom or trigger conditions for each type; decision criteria (rollback vs hotfix vs pause-and-wait); rollback path for each type with specific commands; limitations that prevent rollback; and recovery verification steps. Cross-link from launch-checklist.md rollback section, wasm-admin-migration.md, and emergency-commands.md.

Verification Checklist

A runbook section documents rollback/forward-fix criteria and steps for: frontend-only, indexer, contract, and chain dependency incident types
Each type includes decision criteria, rollback path with commands, limitations, and recovery verification
The section is cross-linked from docs/runbooks/launch-checklist.md rollback section

Labels

security, pre-launch

Cc: @PlasticDigits

### Summary No runbook in `docs/runbooks/` documents a rollback or forward-fix decision tree covering the four incident types required by the checklist. `docs/runbooks/launch-checklist.md` rollback section (2 lines) says only "pause via factory" and "CosmWasm upgrades are out of band." `docs/runbooks/wasm-admin-migration.md` covers migration steps but not rollback decisions. Neither document tells an operator: when to rollback vs forward-fix for a frontend bug, how to rollback an indexer to a previous binary and run down.sql, when a contract issue requires migration back vs emergency pause and manual recovery, or how to handle a chain dependency issue (chain upgrade, LCD provider outage, IBC-hooks vulnerability). The four incident types are entirely absent from the incident runbooks as a structured decision surface. --- ### What Was Checked - `docs/runbooks/launch-checklist.md` lines 177-181: Rollback/incident section. Full text: "CosmWasm upgrades/migrations are out of band for this runbook; document admin keys and wasm migration policy separately. For live incidents: pause via factory if your governance policy allows; communicate hook/pause behavior per security model." No branching by incident type. - `docs/runbooks/wasm-admin-migration.md` read in full: no Rollback section, no decision criteria for when to rollback vs patch forward. - `docs/runbooks/emergency-commands.md` read in full: command cookbook for pause, unpause, blacklist, unblacklist. No rollback decision tree. - `docs/runbooks/blacklist-decision.md`: decision tree for when to apply blacklist. Not a deployment rollback decision tree. - Grep for "rollback" across all docs/runbooks/ files: launch-checklist.md (2-line section), blacklist-decision.md (false-positive rollback for unblacklist only), hook-registration.md (hook revert is atomic rollback -- unrelated), testing.md (down.sql reference). No rollback/forward-fix decision tree for the four required incident types. --- ### Expected (per checklist) A runbook section documents a rollback/forward-fix decision tree covering: - Frontend-only issue (e.g., broken build, wrong env address): decision criteria (hotfix vs revert to prior static build), rollback path (redeploy previous build artifact), recovery verification - Indexer issue (e.g., incorrect data, crashed binary): decision criteria (restart vs rollback binary), rollback path (restart indexer with prior release binary, run down.sql if migration is involved), recovery verification - Contract issue (e.g., logic bug discovered post-deploy): decision criteria (forward-fix migration vs emergency pause and wait vs migrate back to prior code ID), limitations (code ID must still be on-chain, admin must be retained), recovery steps - Chain dependency issue (e.g., chain upgrade breaks compatibility, IBC-hooks patch required): decision criteria (wait for chain upgrade vs pause all pairs vs update LCD provider), coordination path, recovery verification --- ### Actual No such decision tree exists. The launch-checklist.md rollback section is 2 lines and points to "pause via factory" for live incidents with no further branching by incident type. --- ### Evidence - `docs/runbooks/launch-checklist.md` lines 177-181: full rollback section is 2 lines with no incident-type branching - No grep match for rollback decision tree content covering all four incident types across any runbook file --- ### Suggested Fix Add a **Rollback and forward-fix decision tree** section to `docs/runbooks/launch-checklist.md` (or a new `docs/runbooks/rollback-decision.md`) covering the four incident types with: symptom or trigger conditions for each type; decision criteria (rollback vs hotfix vs pause-and-wait); rollback path for each type with specific commands; limitations that prevent rollback; and recovery verification steps. Cross-link from launch-checklist.md rollback section, wasm-admin-migration.md, and emergency-commands.md. --- ### Verification Checklist - [ ] A runbook section documents rollback/forward-fix criteria and steps for: frontend-only, indexer, contract, and chain dependency incident types - [ ] Each type includes decision criteria, rollback path with commands, limitations, and recovery verification - [ ] The section is cross-linked from `docs/runbooks/launch-checklist.md` rollback section --- ### Labels `security`, `pre-launch` Cc: @PlasticDigits

totdking commented

2026-06-30 10:24:13 +00:00

(Migrated from gitlab.com)

mentioned in issue #381

PlasticDigits commented

2026-06-30 11:03:33 +00:00

(Migrated from gitlab.com)

mentioned in commit d5a509b73c

mentioned in commit d5a509b73c66a7ca367808ddfe0a983b9fd40042

PlasticDigits commented

2026-06-30 11:03:43 +00:00

(Migrated from gitlab.com)

mentioned in merge request !976

Brouie commented

2026-06-30 11:19:07 +00:00

(Migrated from gitlab.com)

Went through the SEC-H09 rollback/forward-fix decision tree (d5a509b7, !976). Solid runbook — checked the content and the commands, not just the structure.

make verify-issue-445 → 9/9 PASS. The new docs/runbooks/rollback-decision.md covers all four incident surfaces (frontend / indexer / contract / chain dependency), each with decision criteria, a rollback path with concrete commands, limitations, and recovery verification, plus the top-level mermaid decision tree. Cross-links are wired from launch-checklist, wasm-admin-migration, emergency-commands, the incident template, and security-model.

Spot-checked the load-bearing commands against the actual repo, since a wrong command mid-incident is worse than none:

the indexer down.sql it cites (20260509160000_limit_order_placement_lifecycle.down.sql) and the other two revert files exist as named
scripts/indexer-reorg-recover.sh exists and takes exactly --height / --cleanup-derived / --apply
the emergency-commands anchors resolve (#1-pause-a-pair, #2-unpause-a-pair, #quick-pool-triage-sec-g03)
factory set_pair_paused / wasm migrate flows match how we actually call terrad

Two things for you before it lands:

cannot_be_merged right now — same Makefile .PHONY/target-list conflict as !975 (both append targets to the same line, and main moved on since the branch was cut). Just a rebase; note the two MRs also collide with each other there.
minor: the testing.md cross-link uses anchor #frontend-integration-tests-charts--indexer, but the actual heading is "### Integration Tests (Frontend)" (#integration-tests-frontend) — the file resolves, the fragment doesn't jump. Worth fixing the anchor on the rebase.

Content's good to merge / good to close from my side once rebased.

@PlasticDigits

Went through the SEC-H09 rollback/forward-fix decision tree (d5a509b7, !976). Solid runbook — checked the content and the commands, not just the structure. `make verify-issue-445` → 9/9 PASS. The new docs/runbooks/rollback-decision.md covers all four incident surfaces (frontend / indexer / contract / chain dependency), each with decision criteria, a rollback path with concrete commands, limitations, and recovery verification, plus the top-level mermaid decision tree. Cross-links are wired from launch-checklist, wasm-admin-migration, emergency-commands, the incident template, and security-model. Spot-checked the load-bearing commands against the actual repo, since a wrong command mid-incident is worse than none: - the indexer down.sql it cites (20260509160000_limit_order_placement_lifecycle.down.sql) and the other two revert files exist as named - scripts/indexer-reorg-recover.sh exists and takes exactly --height / --cleanup-derived / --apply - the emergency-commands anchors resolve (#1-pause-a-pair, #2-unpause-a-pair, #quick-pool-triage-sec-g03) - factory set_pair_paused / wasm migrate flows match how we actually call terrad Two things for you before it lands: - cannot_be_merged right now — same Makefile .PHONY/target-list conflict as !975 (both append targets to the same line, and main moved on since the branch was cut). Just a rebase; note the two MRs also collide with each other there. - minor: the testing.md cross-link uses anchor `#frontend-integration-tests-charts--indexer`, but the actual heading is "### Integration Tests (Frontend)" (`#integration-tests-frontend`) — the file resolves, the fragment doesn't jump. Worth fixing the anchor on the rebase. Content's good to merge / good to close from my side once rebased. @PlasticDigits

PlasticDigits commented

2026-06-30 11:22:04 +00:00

(Migrated from gitlab.com)

mentioned in commit 9313da2fb4

mentioned in commit 9313da2fb45d92a400c170de52c937961ef49082

PlasticDigits commented

2026-06-30 11:23:31 +00:00

(Migrated from gitlab.com)

mentioned in commit 6d945d81a8

mentioned in commit 6d945d81a8a249eb34fc39e1b97946f03504c1ce

PlasticDigits commented

2026-06-30 11:43:05 +00:00

(Migrated from gitlab.com)

Verification — #445 (SEC-H09)

Verified on main @ 96f0f851 (includes merge 6d945d81 — rollback decision tree).

Acceptance criteria

Item	Result	How verified
Runbook documents rollback/forward-fix for frontend, indexer, contract, and chain dependency	PASS	`docs/runbooks/rollback-decision.md` — four `## N.` sections with mermaid top-level tree
Each type: decision criteria, rollback path (commands), limitations, recovery verification	PASS	Manual read of all four sections; grep confirms `### Decision criteria`, `### Rollback path (commands)`, `### Limitations`, `### Recovery verification`
Cross-linked from `launch-checklist.md` rollback section	PASS	`## Rollback / incident` links to `rollback-decision.md`; also cross-linked from `wasm-admin-migration.md`, `emergency-commands.md`, `incident-dex-indexer.md`, `security-model.md`, `skills/AGENTS_ROLLBACK_DECISION.md`

Automated

make verify-issue-445 → 9/9 PASS
make check-rollback-decision-docs → OK

Manual spot-checks (load-bearing commands)

Check	Result
Cited `indexer/migrations/revert/*.down.sql` files exist (incl. `20260509160000_limit_order_placement_lifecycle.down.sql`)	PASS
`scripts/indexer-reorg-recover.sh` accepts `--height`, `--cleanup-derived`, `--apply`	PASS
`emergency-commands.md` anchors `#1-pause-a-pair`, `#2-unpause-a-pair`, `#quick-pool-triage-sec-g03` resolve	PASS
Factory `set_pair_paused` / `terrad tx wasm migrate` patterns match repo conventions	PASS

Follow-up (non-blocking)

rollback-decision.md links to docs/testing.md#frontend-integration-tests-charts--indexer but the heading is ### Integration Tests (Frontend) (#integration-tests-frontend). File resolves; fragment does not jump. Worth a one-line anchor fix in a future docs pass.

Overall: PASS — closing #445.

## Verification — #445 (SEC-H09) Verified on `main` @ `96f0f851` (includes merge `6d945d81` — rollback decision tree). ### Acceptance criteria | Item | Result | How verified | |------|--------|--------------| | Runbook documents rollback/forward-fix for frontend, indexer, contract, and chain dependency | **PASS** | `docs/runbooks/rollback-decision.md` — four `## N.` sections with mermaid top-level tree | | Each type: decision criteria, rollback path (commands), limitations, recovery verification | **PASS** | Manual read of all four sections; grep confirms `### Decision criteria`, `### Rollback path (commands)`, `### Limitations`, `### Recovery verification` | | Cross-linked from `launch-checklist.md` rollback section | **PASS** | `## Rollback / incident` links to `rollback-decision.md`; also cross-linked from `wasm-admin-migration.md`, `emergency-commands.md`, `incident-dex-indexer.md`, `security-model.md`, `skills/AGENTS_ROLLBACK_DECISION.md` | ### Automated ``` make verify-issue-445 → 9/9 PASS make check-rollback-decision-docs → OK ``` ### Manual spot-checks (load-bearing commands) | Check | Result | |-------|--------| | Cited `indexer/migrations/revert/*.down.sql` files exist (incl. `20260509160000_limit_order_placement_lifecycle.down.sql`) | **PASS** | | `scripts/indexer-reorg-recover.sh` accepts `--height`, `--cleanup-derived`, `--apply` | **PASS** | | `emergency-commands.md` anchors `#1-pause-a-pair`, `#2-unpause-a-pair`, `#quick-pool-triage-sec-g03` resolve | **PASS** | | Factory `set_pair_paused` / `terrad tx wasm migrate` patterns match repo conventions | **PASS** | ### Follow-up (non-blocking) - `rollback-decision.md` links to `docs/testing.md#frontend-integration-tests-charts--indexer` but the heading is `### Integration Tests (Frontend)` (`#integration-tests-frontend`). File resolves; fragment does not jump. Worth a one-line anchor fix in a future docs pass. **Overall: PASS — closing #445.**

PlasticDigits (Migrated from gitlab.com) closed this issue

2026-06-30 11:43:06 +00:00