code/cl8y-dex-terraclassic

Fork 0

Security: no monitoring runbook listing commands or log queries for launch signals [SEC-G01] #434

New issue

Closed

opened 2026-06-29 16:07:51 +00:00 by totdking · 11 comments

totdking commented

2026-06-29 16:07:51 +00:00

(Migrated from gitlab.com)

No description provided.

totdking commented

2026-06-29 16:08:21 +00:00

(Migrated from gitlab.com)

Summary

The repo has runbooks for emergency response (pause/blacklist), reorg recovery, and blacklist decisions, but no runbook defines minimum launch monitoring for the signals listed in SEC-G01. The only proactive alerting documented is for reorg halt (INDEXER_REORG_HALT stderr prefix and REORG_ALERT_WEBHOOK_URL), covered in docs/operator-secrets.md and docs/runbooks/indexer-reorg-replay-dedup.md. No runbook lists commands, log queries, or dashboards for: contract error spikes, indexer lag (behind chain tip), API 429/5xx rate, large swap detection, large LP withdrawal detection, blacklist hits, or pause state changes.

What Was Checked

docs/operator-secrets.md line 42-44: documents that there is no Prometheus /metrics endpoint; directs operators to use tracing logs; covers only reorg halt alerting (INDEXER_REORG_HALT, REORG_ALERT_WEBHOOK_URL).
docs/runbooks/indexer-reorg-replay-dedup.md: covers reorg halt alerting and recovery. Does not cover indexer lag, API errors, or swap size monitoring.
docs/runbooks/emergency-commands.md: covers pause/blacklist commands for triage. No proactive monitoring queries.
docs/templates/incident-dex-indexer.md: covers incident triage checklist. Does not define pre-incident monitoring signals.
All other runbooks and docs files searched: no monitoring runbook or command list found for the required signals.

Expected (per checklist)

A runbook must list commands, log queries, or dashboard entries covering at minimum: contract error spikes, indexer lag behind chain tip, API 429/5xx rate, large swaps, large LP withdrawals, blacklist hits, and pause state changes. These define the operator's eyes on the system at launch.

Actual

Only reorg halt alerting is documented. The remaining six signal categories (contract errors, indexer lag, API error rate, swap/LP size, blacklist hits, pause changes) have no documented monitoring commands or queries. An operator responding to a report would have no runbook to consult for rapid signal triage.

Evidence

docs/operator-secrets.md lines 42-44: only reorg halt alerting documented
No monitoring runbook found under docs/runbooks/ covering contract errors, indexer lag, 429/5xx, swap/LP size, blacklist hits, or pause state changes
docs/runbooks/indexer-reorg-replay-dedup.md: reorg only, not general indexer health

Suggested Fix

Create docs/runbooks/launch-monitoring.md listing: (1) log query for contract execution errors from tracing output; (2) indexer lag check (compare last_indexed_height from indexer DB against chain tip via LCD); (3) API rate query for 429 responses (from access log or tracing tower_governor events); (4) SQL query for swaps or LP events above a size threshold; (5) indexer API call or SQL query for blacklist hits against the compliance endpoint; (6) on-chain query for current pair pause state. Each entry should include the command, expected baseline output, and escalation signal.

Verification Checklist

A runbook or section in an existing runbook lists log queries or commands for: contract error spikes, indexer lag, API 429/5xx, large swaps, large LP withdrawals, blacklist hits, pause state changes
Each entry includes the command to run, the baseline expectation, and the escalation threshold

Labels

security, pre-launch

Cc: @PlasticDigits

### Summary The repo has runbooks for emergency response (pause/blacklist), reorg recovery, and blacklist decisions, but no runbook defines minimum launch monitoring for the signals listed in SEC-G01. The only proactive alerting documented is for reorg halt (`INDEXER_REORG_HALT` stderr prefix and `REORG_ALERT_WEBHOOK_URL`), covered in `docs/operator-secrets.md` and `docs/runbooks/indexer-reorg-replay-dedup.md`. No runbook lists commands, log queries, or dashboards for: contract error spikes, indexer lag (behind chain tip), API 429/5xx rate, large swap detection, large LP withdrawal detection, blacklist hits, or pause state changes. --- ### What Was Checked - `docs/operator-secrets.md` line 42-44: documents that there is no Prometheus `/metrics` endpoint; directs operators to use `tracing` logs; covers only reorg halt alerting (`INDEXER_REORG_HALT`, `REORG_ALERT_WEBHOOK_URL`). - `docs/runbooks/indexer-reorg-replay-dedup.md`: covers reorg halt alerting and recovery. Does not cover indexer lag, API errors, or swap size monitoring. - `docs/runbooks/emergency-commands.md`: covers pause/blacklist commands for triage. No proactive monitoring queries. - `docs/templates/incident-dex-indexer.md`: covers incident triage checklist. Does not define pre-incident monitoring signals. - All other runbooks and docs files searched: no monitoring runbook or command list found for the required signals. --- ### Expected (per checklist) A runbook must list commands, log queries, or dashboard entries covering at minimum: contract error spikes, indexer lag behind chain tip, API 429/5xx rate, large swaps, large LP withdrawals, blacklist hits, and pause state changes. These define the operator's eyes on the system at launch. --- ### Actual Only reorg halt alerting is documented. The remaining six signal categories (contract errors, indexer lag, API error rate, swap/LP size, blacklist hits, pause changes) have no documented monitoring commands or queries. An operator responding to a report would have no runbook to consult for rapid signal triage. --- ### Evidence - `docs/operator-secrets.md` lines 42-44: only reorg halt alerting documented - No monitoring runbook found under `docs/runbooks/` covering contract errors, indexer lag, 429/5xx, swap/LP size, blacklist hits, or pause state changes - `docs/runbooks/indexer-reorg-replay-dedup.md`: reorg only, not general indexer health --- ### Suggested Fix Create `docs/runbooks/launch-monitoring.md` listing: (1) log query for contract execution errors from tracing output; (2) indexer lag check (compare `last_indexed_height` from indexer DB against chain tip via LCD); (3) API rate query for 429 responses (from access log or tracing `tower_governor` events); (4) SQL query for swaps or LP events above a size threshold; (5) indexer API call or SQL query for blacklist hits against the compliance endpoint; (6) on-chain query for current pair pause state. Each entry should include the command, expected baseline output, and escalation signal. --- ### Verification Checklist - [ ] A runbook or section in an existing runbook lists log queries or commands for: contract error spikes, indexer lag, API 429/5xx, large swaps, large LP withdrawals, blacklist hits, pause state changes - [ ] Each entry includes the command to run, the baseline expectation, and the escalation threshold --- ### Labels `security`, `pre-launch` Cc: @PlasticDigits

totdking commented

2026-06-29 16:15:51 +00:00

(Migrated from gitlab.com)

mentioned in issue #381

Brouie commented

2026-06-29 16:19:21 +00:00

(Migrated from gitlab.com)

mentioned in merge request !964

Brouie commented

2026-06-29 16:19:44 +00:00

(Migrated from gitlab.com)

Took this one — no agent was on it (no label, no MR, no branch), so I built the runbook + a docs-check (MR !964).

What's there:

docs/runbooks/launch-monitoring.md — every signal you listed gets a tested command, a healthy baseline, and an escalation trigger: contract/index error spikes, indexer lag vs chain tip, API 429/5xx, large swaps, large LP withdrawals, blacklist hits, pause-state changes, plus the existing reorg halt (signal 8). Every command was actually run against the live stack, so the baselines are real — indexer lag 6, swap_events p99 return 442,197,580, indexer_failed_blocks 0, the liquidity_events add/remove split, the blacklist-check endpoint shape, the is_paused query.
scripts/check_launch_monitoring_docs.py + make check-launch-monitoring-docs (wired into make lint) so the runbook can't silently lose a signal — adversarially confirmed it fails if a section is dropped.
A Phase-4 item in launch-checklist.md.

One thing worth flagging that I hit while validating: a naive grep -ci error on the indexer log returns ~7100 false positives (the substring "Parse error" inside benign oracle WARNs), while the real ERROR-level count was 4 — and those 4 were exactly the reorg-halt events. So the runbook tells operators to grep the level token after stripping ANSI, not the bare word; same trap on "429".

Good to close from my side once !964 merges. @PlasticDigits

Took this one — no agent was on it (no label, no MR, no branch), so I built the runbook + a docs-check (MR !964). What's there: - docs/runbooks/launch-monitoring.md — every signal you listed gets a tested command, a healthy baseline, and an escalation trigger: contract/index error spikes, indexer lag vs chain tip, API 429/5xx, large swaps, large LP withdrawals, blacklist hits, pause-state changes, plus the existing reorg halt (signal 8). Every command was actually run against the live stack, so the baselines are real — indexer lag 6, swap_events p99 return 442,197,580, indexer_failed_blocks 0, the liquidity_events add/remove split, the blacklist-check endpoint shape, the is_paused query. - scripts/check_launch_monitoring_docs.py + make check-launch-monitoring-docs (wired into make lint) so the runbook can't silently lose a signal — adversarially confirmed it fails if a section is dropped. - A Phase-4 item in launch-checklist.md. One thing worth flagging that I hit while validating: a naive grep -ci error on the indexer log returns ~7100 false positives (the substring "Parse error" inside benign oracle WARNs), while the real ERROR-level count was 4 — and those 4 were exactly the reorg-halt events. So the runbook tells operators to grep the level token after stripping ANSI, not the bare word; same trap on "429". Good to close from my side once !964 merges. @PlasticDigits

PlasticDigits commented

2026-06-29 16:26:07 +00:00

(Migrated from gitlab.com)

mentioned in commit 131058d111

mentioned in commit 131058d111fdcae7f32be3d1ea9c52f8eded2e17

Brouie commented

2026-06-30 02:59:03 +00:00

(Migrated from gitlab.com)

Re-ran my checks on !964 — still clean.

check_launch_monitoring_docs.py -> OK (covers all 8 launch signals)
ran the runbook's baseline queries against the live indexer: indexer_failed_blocks=0, swap_events 9984 / max return 2,963,621,042, liquidity add=421 / remove=1, last_indexed_height 467989 vs tip 467999 (lag 10), /health 200 — all match the baselines the runbook states.

Branch is mergeable, no conflicts. Nothing left on my side — @PlasticDigits good to merge !964 + close #434.

Re-ran my checks on !964 — still clean. - check_launch_monitoring_docs.py -> OK (covers all 8 launch signals) - ran the runbook's baseline queries against the live indexer: indexer_failed_blocks=0, swap_events 9984 / max return 2,963,621,042, liquidity add=421 / remove=1, last_indexed_height 467989 vs tip 467999 (lag 10), /health 200 — all match the baselines the runbook states. Branch is mergeable, no conflicts. Nothing left on my side — @PlasticDigits good to merge !964 + close #434.

PlasticDigits commented

2026-06-30 03:52:24 +00:00

(Migrated from gitlab.com)

mentioned in commit 32063409d0

mentioned in commit 32063409d0caea28c66b3a142e9199a10c972c88

PlasticDigits commented

2026-06-30 05:08:11 +00:00

(Migrated from gitlab.com)

mentioned in commit 11666a0059

mentioned in commit 11666a0059604777c00023628d12ed3e86ad6cc3

PlasticDigits commented

2026-06-30 05:08:14 +00:00

(Migrated from gitlab.com)

mentioned in merge request !969

PlasticDigits commented

2026-06-30 05:08:33 +00:00

(Migrated from gitlab.com)

mentioned in merge request !968

PlasticDigits commented

2026-06-30 07:05:01 +00:00

(Migrated from gitlab.com)

Verification complete — PASS

Verified SEC-G01 / #434 on main after merge of !964 (e3806af0).

Acceptance criteria

Item	Result	How verified
Runbook lists commands/queries for all 7 required launch signals + reorg halt	PASS	`docs/runbooks/launch-monitoring.md` sections 1–8; `make check-launch-monitoring-docs` → `OK: … covers all 8 launch signals`
Each entry has command, baseline, escalation threshold	PASS	All 8 sections contain fenced command blocks plus `Baseline:` and `Escalate if:` lines
Docs guardrail wired into lint	PASS	`make check-launch-monitoring-docs` in `Makefile` `lint` target; adversarial removal of pause section → exit 1
Launch checklist cross-link	PASS	`docs/runbooks/launch-checklist.md` Phase 4 item references runbook + `make check-launch-monitoring-docs`

Live-stack command run (fresh `make setup-cloud-localterra --fresh --skip-build`)

Signal	Command output (healthy)
1 — errors	`indexer_failed_blocks=0`, `ERROR`-level count `0`
2 — lag	`chain_tip=1818`, `last_indexed=1813`, `lag=5` (single-digit)
3 — API 429/5xx	`/health` `200`; 20-request burst all `200`
4 — large swaps	`swap_events` count `60`, max return `776306256`, above-threshold `0` at `THRESH=2e9`
5 — large LP withdrawals	add `26`, remove `1`, above-threshold remove `1` at `THRESH_LP=1e9`
6 — blacklist	`{"blocked":false,…}` for test address; no `BLACKLIST HIT` lines
7 — pause	`{"paused":false}` via `terrad query … is_paused`
8 — reorg halt	`0` `INDEXER_REORG_HALT` / `indexer_reorg_halt` lines in indexer log

Quick sweep also ran clean (lag 5, failed_blocks 0, ERROR-level 0, reorg-halt 0).

No repo changes required from this verification pass. Closing #434.

## Verification complete — PASS Verified SEC-G01 / #434 on `main` after merge of !964 (`e3806af0`). ### Acceptance criteria | Item | Result | How verified | |------|--------|--------------| | Runbook lists commands/queries for all 7 required launch signals + reorg halt | **PASS** | `docs/runbooks/launch-monitoring.md` sections 1–8; `make check-launch-monitoring-docs` → `OK: … covers all 8 launch signals` | | Each entry has command, baseline, escalation threshold | **PASS** | All 8 sections contain fenced command blocks plus `**Baseline:**` and `**Escalate if:**` lines | | Docs guardrail wired into lint | **PASS** | `make check-launch-monitoring-docs` in `Makefile` `lint` target; adversarial removal of pause section → exit 1 | | Launch checklist cross-link | **PASS** | `docs/runbooks/launch-checklist.md` Phase 4 item references runbook + `make check-launch-monitoring-docs` | ### Live-stack command run (fresh `make setup-cloud-localterra --fresh --skip-build`) | Signal | Command output (healthy) | |--------|--------------------------| | 1 — errors | `indexer_failed_blocks=0`, `ERROR`-level count `0` | | 2 — lag | `chain_tip=1818`, `last_indexed=1813`, `lag=5` (single-digit) | | 3 — API 429/5xx | `/health` `200`; 20-request burst all `200` | | 4 — large swaps | `swap_events` count `60`, max return `776306256`, above-threshold `0` at `THRESH=2e9` | | 5 — large LP withdrawals | add `26`, remove `1`, above-threshold remove `1` at `THRESH_LP=1e9` | | 6 — blacklist | `{"blocked":false,…}` for test address; no `BLACKLIST HIT` lines | | 7 — pause | `{"paused":false}` via `terrad query … is_paused` | | 8 — reorg halt | `0` `INDEXER_REORG_HALT` / `indexer_reorg_halt` lines in indexer log | Quick sweep also ran clean (lag 5, failed_blocks 0, ERROR-level 0, reorg-halt 0). No repo changes required from this verification pass. Closing #434.

PlasticDigits (Migrated from gitlab.com) closed this issue

2026-06-30 07:05:08 +00:00