Security (EVM-10): oracle-service has no liveness alerting; silent failure compounds CW-05

Brouie commented

2026-04-21 08:14:44 +00:00

(Migrated from gitlab.com)

@PlasticDigits finding from security review on #5.

Severity

Medium. Operational — combines with CW-05 to create invisible attack surface.

Description

oracle-service/src/main.rs main loop logs warn!(error = %e, "tick failed") on any error but exposes no external signal of liveness. There is no metrics endpoint, no heartbeat file, no external alert path, no exit-on-prolonged-failure.

The service will silently retry every poll_interval_secs (default 6h) forever if BSC RPC is down, Terra LCD is unreachable, the mnemonic is wrong, or Venus exchangeRateStored reverts.

Why this matters

Paired with CW-05 (no staleness guard on ust1-window), this is the real attack surface:

oracle-service silently fails at T=0 (any cause)
on-chain oracle rate freezes at whatever it was at T=0
ust1-window keeps executing swaps at that frozen rate
Market rate drifts over hours/days while on-chain rate stays stuck
Users either extract value (if real rate has risen) or lose value (if real rate has fallen)

No one notices until either a user complains or someone eyeballs the logs.

Recommendation

At minimum:

Emit a heartbeat file / metric / stdout timestamp after every successful sign_and_broadcast_execute AND after every successful check_rate_update (even no-op ticks count as liveness)
Track time-since-last-successful-broadcast; exit with non-zero or emit loud alert if > N hours (tunable, e.g. 8h default)
Optional: lightweight Prometheus metrics endpoint (one counter, one gauge for last_broadcast_epoch_seconds)

Pair with systemd / docker healthcheck / external uptime monitor to catch the exit/alert.

Fixing this largely reduces the severity of CW-05 — if the staleness guard is missing on-chain but the off-chain liveness is solid, stale rates can't persist long enough to matter.

Review parent: #5
Critical pair: CW-05 (on-chain staleness guard) — #6

@PlasticDigits finding from security review on #5. ## Severity Medium. Operational — combines with CW-05 to create invisible attack surface. ## Description `oracle-service/src/main.rs` `main` loop logs `warn!(error = %e, "tick failed")` on any error but exposes no external signal of liveness. There is no metrics endpoint, no heartbeat file, no external alert path, no exit-on-prolonged-failure. The service will silently retry every `poll_interval_secs` (default 6h) forever if BSC RPC is down, Terra LCD is unreachable, the mnemonic is wrong, or Venus `exchangeRateStored` reverts. ## Why this matters Paired with **CW-05** (no staleness guard on ust1-window), this is the real attack surface: 1. oracle-service silently fails at T=0 (any cause) 2. on-chain oracle rate freezes at whatever it was at T=0 3. ust1-window keeps executing swaps at that frozen rate 4. Market rate drifts over hours/days while on-chain rate stays stuck 5. Users either extract value (if real rate has risen) or lose value (if real rate has fallen) No one notices until either a user complains or someone eyeballs the logs. ## Recommendation At minimum: 1. Emit a heartbeat file / metric / stdout timestamp after every successful `sign_and_broadcast_execute` AND after every successful `check_rate_update` (even no-op ticks count as liveness) 2. Track time-since-last-successful-broadcast; exit with non-zero or emit loud alert if > N hours (tunable, e.g. 8h default) 3. Optional: lightweight Prometheus metrics endpoint (one counter, one gauge for `last_broadcast_epoch_seconds`) Pair with systemd / docker healthcheck / external uptime monitor to catch the exit/alert. Fixing this largely reduces the severity of CW-05 — if the staleness guard is missing on-chain but the off-chain liveness is solid, stale rates can't persist long enough to matter. ## Related - Review parent: #5 - Critical pair: CW-05 (on-chain staleness guard) — #6

Brouie commented

2026-04-21 08:17:08 +00:00

(Migrated from gitlab.com)

mentioned in issue #5

PlasticDigits commented

2026-04-21 13:46:34 +00:00

(Migrated from gitlab.com)

mentioned in commit 31a8b3a64f

mentioned in commit 31a8b3a64fe4bbb2fae8ae40c6ffac1db63c3b2f

PlasticDigits commented

2026-04-21 13:46:39 +00:00

(Migrated from gitlab.com)

Fix merged (main @ `7aa1094`)

Liveness / logging (no Prometheus)

Every check_rate_update result is logged with structured fields (event = "check_rate_update", outcome = "ok" | "failed").
Every sign_and_broadcast_execute completes with info! on success (tx hash + contract) or warn! on failure; LCD responses with non-zero tx_response.code are treated as errors so we do not record a successful broadcast.
In-process LivenessTracker: time since last successful on-chain broadcast; each poll tick emits error! with alert = "LIVENESS_ORACLE_NO_BROADCAST" when silence exceeds ORACLE_MAX_SILENCE_SECS (default 28800 = 8h). Threshold is tunable via env.

Docs

README section Oracle service: observability and deployment — notes structured logs only, log-based alert, and that hosts such as Render should provide HTTP health checks and external uptime/alerting; any deployment must offer comparable monitoring.

No metrics server added; service stays light for Render.

## Fix merged (main @ 7aa1094) **Liveness / logging (no Prometheus)** - Every `check_rate_update` result is logged with structured fields (`event = "check_rate_update"`, `outcome = "ok" | "failed"`). - Every `sign_and_broadcast_execute` completes with `info!` on success (tx hash + contract) or `warn!` on failure; LCD responses with non-zero `tx_response.code` are treated as errors so we do not record a successful broadcast. - In-process `LivenessTracker`: time since last **successful** on-chain broadcast; each poll tick emits `error!` with `alert = "LIVENESS_ORACLE_NO_BROADCAST"` when silence exceeds `ORACLE_MAX_SILENCE_SECS` (default **28800** = 8h). Threshold is tunable via env. **Docs** - README section **Oracle service: observability and deployment** — notes structured logs only, log-based alert, and that hosts such as **Render** should provide HTTP health checks and external uptime/alerting; any deployment must offer comparable monitoring. No metrics server added; service stays light for Render.

Brouie commented

2026-04-22 00:30:37 +00:00

(Migrated from gitlab.com)

@PlasticDigits verified fix on 31a8b3a.

New liveness.rs module:

LivenessTracker tracks last_successful_broadcast: Option<Instant>
record_successful_broadcast() called after each successful Terra tx
silence_since_last_broadcast() falls back to process start when no broadcast yet
should_alert(max_silence) → ORACLE_MAX_SILENCE_SECS default 8h, configurable

Integration:

Every check_rate_update outcome logged (ok vs policy rejection)
Every sign_and_broadcast_execute success/failure logged
Non-zero tx_response.code now treated as broadcast failure (catches silent Terra LCD failures)
Error-level LIVENESS alert when silence > threshold
README documents external monitoring pattern

Tracker implementation is clean — no Prometheus dep bloat, pure in-process Instant tracking suitable for sidecar/container deployments.

@PlasticDigits verified fix on 31a8b3a. New `liveness.rs` module: - `LivenessTracker` tracks `last_successful_broadcast: Option<Instant>` - `record_successful_broadcast()` called after each successful Terra tx - `silence_since_last_broadcast()` falls back to process start when no broadcast yet - `should_alert(max_silence)` → `ORACLE_MAX_SILENCE_SECS` default 8h, configurable Integration: - Every `check_rate_update` outcome logged (ok vs policy rejection) - Every `sign_and_broadcast_execute` success/failure logged - Non-zero `tx_response.code` now treated as broadcast failure (catches silent Terra LCD failures) - Error-level LIVENESS alert when silence > threshold - README documents external monitoring pattern Tracker implementation is clean — no Prometheus dep bloat, pure in-process `Instant` tracking suitable for sidecar/container deployments.

Brouie (Migrated from gitlab.com) closed this issue

2026-04-22 00:30:39 +00:00

Rows
Columns

Security (EVM-10): oracle-service has no liveness alerting; silent failure compounds CW-05 #11

Severity

Description

Why this matters

Recommendation

Related

Fix merged (main @ 7aa1094)

Fix merged (main @ `7aa1094`)