code/ust1-window
Watch
4
0
Fork
You've already forked ust1-window
 0
Code Issues 2 Pull requests 2 Projects Releases Packages Wiki Activity

feat(oracle-service): heartbeat same-rate refresh + confirm UpdateRate from DeliverTx wasm events #32

New issue
Closed
opened 2026-08-16 12:58:08 +00:00 by PlasticDigits · 9 comments
PlasticDigits commented 2026-08-16 12:58:08 +00:00 (Migrated from gitlab.com)

Summary

Bundle two oracle-service production fixes observed on columbus-5 after /ust1 reported Oracle stale while the bot was still polling Venus:

  1. Heartbeat — submit UpdateRate with the current Venus-normalized rate when the number is unchanged, as soon as INV-ORACLE-THROTTLE-001 allows (4h), so last_update_sec never exceeds window max_oracle_age_sec (6h).
  2. Confirm from DeliverTx wasm events — record liveness when the included tx emits action=update_rate + matching rate, instead of fail-closing on a lagged LCD State query.

Incident: #31. Ops hub: DEX #503. Related hygiene (do not expand into this issue): #29. Confirm-before-liveness origin: #23 (INV-ORACLE-LIVENESS-001). Poll/silence budget: #24.

Current codebase

Why /ust1 goes stale while the bot looks “healthy”

Window effective_swap / ensure_oracle_usable fail closed when now - oracle.last_update_sec > max_oracle_age_sec (21600 on mainnet). DEX UI ust1WindowGates.ts isOracleStale uses the same fields. Staleness is timestamp age, not “Venus disagrees”.

oracle-service polls BSC (~1h, INV-ORACLE-OPS-POLL-001) and only broadcasts when decide_tick_action returns Submit. Equal Venus vs on-chain rate short-circuits before policy:

TickAction::SkipEqualRate  →  info!("no rate change from BSC")  →  no tx, no liveness record

On-chain check_rate_update (INV-ORACLE-THROTTLE-001 / DAILY-001 / MONO-001) already allows new_rate == old_rate (monotonic >=). The skip is off-chain only. After 6h of a bit-identical Venus exchangeRateStored, the window rejects mint/redeem even though the rate is still the live Venus reading.

Live evidence (Aug 11–16 2026): gaps of 9–30h between update_rate txs while logs showed repeating no rate change from BSC and LIVENESS_ORACLE_NO_BROADCAST (silence_secs 21602 → 115000+). Most recent user-facing stale window: 2026-08-16 ~08:53–11:53 UTC (9h after 46276C8F…). Probe after C903047F… (11:53 UTC): age ~57 min, rate 1225669001749054409, not paused.

Why liveness pages after a successful include

INV-ORACLE-LIVENESS-001 (#23): LivenessTracker::record_successful_broadcast only after DeliverTx code == 0 and confirm::oracle_state_matches_intended_update (LCD QueryMsg::State: rate == proposed and last_update_sec advanced).

submit_and_confirm_oracle_update (main.rs):

  1. sign_and_broadcast_execute (CheckTx / SYNC — not success).
  2. wait_for_deliver_tx_success — polls /cosmos/tx/v1beta1/txs/{hash} until code == 0. TxResponseSummary keeps only txhash, code, raw_log — wasm events are discarded.
  3. Immediate query_wasm_smart(State). On mismatch: oracle state rate mismatch after inclusion … (fail-closed) → tick Err → no liveness record.

Mainnet counterexamples (DeliverTx code 0, wasm action=update_rate with the expected rate; LCD State still showed the previous rate):

Tx UTC Event rate LCD State at confirm
174D7855…19213 2026-08-13 21:51 1225553374561294951 1225547652182639438
2067CBE1…3FD6C 2026-08-14 03:52 1225566211664260479 1225553374561294951
16B487DB…DA5C1 2026-08-15 00:52 1225603380605508930 1225566211664260479

PublicNode LCD lag, not a failed execute. Fail-closed then keeps LIVENESS_ORACLE_NO_BROADCAST climbing even when on-chain last_update_sec already advanced. A later tick may hit UpdateTooSoon { min_interval: 14400 } against the rate that did land.

Timing constants (must stay aligned)

Knob Value Where
MIN_ORACLE_UPDATE_INTERVAL_SECS 14400 (4h) ust1-common
DEFAULT_MAX_ORACLE_AGE_SECS 21600 (6h) ust1-common / window config
DEFAULT_POLL_INTERVAL_SECS 3600 (1h) oracle-service config.rs
DEFAULT_ORACLE_MAX_SILENCE_SECS 21600 ORACLE_MAX_SILENCE_SECS
ORACLE_TX_CONFIRM_TIMEOUT_SECS 90 env

There is a 2h gap after throttle where a same-rate refresh can land before the window expires. Poll at 1h gives ~two chances. Heartbeat must use that gap; it must not widen max_oracle_age_sec or disable age checks (O2 / DEX runbook).

Contract behavior (no migrate required if heartbeat = same-rate UpdateRate)

ust1-oracle execute_update_rate calls shared check_rate_update then writes st.rate = new_rate and st.last_update_sec = now, emitting action=update_rate + rate. Same-rate after 4h advances the timestamp. Operator-only; paused oracle skips. Governance pause / monotonic / daily +2% caps unchanged.

Why this is needed

  1. User funds / product: /ust1 mint-arb is the $1 source of UST1. Stale oracle fail-closed pushes flow onto thin AMM pools (incident #31). Heartbeat keeps the window open while Venus is flat.
  2. Alert fidelity: LIVENESS_ORACLE_NO_BROADCAST is supposed to mean “no confirmed on-chain update” (INV-ORACLE-OPS-SILENCE-001). Today it also means “Venus unchanged” and “LCD State lagged after a real include”. That pages operators into #31-style incidents and hides the real dead-bot case.
  3. INV-ORACLE-LIVENESS-001 intent: success = this operator’s UpdateRate included and applied. Wasm events on the confirmed tx are the inclusion proof. A follow-up State read is a consistency check, not the source of truth, and must not undo a proven include.

Constraints / guardrails

  1. Do not disable or widen on-chain oracle age checks to “fix” staleness (DEX O2). Restore operator behavior; heartbeat is the restore.
  2. Do not weaken INV-ORACLE-THROTTLE-001 — no UpdateRate (including heartbeat) inside 4h of last_update_sec > 0.
  3. Do not weaken INV-ORACLE-MONO-001 / DAILY-001 — heartbeat uses the current Venus-normalized proposed (multi-RPC consensus). Never post a cached/old rate, never decrease, never breach daily cap.
  4. Do not weaken INV-ORACLE-LIVENESS-001 — CheckTx / BROADCAST_MODE_SYNC alone is still not success. DeliverTx code != 0, hash mismatch, timeout, missing/wrong wasm attrs → fail-closed, no liveness record.
  5. Do not auto-unpause — INV-ORACLE-PAUSE-001; skip all submits (including heartbeat) when state.paused.
  6. Do not skip EVM consensus — still ≥2 BSC RPCs, 0.01% agreement (evm_rpc.rs). Heartbeat must not short-circuit to a single provider.
  7. Event confirm must bind to this tx and this contract: txhash match, code == 0, wasm _contract_address == ORACLE_CONTRACT, action == update_rate, rate equal to this tick’s proposed (string/Uint128 canonical). Ignore other contracts’ events in the same tx.
  8. LCD State lag must not fail-close when events already prove the write. Optional State retry is observability only after event success. If events are absent (stripped LCD), fall back to retried State, then fail-closed.
  9. No new “touch” execute / code-id migrate unless same-rate UpdateRate is proven insufficient. Prefer off-chain tick change first.
  10. No secrets in logs. Keep LCD URL redaction; do not log mnemonic.
  11. Do not spend gas every poll — heartbeat only when policy would Ok (i.e. after 4h). SkipEqualRate inside the throttle window stays a cheap no-op.
  12. Agents must not broadcast governance txs. Operator updater key only for UpdateRate.
  13. Scope: oracle-service + skills/DEPLOYMENT. Contract change only if heartbeat cannot land same-rate. DEX runbook/skill pointers are a small follow-up, not a DEX contract change.

Relevant files

ust1-window (this repo)

Path Role
oracle-service/src/main.rs decide_tick_action, SkipEqualRate, run_once, submit_and_confirm_oracle_update
oracle-service/src/confirm.rs oracle_state_matches_intended_update (INV-ORACLE-LIVENESS-001)
oracle-service/src/liveness.rs in-process silence; record_successful_broadcast
oracle-service/src/terra_tx.rs TxResponseSummary, wait_for_deliver_tx_success, query_tx_by_hash
oracle-service/src/config.rs poll / silence / confirm timeouts
oracle-service/src/evm_rpc.rs / bsc.rs Venus consensus + normalize
smartcontracts-terraclassic/packages/ust1-common/src/oracle_policy.rs shared check_rate_update (already allows equal rate)
contracts/ust1-oracle/src/contract.rs execute_update_rate + wasm attrs
skills/oracle-liveness-confirm/SKILL.md confirm-before-liveness
skills/oracle-ops-poll-silence/SKILL.md poll ≪ max age; silence ≤ max age
docs/DEPLOYMENT.md operator env / expected logs

DEX (docs / UI gates — follow-up pointers, not the implementation)

Path Role
frontend-dapp/src/utils/ust1WindowGates.ts isOracleStale
scripts/check-ust1-wrap-ops-health.sh LCD age vs max_oracle_age_sec
docs/runbooks/ust1-wrap-production-ops.md O2, LIVENESS_ORACLE_NO_BROADCAST
docs/user-incident-faq.md stale-oracle user copy

Recommended direction

A. Heartbeat (same-rate UpdateRate)

Remove the unconditional proposed == state.rate → SkipEqualRate short-circuit as a hard skip. Route equal rates through check_rate_update:

  • UpdateTooSoon → keep skip (no gas); log at info (optionally outcome="throttled" not "failed" — do not expand into #29 unless one-line).
  • Ok with proposed == state.rate → TickAction::Heartbeat (or Submit { heartbeat: true }) → same UpdateRate { new_rate: proposed } path.
  • Ok with proposed != state.rate → existing Submit.
  • Paused / RateDecreased / DailyCapExceeded / ZeroRate → unchanged skips.

Log heartbeat distinctly (event="oracle_heartbeat" / tick_kind=heartbeat) so ops can tell refresh vs rate move.

Keep using Venus proposed from this tick’s consensus, not a stored last-success rate.

Optional env ORACLE_HEARTBEAT_ENABLE default on; if introduced, document in DEPLOYMENT.md and default must be safe for prod (enabled). Prefer no knob if tests cover the policy routing.

No window max_oracle_age_sec change. After deploy, max silence between confirmed updates should be ≤ 4h + one poll + confirm timeout while the process is healthy (~5h), inside the 6h window budget.

B. Confirm from DeliverTx wasm events

  1. Parse wasm attributes from the same LCD tx body used for DeliverTx code == 0 (extend TxResponseSummary or return events from wait_for_deliver_tx_success).
  2. New helper (e.g. confirm.rs): oracle_tx_events_match_update(contract, proposed, events) -> Result<()>.
  3. Success path: DeliverTx code 0 and event match → record liveness. Then optionally poll State (bounded retries) for rate + last_update_sec; on lag warn! only (oracle_state_lag_after_event_ok), do not Err the tick.
  4. If wasm attrs missing/unparseable: retry State with the existing matcher (and a short bounded retry for LCD lag), then fail-closed.
  5. Keep existing wiremock tests: DeliverTx reject, timeout, hash mismatch still must not record liveness.

Do not treat raw_log substring matching as sufficient if structured events exist.

Acceptance criteria

  • While Venus oracle_r is unchanged, the service still submits UpdateRate with that rate once now - last_update_sec >= 14400, and a successful include advances on-chain last_update_sec.
  • No UpdateRate (heartbeat or otherwise) when state.paused, when UpdateTooSoon, when proposed < state.rate, or when daily cap would fail.
  • Heartbeat uses this tick’s multi-RPC Venus proposed, never a stale cache.
  • LIVENESS_ORACLE_NO_BROADCAST does not fire solely because Venus was unchanged for 6h while heartbeats confirmed.
  • After DeliverTx code == 0 with matching wasm update_rate / rate, liveness is recorded even if the next State query still shows the previous rate.
  • DeliverTx code != 0, confirm timeout, txhash mismatch, wrong _contract_address, missing action, or event rate != proposed → no liveness record.
  • CheckTx-only / SYNC accept still does not record liveness.
  • Skills + docs/DEPLOYMENT.md describe heartbeat + event-confirm; silence alert text still means “no confirmed include”.
  • No contract migrate required for the default design (same-rate UpdateRate). If a migrate is proposed, call it out explicitly and do not mix it into the default path.
  • DEX ops runbook (O2) still says: do not disable age checks; heartbeat is the operator fix. (May land as a linked DEX docs MR.)

Test plan — functional paths

Unit / wiremock (oracle-service tests in main.rs, confirm.rs, terra_tx.rs, liveness.rs):

  1. Equal rate + throttle — proposed == state.rate, now - last < 14400 → no broadcast, no liveness.
  2. Equal rate + heartbeat eligible — proposed == state.rate, age ≥ 14400, not paused → broadcast UpdateRate with that rate; after DeliverTx + events, liveness recorded; last_update_sec would advance (contract multitest and/or mocked State after include).
  3. Rate change + eligible — existing Submit path still works; not labeled heartbeat.
  4. Paused — equal or changed rate → no broadcast.
  5. Monotonic decrease — skip, no broadcast.
  6. Daily cap exceeded — skip, no broadcast.
  7. First update last_update_sec == 0 — bootstrap still allowed (existing policy tests).
  8. DeliverTx code 0 + matching wasm events + lagged State — Ok, liveness recorded (this is the production bug).
  9. DeliverTx code 0 + matching events + matching State — Ok (happy path).
  10. DeliverTx code 0 + events missing + State match after retry — Ok (fallback).
  11. DeliverTx code 0 + events missing + State mismatch / no advance — Err, no liveness.
  12. DeliverTx code ≠ 0 — Err, no liveness (existing).
  13. Confirm timeout / hash mismatch — Err, no liveness (existing).
  14. Wrong contract in wasm _contract_address — Err even if rate string matches.
  15. Event rate ≠ proposed — Err (other operator / wrong msg).
  16. EVM consensus still required on heartbeat ticks (two URLs; disagreement does not submit).
  17. Policy unit tests in ust1-common already allow equal rate after throttle — add an explicit “same-rate after 4h is Ok” test if missing.
  18. ust1-oracle multitest: same-rate UpdateRate after 4h advances last_update_sec and emits update_rate.

Integration / ops (staging or LocalTerra if available; else document as manual):

  1. Run service with frozen Venus mock for >4h (or injected clock) and assert a heartbeat tx.
  2. scripts/verify_oracle_operator_env.sh still passes; new env knobs documented if any.

Test plan — attack, hack, and abuse vectors

Vector Expectation
A1. Heartbeat as “freshness wash” with a poisoned Venus rate Heartbeat cannot bypass 2-of-N BSC consensus. Single RPC / >0.01% disagreement → no submit. Canonical VENUS_VTOKEN_ADDRESS still enforced on chain id 56.
A2. Heartbeat to hide a dead Venus feed If RPCs return a frozen rate, heartbeat will refresh the same on-chain rate (honest). Age check no longer halts the window — this is intended. Detection remains: Venus normalize logs, external FDUSD monitors, daily cap if a later jump is huge. Do not add a hidden “force rate” env.
A3. Bypass 4h throttle via heartbeat Contract + off-chain check_rate_update must still return UpdateTooSoon. Fuzz/unit: equal rate at last+14399 rejected.
A4. Rate decrease smuggled as heartbeat proposed < state.rate still RateDecreased; no tx.
A5. Daily cap bypass by many heartbeats Same rate cannot exceed cap. Changed rate still capped +2%/UTC day vs baseline.
A6. Paused oracle still “kept fresh” SkipPaused; no fee burn; no liveness success from heartbeat.
A7. CheckTx success / mempool replacement counted as live Unchanged: only DeliverTx code 0 + event (or State fallback).
A8. LCD lies about State to paper over a failed execute Event path requires code == 0 on the broadcast hash. Failed DeliverTx cannot be rescued by a stale-matching State.
A9. LCD lies about wasm events on an unrelated tx Hash must eq_ignore_ascii_case the hash we broadcast. Wrong hash → fail-closed.
A10. Extra wasm execute in the same tx (confused deputy) Match _contract_address == ORACLE_CONTRACT and action=update_rate and rate == proposed. Other contracts’ events ignored.
A11. Event rate spoof / decimal encoding mismatch Compare using the same canonical encoding the contract emits (string of Uint128 as in mainnet rate=12255…). Reject empty/ambiguous attrs.
A12. Reorg / replaced tx Existing hash + code checks; do not record liveness on timeout. Optional: if a later query shows code ≠ 0, do not keep a success (today we return on first code 0).
A13. Two operators racing If our tx included with our proposed rate, events match → our liveness may record. If our tx fails because chain already moved, code ≠ 0 or event rate ≠ proposed → no record. Do not adopt the other operator’s rate as “our” success without our tx.
A14. Gas grief / spam heartbeat every poll Throttle 4h on-chain; off-chain skip inside interval. No submit every 1h poll when equal.
A15. Disable age checks via this change Window max_oracle_age_sec untouched. DEX health script / UI gates unchanged.
A16. Operator key compromise Unchanged: attacker can still UpdateRate within policy. Heartbeat does not add a new execute. Pause remains governance circuit breaker.
A17. Log injection / secret leak No mnemonic; LCD URLs redacted; event rate is public on-chain.
A18. /healthz process-up vs chain liveness Healthz stays process-up only; do not mark healthy solely because heartbeat skipped. Silence alert remains the pager.

Verification criteria

Code / CI

  • cargo test -p ust1-oracle-service (or workspace equivalent) covers heartbeat routing + event-confirm vs lagged State.
  • cargo test -p ust1-common / ust1-oracle include same-rate-after-throttle.
  • Skills oracle-liveness-confirm and oracle-ops-poll-silence updated; DEPLOYMENT.md log examples include oracle_heartbeat and event-confirm success.

Staging / mainnet (after Coolify deploy)

  • During a ≥4h Venus-flat period: Coolify logs show a heartbeat UpdateRate, Finder wasm action=update_rate, window effective_swap.oracle.last_update_sec advances, /ust1 stays Fresh (age ≤ 21600).
  • Inject or observe LCD State lag: tick is not tick failed / oracle_state_confirm outcome=failed when events match; liveness resets (LIVENESS_ORACLE_NO_BROADCAST silence_secs drops after success).
  • UST1_OPS_STRICT_STALE=1 ./scripts/check-ust1-wrap-ops-health.sh (DEX repo) PASSes while the service is healthy.
  • No heartbeat txs inside 4h of the previous include (throttle).
  • Pager does not fire at 6h solely due to unchanged Venus if heartbeats are confirming.

Close #31 only after mainnet age stays inside 6h across a multi-day Venus-flat stretch or an explicit waiver with evidence that remaining gaps are LCD/key/RPC outages (true dead-bot), not SkipEqualRate.

## Summary Bundle two **oracle-service** production fixes observed on columbus-5 after `/ust1` reported **Oracle stale** while the bot was still polling Venus: 1. **Heartbeat** — submit `UpdateRate` with the current Venus-normalized rate when the number is unchanged, as soon as **INV-ORACLE-THROTTLE-001** allows (4h), so `last_update_sec` never exceeds window `max_oracle_age_sec` (6h). 2. **Confirm from DeliverTx wasm events** — record liveness when the included tx emits `action=update_rate` + matching `rate`, instead of fail-closing on a lagged LCD `State` query. Incident: [#31](https://gitlab.com/PlasticDigits/ust1-window/-/issues/31). Ops hub: [DEX #503](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/503). Related hygiene (do not expand into this issue): [#29](https://gitlab.com/PlasticDigits/ust1-window/-/issues/29). Confirm-before-liveness origin: [#23](https://gitlab.com/PlasticDigits/ust1-window/-/issues/23) (**INV-ORACLE-LIVENESS-001**). Poll/silence budget: [#24](https://gitlab.com/PlasticDigits/ust1-window/-/issues/24). --- ## Current codebase ### Why `/ust1` goes stale while the bot looks “healthy” Window `effective_swap` / `ensure_oracle_usable` fail closed when `now - oracle.last_update_sec > max_oracle_age_sec` (**21600** on mainnet). DEX UI [`ust1WindowGates.ts`](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/blob/main/frontend-dapp/src/utils/ust1WindowGates.ts) `isOracleStale` uses the same fields. Staleness is **timestamp age**, not “Venus disagrees”. `oracle-service` polls BSC (~1h, **INV-ORACLE-OPS-POLL-001**) and only broadcasts when `decide_tick_action` returns `Submit`. Equal Venus vs on-chain rate short-circuits **before** policy: ```text TickAction::SkipEqualRate → info!("no rate change from BSC") → no tx, no liveness record ``` On-chain `check_rate_update` (**INV-ORACLE-THROTTLE-001 / DAILY-001 / MONO-001**) already **allows `new_rate == old_rate`** (monotonic `>=`). The skip is off-chain only. After 6h of a bit-identical Venus `exchangeRateStored`, the window rejects mint/redeem even though the rate is still the live Venus reading. Live evidence (Aug 11–16 2026): gaps of **9–30h** between `update_rate` txs while logs showed repeating `no rate change from BSC` and `LIVENESS_ORACLE_NO_BROADCAST` (`silence_secs` 21602 → 115000+). Most recent user-facing stale window: **2026-08-16 ~08:53–11:53 UTC** (9h after `46276C8F…`). Probe after `C903047F…` (11:53 UTC): age ~57 min, rate `1225669001749054409`, not paused. ### Why liveness pages after a successful include **INV-ORACLE-LIVENESS-001** ([#23](https://gitlab.com/PlasticDigits/ust1-window/-/issues/23)): `LivenessTracker::record_successful_broadcast` only after DeliverTx `code == 0` **and** `confirm::oracle_state_matches_intended_update` (LCD `QueryMsg::State`: rate == proposed **and** `last_update_sec` advanced). `submit_and_confirm_oracle_update` (`main.rs`): 1. `sign_and_broadcast_execute` (CheckTx / SYNC — not success). 2. `wait_for_deliver_tx_success` — polls `/cosmos/tx/v1beta1/txs/{hash}` until `code == 0`. `TxResponseSummary` keeps **only** `txhash`, `code`, `raw_log` — **wasm events are discarded**. 3. Immediate `query_wasm_smart(State)`. On mismatch: `oracle state rate mismatch after inclusion … (fail-closed)` → tick `Err` → **no liveness record**. Mainnet counterexamples (DeliverTx **code 0**, wasm `action=update_rate` with the **expected** rate; LCD `State` still showed the **previous** rate): | Tx | UTC | Event rate | LCD State at confirm | |----|-----|------------|----------------------| | `174D7855…19213` | 2026-08-13 21:51 | `1225553374561294951` | `1225547652182639438` | | `2067CBE1…3FD6C` | 2026-08-14 03:52 | `1225566211664260479` | `1225553374561294951` | | `16B487DB…DA5C1` | 2026-08-15 00:52 | `1225603380605508930` | `1225566211664260479` | PublicNode LCD lag, not a failed execute. Fail-closed then keeps `LIVENESS_ORACLE_NO_BROADCAST` climbing even when on-chain `last_update_sec` already advanced. A later tick may hit `UpdateTooSoon { min_interval: 14400 }` against the rate that **did** land. ### Timing constants (must stay aligned) | Knob | Value | Where | |------|-------|--------| | `MIN_ORACLE_UPDATE_INTERVAL_SECS` | **14400** (4h) | `ust1-common` | | `DEFAULT_MAX_ORACLE_AGE_SECS` | **21600** (6h) | `ust1-common` / window config | | `DEFAULT_POLL_INTERVAL_SECS` | **3600** (1h) | `oracle-service` `config.rs` | | `DEFAULT_ORACLE_MAX_SILENCE_SECS` | **21600** | `ORACLE_MAX_SILENCE_SECS` | | `ORACLE_TX_CONFIRM_TIMEOUT_SECS` | 90 | env | There is a **2h** gap after throttle where a same-rate refresh can land before the window expires. Poll at 1h gives ~two chances. Heartbeat must use that gap; it must **not** widen `max_oracle_age_sec` or disable age checks (**O2** / DEX runbook). ### Contract behavior (no migrate required if heartbeat = same-rate `UpdateRate`) `ust1-oracle` `execute_update_rate` calls shared `check_rate_update` then writes `st.rate = new_rate` and `st.last_update_sec = now`, emitting `action=update_rate` + `rate`. Same-rate after 4h advances the timestamp. Operator-only; paused oracle skips. Governance pause / monotonic / daily +2% caps unchanged. --- ## Why this is needed 1. **User funds / product:** `/ust1` mint-arb is the $1 source of UST1. Stale oracle fail-closed pushes flow onto thin AMM pools (incident #31). Heartbeat keeps the window open while Venus is flat. 2. **Alert fidelity:** `LIVENESS_ORACLE_NO_BROADCAST` is supposed to mean “no confirmed on-chain update” (INV-ORACLE-OPS-SILENCE-001). Today it also means “Venus unchanged” and “LCD State lagged after a real include”. That pages operators into #31-style incidents and hides the real dead-bot case. 3. **INV-ORACLE-LIVENESS-001 intent:** success = this operator’s `UpdateRate` **included and applied**. Wasm events on the confirmed tx are the inclusion proof. A follow-up `State` read is a **consistency check**, not the source of truth, and must not undo a proven include. --- ## Constraints / guardrails 1. **Do not disable or widen on-chain oracle age checks** to “fix” staleness (DEX **O2**). Restore operator behavior; heartbeat is the restore. 2. **Do not weaken INV-ORACLE-THROTTLE-001** — no `UpdateRate` (including heartbeat) inside 4h of `last_update_sec > 0`. 3. **Do not weaken INV-ORACLE-MONO-001 / DAILY-001** — heartbeat uses the **current** Venus-normalized `proposed` (multi-RPC consensus). Never post a cached/old rate, never decrease, never breach daily cap. 4. **Do not weaken INV-ORACLE-LIVENESS-001** — CheckTx / `BROADCAST_MODE_SYNC` alone is still not success. DeliverTx `code != 0`, hash mismatch, timeout, missing/wrong wasm attrs → fail-closed, no liveness record. 5. **Do not auto-unpause** — **INV-ORACLE-PAUSE-001**; skip all submits (including heartbeat) when `state.paused`. 6. **Do not skip EVM consensus** — still ≥2 BSC RPCs, 0.01% agreement (`evm_rpc.rs`). Heartbeat must not short-circuit to a single provider. 7. **Event confirm must bind to this tx and this contract:** `txhash` match, `code == 0`, wasm `_contract_address == ORACLE_CONTRACT`, `action == update_rate`, `rate` equal to this tick’s `proposed` (string/Uint128 canonical). Ignore other contracts’ events in the same tx. 8. **LCD State lag must not fail-close when events already prove the write.** Optional State retry is observability only after event success. If events are absent (stripped LCD), fall back to retried State, then fail-closed. 9. **No new “touch” execute / code-id migrate** unless same-rate `UpdateRate` is proven insufficient. Prefer off-chain tick change first. 10. **No secrets in logs.** Keep LCD URL redaction; do not log mnemonic. 11. **Do not spend gas every poll** — heartbeat only when policy would `Ok` (i.e. after 4h). `SkipEqualRate` inside the throttle window stays a cheap no-op. 12. **Agents must not broadcast governance txs.** Operator updater key only for `UpdateRate`. 13. **Scope:** `oracle-service` + skills/DEPLOYMENT. Contract change only if heartbeat cannot land same-rate. DEX runbook/skill pointers are a small follow-up, not a DEX contract change. --- ## Relevant files **ust1-window (this repo)** | Path | Role | |------|------| | `oracle-service/src/main.rs` | `decide_tick_action`, `SkipEqualRate`, `run_once`, `submit_and_confirm_oracle_update` | | `oracle-service/src/confirm.rs` | `oracle_state_matches_intended_update` (INV-ORACLE-LIVENESS-001) | | `oracle-service/src/liveness.rs` | in-process silence; `record_successful_broadcast` | | `oracle-service/src/terra_tx.rs` | `TxResponseSummary`, `wait_for_deliver_tx_success`, `query_tx_by_hash` | | `oracle-service/src/config.rs` | poll / silence / confirm timeouts | | `oracle-service/src/evm_rpc.rs` / `bsc.rs` | Venus consensus + normalize | | `smartcontracts-terraclassic/packages/ust1-common/src/oracle_policy.rs` | shared `check_rate_update` (already allows equal rate) | | `contracts/ust1-oracle/src/contract.rs` | `execute_update_rate` + wasm attrs | | `skills/oracle-liveness-confirm/SKILL.md` | confirm-before-liveness | | `skills/oracle-ops-poll-silence/SKILL.md` | poll ≪ max age; silence ≤ max age | | `docs/DEPLOYMENT.md` | operator env / expected logs | **DEX (docs / UI gates — follow-up pointers, not the implementation)** | Path | Role | |------|------| | `frontend-dapp/src/utils/ust1WindowGates.ts` | `isOracleStale` | | `scripts/check-ust1-wrap-ops-health.sh` | LCD age vs `max_oracle_age_sec` | | `docs/runbooks/ust1-wrap-production-ops.md` | **O2**, `LIVENESS_ORACLE_NO_BROADCAST` | | `docs/user-incident-faq.md` | stale-oracle user copy | --- ## Recommended direction ### A. Heartbeat (same-rate `UpdateRate`) Remove the unconditional `proposed == state.rate → SkipEqualRate` short-circuit **as a hard skip**. Route equal rates through `check_rate_update`: - `UpdateTooSoon` → keep skip (no gas); log at info (optionally `outcome="throttled"` not `"failed"` — do not expand into #29 unless one-line). - `Ok` with `proposed == state.rate` → `TickAction::Heartbeat` (or `Submit { heartbeat: true }`) → same `UpdateRate { new_rate: proposed }` path. - `Ok` with `proposed != state.rate` → existing `Submit`. - Paused / `RateDecreased` / `DailyCapExceeded` / `ZeroRate` → unchanged skips. Log heartbeat distinctly (`event="oracle_heartbeat"` / `tick_kind=heartbeat`) so ops can tell refresh vs rate move. Keep using Venus `proposed` from this tick’s consensus, not a stored last-success rate. Optional env `ORACLE_HEARTBEAT_ENABLE` default **on**; if introduced, document in `DEPLOYMENT.md` and default must be safe for prod (enabled). Prefer no knob if tests cover the policy routing. No window `max_oracle_age_sec` change. After deploy, max silence between confirmed updates should be **≤ 4h + one poll + confirm timeout** while the process is healthy (~5h), inside the 6h window budget. ### B. Confirm from DeliverTx wasm events 1. Parse wasm attributes from the **same** LCD tx body used for DeliverTx `code == 0` (extend `TxResponseSummary` or return events from `wait_for_deliver_tx_success`). 2. New helper (e.g. `confirm.rs`): `oracle_tx_events_match_update(contract, proposed, events) -> Result<()>`. 3. Success path: DeliverTx code 0 **and** event match → record liveness. Then **optionally** poll `State` (bounded retries) for `rate` + `last_update_sec`; on lag `warn!` only (`oracle_state_lag_after_event_ok`), do **not** `Err` the tick. 4. If wasm attrs missing/unparseable: retry State with the existing matcher (and a short bounded retry for LCD lag), then fail-closed. 5. Keep existing wiremock tests: DeliverTx reject, timeout, hash mismatch still must **not** record liveness. Do not treat raw_log substring matching as sufficient if structured `events` exist. --- ## Acceptance criteria - [ ] While Venus `oracle_r` is unchanged, the service still submits `UpdateRate` with that rate once `now - last_update_sec >= 14400`, and a successful include advances on-chain `last_update_sec`. - [ ] No `UpdateRate` (heartbeat or otherwise) when `state.paused`, when `UpdateTooSoon`, when `proposed < state.rate`, or when daily cap would fail. - [ ] Heartbeat uses this tick’s multi-RPC Venus `proposed`, never a stale cache. - [ ] `LIVENESS_ORACLE_NO_BROADCAST` does **not** fire solely because Venus was unchanged for 6h while heartbeats confirmed. - [ ] After DeliverTx `code == 0` with matching wasm `update_rate` / `rate`, liveness is recorded even if the next `State` query still shows the previous rate. - [ ] DeliverTx `code != 0`, confirm timeout, txhash mismatch, wrong `_contract_address`, missing `action`, or event `rate != proposed` → no liveness record. - [ ] CheckTx-only / SYNC accept still does not record liveness. - [ ] Skills + `docs/DEPLOYMENT.md` describe heartbeat + event-confirm; silence alert text still means “no confirmed include”. - [ ] No contract migrate required for the default design (same-rate `UpdateRate`). If a migrate is proposed, call it out explicitly and do not mix it into the default path. - [ ] DEX ops runbook (**O2**) still says: do not disable age checks; heartbeat is the operator fix. (May land as a linked DEX docs MR.) --- ## Test plan — functional paths Unit / wiremock (`oracle-service` tests in `main.rs`, `confirm.rs`, `terra_tx.rs`, `liveness.rs`): 1. **Equal rate + throttle** — `proposed == state.rate`, `now - last < 14400` → no broadcast, no liveness. 2. **Equal rate + heartbeat eligible** — `proposed == state.rate`, age ≥ 14400, not paused → broadcast `UpdateRate` with that rate; after DeliverTx + events, liveness recorded; `last_update_sec` would advance (contract multitest and/or mocked State after include). 3. **Rate change + eligible** — existing Submit path still works; not labeled heartbeat. 4. **Paused** — equal or changed rate → no broadcast. 5. **Monotonic decrease** — skip, no broadcast. 6. **Daily cap exceeded** — skip, no broadcast. 7. **First update `last_update_sec == 0`** — bootstrap still allowed (existing policy tests). 8. **DeliverTx code 0 + matching wasm events + lagged State** — **Ok**, liveness recorded (this is the production bug). 9. **DeliverTx code 0 + matching events + matching State** — Ok (happy path). 10. **DeliverTx code 0 + events missing + State match after retry** — Ok (fallback). 11. **DeliverTx code 0 + events missing + State mismatch / no advance** — Err, no liveness. 12. **DeliverTx code ≠ 0** — Err, no liveness (existing). 13. **Confirm timeout / hash mismatch** — Err, no liveness (existing). 14. **Wrong contract in wasm `_contract_address`** — Err even if rate string matches. 15. **Event rate ≠ proposed** — Err (other operator / wrong msg). 16. **EVM consensus still required** on heartbeat ticks (two URLs; disagreement does not submit). 17. **Policy unit tests** in `ust1-common` already allow equal rate after throttle — add an explicit “same-rate after 4h is Ok” test if missing. 18. **ust1-oracle multitest:** same-rate `UpdateRate` after 4h advances `last_update_sec` and emits `update_rate`. Integration / ops (staging or LocalTerra if available; else document as manual): 19. Run service with frozen Venus mock for >4h (or injected clock) and assert a heartbeat tx. 20. `scripts/verify_oracle_operator_env.sh` still passes; new env knobs documented if any. --- ## Test plan — attack, hack, and abuse vectors | Vector | Expectation | |--------|-------------| | **A1. Heartbeat as “freshness wash” with a poisoned Venus rate** | Heartbeat cannot bypass 2-of-N BSC consensus. Single RPC / >0.01% disagreement → no submit. Canonical `VENUS_VTOKEN_ADDRESS` still enforced on chain id 56. | | **A2. Heartbeat to hide a dead Venus feed** | If RPCs return a frozen rate, heartbeat will refresh the **same** on-chain rate (honest). Age check no longer halts the window — this is intended. Detection remains: Venus normalize logs, external FDUSD monitors, daily cap if a later jump is huge. Do not add a hidden “force rate” env. | | **A3. Bypass 4h throttle via heartbeat** | Contract + off-chain `check_rate_update` must still return `UpdateTooSoon`. Fuzz/unit: equal rate at `last+14399` rejected. | | **A4. Rate decrease smuggled as heartbeat** | `proposed < state.rate` still `RateDecreased`; no tx. | | **A5. Daily cap bypass by many heartbeats** | Same rate cannot exceed cap. Changed rate still capped +2%/UTC day vs baseline. | | **A6. Paused oracle still “kept fresh”** | SkipPaused; no fee burn; no liveness success from heartbeat. | | **A7. CheckTx success / mempool replacement counted as live** | Unchanged: only DeliverTx code 0 + event (or State fallback). | | **A8. LCD lies about State to paper over a failed execute** | Event path requires `code == 0` on the **broadcast hash**. Failed DeliverTx cannot be rescued by a stale-matching State. | | **A9. LCD lies about wasm events on an unrelated tx** | Hash must `eq_ignore_ascii_case` the hash we broadcast. Wrong hash → fail-closed. | | **A10. Extra wasm execute in the same tx (confused deputy)** | Match `_contract_address == ORACLE_CONTRACT` and `action=update_rate` and rate == proposed. Other contracts’ events ignored. | | **A11. Event `rate` spoof / decimal encoding mismatch** | Compare using the same canonical encoding the contract emits (string of `Uint128` as in mainnet `rate=12255…`). Reject empty/ambiguous attrs. | | **A12. Reorg / replaced tx** | Existing hash + code checks; do not record liveness on timeout. Optional: if a later query shows code ≠ 0, do not keep a success (today we return on first code 0). | | **A13. Two operators racing** | If our tx included with our proposed rate, events match → our liveness may record. If our tx fails because chain already moved, code ≠ 0 or event rate ≠ proposed → no record. Do not adopt the other operator’s rate as “our” success without our tx. | | **A14. Gas grief / spam heartbeat every poll** | Throttle 4h on-chain; off-chain skip inside interval. No submit every 1h poll when equal. | | **A15. Disable age checks via this change** | Window `max_oracle_age_sec` untouched. DEX health script / UI gates unchanged. | | **A16. Operator key compromise** | Unchanged: attacker can still `UpdateRate` within policy. Heartbeat does not add a new execute. Pause remains governance circuit breaker. | | **A17. Log injection / secret leak** | No mnemonic; LCD URLs redacted; event rate is public on-chain. | | **A18. `/healthz` process-up vs chain liveness** | Healthz stays process-up only; do not mark healthy solely because heartbeat skipped. Silence alert remains the pager. | --- ## Verification criteria **Code / CI** - [ ] `cargo test -p ust1-oracle-service` (or workspace equivalent) covers heartbeat routing + event-confirm vs lagged State. - [ ] `cargo test -p ust1-common` / `ust1-oracle` include same-rate-after-throttle. - [ ] Skills `oracle-liveness-confirm` and `oracle-ops-poll-silence` updated; `DEPLOYMENT.md` log examples include `oracle_heartbeat` and event-confirm success. **Staging / mainnet (after Coolify deploy)** - [ ] During a ≥4h Venus-flat period: Coolify logs show a heartbeat `UpdateRate`, Finder wasm `action=update_rate`, window `effective_swap.oracle.last_update_sec` advances, `/ust1` stays **Fresh** (`age ≤ 21600`). - [ ] Inject or observe LCD State lag: tick is **not** `tick failed` / `oracle_state_confirm outcome=failed` when events match; liveness resets (`LIVENESS_ORACLE_NO_BROADCAST` silence_secs drops after success). - [ ] `UST1_OPS_STRICT_STALE=1 ./scripts/check-ust1-wrap-ops-health.sh` (DEX repo) PASSes while the service is healthy. - [ ] No heartbeat txs inside 4h of the previous include (throttle). - [ ] Pager does not fire at 6h solely due to unchanged Venus if heartbeats are confirming. **Close #31** only after mainnet age stays inside 6h across a multi-day Venus-flat stretch **or** an explicit waiver with evidence that remaining gaps are LCD/key/RPC outages (true dead-bot), not `SkipEqualRate`.
PlasticDigits commented 2026-08-16 12:58:09 +00:00 (Migrated from gitlab.com)

marked as related to #31

marked as related to #31
PlasticDigits commented 2026-08-16 12:58:10 +00:00 (Migrated from gitlab.com)

marked as related to #23

marked as related to #23
PlasticDigits commented 2026-08-16 12:58:11 +00:00 (Migrated from gitlab.com)

marked as related to #29

marked as related to #29
PlasticDigits commented 2026-08-16 13:26:54 +00:00 (Migrated from gitlab.com)

mentioned in commit a86eda7892

mentioned in commit a86eda78927efb81a60869d5daca3e05b9f6e41a
PlasticDigits commented 2026-08-16 13:26:57 +00:00 (Migrated from gitlab.com)

mentioned in merge request !46

mentioned in merge request !46
PlasticDigits commented 2026-08-16 13:31:31 +00:00 (Migrated from gitlab.com)

mentioned in commit de0e8f01ca

mentioned in commit de0e8f01ca3948e46457ddc43f778b5f0a0a1443
PlasticDigits (Migrated from gitlab.com) closed this issue 2026-08-16 13:31:31 +00:00
PlasticDigits commented 2026-08-16 13:32:55 +00:00 (Migrated from gitlab.com)

Merge follow-up for !46 (de0e8f0 on main)

!46 is merged (no conflicts; not auto-merge / not waiting on CI). Code review vs this issue’s acceptance criteria: code AC 1–9 PASS. Unit tests: ust1-oracle-service 106, ust1-common 25, ust1-oracle 10 — 141 passed.

Remaining gaps (not merge blockers; track after Coolify deploy):

  1. DEX O2 runbook not updated in this repo — AC10 is only partial here. docs/runbooks/ust1-wrap-production-ops.md still needs the “do not disable age checks; heartbeat is the operator fix” pointer. Follow-up: cl8y-dex-terraclassic#503.
  2. Post-deploy / mainnet soak still open — verification criteria (≥4h Venus-flat heartbeat, Finder action=update_rate, window last_update_sec advances, /ust1 Fresh, LCD State lag does not tick failed, no heartbeat inside 4h) are not satisfied by the merge alone. Leave #31 open until age stays inside 6h across a Venus-flat stretch (or an explicit waiver).
  3. scripts/verify_oracle_operator_env.sh unchanged — test plan item 20. Low risk: no new env knobs; behavior is policy-driven. Frozen-Venus integration (item 19) was not run.

No code defects found that would require reverting !46.

## Merge follow-up for !46 (`de0e8f0` on `main`) !46 is merged (no conflicts; not auto-merge / not waiting on CI). Code review vs this issue’s acceptance criteria: **code AC 1–9 PASS**. Unit tests: `ust1-oracle-service` 106, `ust1-common` 25, `ust1-oracle` 10 — **141 passed**. Remaining gaps (not merge blockers; track after Coolify deploy): 1. **DEX O2 runbook not updated in this repo** — AC10 is only partial here. `docs/runbooks/ust1-wrap-production-ops.md` still needs the “do not disable age checks; heartbeat is the operator fix” pointer. Follow-up: [cl8y-dex-terraclassic#503](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/503). 2. **Post-deploy / mainnet soak still open** — verification criteria (≥4h Venus-flat heartbeat, Finder `action=update_rate`, window `last_update_sec` advances, `/ust1` Fresh, LCD State lag does not `tick failed`, no heartbeat inside 4h) are **not** satisfied by the merge alone. **Leave #31 open** until age stays inside 6h across a Venus-flat stretch (or an explicit waiver). 3. **`scripts/verify_oracle_operator_env.sh` unchanged** — test plan item 20. Low risk: no new env knobs; behavior is policy-driven. Frozen-Venus integration (item 19) was not run. No code defects found that would require reverting !46.
PlasticDigits commented 2026-08-16 13:42:02 +00:00 (Migrated from gitlab.com)

Post-deploy live check (Coolify just started)

New binary is live: startup log includes heartbeat + event-confirm (#32 / #23 / #24). First tick after deploy is correct SkipEqualRate, not a missed heartbeat.

Probe (2026-08-16 ~13:40 UTC) Value
Venus consensus oracle_r (2 RPCs) 1225669001749054409
On-chain State.rate 1225669001749054409 (match → no rate change)
last_update_sec 1786881183 = 2026-08-16 11:53:03 UTC
Last include C903047F… wasm action=update_rate rate match, code=0
Oracle age ~6432 s (~1.8 h) vs throttle 14400 / max age 21600
Window / oracle paused both false
/ust1 Oracle Fresh
UST1_OPS_STRICT_STALE=1 ./scripts/check-ust1-wrap-ops-health.sh PASS (14 pass / 2 inventory WARNs unrelated to oracle)

Heartbeat eligibility: 11:53 + 4h = 15:53 UTC. With hourly poll from this tick, expect skip at ~14:40 and ~15:40, then event=oracle_heartbeat around ~16:40 UTC (still ~1h before 6h stale at 17:53). No heartbeat inside 4h is required.

Still open for soak: observe that heartbeat include, Finder wasm, last_update_sec advance, /ust1 stays Fresh, and LCD lag does not tick failed. Duplicate Venus normalized … INFO is #29, not this deploy.

## Post-deploy live check (Coolify just started) New binary is live: startup log includes heartbeat + event-confirm (#32 / #23 / #24). First tick after deploy is **correct `SkipEqualRate`**, not a missed heartbeat. | Probe (2026-08-16 ~13:40 UTC) | Value | |---|---| | Venus consensus `oracle_r` (2 RPCs) | `1225669001749054409` | | On-chain `State.rate` | `1225669001749054409` (match → no rate change) | | `last_update_sec` | `1786881183` = **2026-08-16 11:53:03 UTC** | | Last include | [`C903047F…`](https://finder.terra.money/columbus-5/tx/C903047F96FAA558462481D2A515B3F7911AD56DB30E669FF40128746E7748F9) wasm `action=update_rate` rate match, `code=0` | | Oracle age | **~6432 s (~1.8 h)** vs throttle **14400** / max age **21600** | | Window / oracle paused | both `false` | | [`/ust1`](https://dex.cl8y.com/ust1) | **Oracle Fresh** | | `UST1_OPS_STRICT_STALE=1 ./scripts/check-ust1-wrap-ops-health.sh` | **PASS** (14 pass / 2 inventory WARNs unrelated to oracle) | Heartbeat eligibility: `11:53 + 4h = 15:53 UTC`. With hourly poll from this tick, expect skip at ~14:40 and ~15:40, then **`event=oracle_heartbeat` around ~16:40 UTC** (still ~1h before 6h stale at 17:53). No heartbeat inside 4h is required. Still open for soak: observe that heartbeat include, Finder wasm, `last_update_sec` advance, `/ust1` stays Fresh, and LCD lag does not `tick failed`. Duplicate Venus `normalized …` INFO is [#29](https://gitlab.com/PlasticDigits/ust1-window/-/issues/29), not this deploy.
PlasticDigits commented 2026-08-16 13:42:04 +00:00 (Migrated from gitlab.com)

mentioned in issue #31

mentioned in issue #31
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue/50
renovate/configure
No results found.
Labels
Clear labels   
enhancement

   
governance

   
QA

   
testing
No labels
enhancement
governance
QA
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
code/ust1-window#32
No description provided.