Oracle stale ~9h (max age 6h): window fails closed, UST1 minting currently impossible on mainnet #31

New issue

Closed

opened 2026-08-16 11:41:23 +00:00 by leonardocolucci · 5 comments

leonardocolucci commented 2026-08-16 11:41:23 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Summary

The ust1-oracle (terra1fmht0t6svq3n24zx03nkfja0m40zhfyyxkdcvlrkl6u7gfe6aagq4gch8n) last updated at 2026-08-16 02:52 UTC (last_update_sec = 1786848774, rate 1225650379515835109 ≈ 1.22565) and has not ticked since. With the window's max_oracle_age_sec = 21600, ensure_oracle_usable has been failing closed for ~3 hours and counting (oracle age ~9h at time of filing), so no one can currently mint UST1 on mainnet — the Deposit path reverts.

Everything else is healthy: window paused = false, bridge paused = false (fee 30 bps), vFDUSD bridging works.

Likely cause

The oracle-service updater isn't running / confirming ticks — README still lists "oracle-service Coolify (normalized ticks)" as pending under mainnet status. Per INV-ORACLE-OPS-SILENCE-001 the service should emit a high-visibility liveness alert on >6h silence; that alert path may also be worth verifying since the silence has now exceeded it.

Impact

With the click-wrap live and the UST1 pools seeded, the mint window is the intended $1.00 source of UST1. While it's closed, UST1 can only be acquired from the thin DEX pools — which is exactly how UST1 got pushed to ~$1.16 over the launch weekend, and it undercuts the mint-arb that's supposed to hold UST1 at $1.

Ready to verify

We have ~114 vFDUSD staged (BSC, Venus) and the Terra side automated (bridge claim + window deposit) — happy to be the live probe the moment the oracle is fresh and confirm the full mint path end-to-end here.

## Summary The ust1-oracle (`terra1fmht0t6svq3n24zx03nkfja0m40zhfyyxkdcvlrkl6u7gfe6aagq4gch8n`) last updated at **2026-08-16 02:52 UTC** (`last_update_sec` = 1786848774, rate 1225650379515835109 ≈ 1.22565) and has not ticked since. With the window's `max_oracle_age_sec = 21600`, `ensure_oracle_usable` has been failing closed for ~3 hours and counting (oracle age ~9h at time of filing), so **no one can currently mint UST1 on mainnet** — the `Deposit` path reverts. Everything else is healthy: window `paused = false`, bridge `paused = false` (fee 30 bps), vFDUSD bridging works. ## Likely cause The `oracle-service` updater isn't running / confirming ticks — README still lists "oracle-service Coolify (normalized ticks)" as pending under mainnet status. Per **INV-ORACLE-OPS-SILENCE-001** the service should emit a high-visibility liveness alert on >6h silence; that alert path may also be worth verifying since the silence has now exceeded it. ## Impact With the click-wrap live and the UST1 pools seeded, the mint window is the intended $1.00 source of UST1. While it's closed, UST1 can only be acquired from the thin DEX pools — which is exactly how UST1 got pushed to ~$1.16 over the launch weekend, and it undercuts the mint-arb that's supposed to hold UST1 at $1. ## Ready to verify We have ~114 vFDUSD staged (BSC, Venus) and the Terra side automated (bridge claim + window deposit) — happy to be the live probe the moment the oracle is fresh and confirm the full mint path end-to-end here.

PlasticDigits commented 2026-08-16 12:58:09 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in issue #32

mentioned in issue #32

PlasticDigits commented 2026-08-16 12:58:09 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

marked as related to #32

marked as related to #32

PlasticDigits commented 2026-08-16 13:26:57 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

mentioned in merge request !46

mentioned in merge request !46

PlasticDigits commented 2026-08-16 13:42:04 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Live status after oracle-service Coolify deploy (!46 / #32)

The 9h stale window from this incident is not current. At 2026-08-16 ~13:40 UTC:

• Oracle last_update_sec=1786881183 (11:53:03 UTC, tx C903047F…), age ~1.8 h / max 6 h
• Window + oracle unpaused; /ust1 shows Oracle Fresh
• DEX UST1_OPS_STRICT_STALE=1 health script PASS

Mint/redeem should be possible again on the window. Leave this issue open until a Venus-flat stretch shows the new heartbeat (event=oracle_heartbeat after 4h) keeping age inside 6h — first eligible include expected ~16:40 UTC today if Venus stays unchanged. Details on #32.

## Live status after oracle-service Coolify deploy (!46 / #32) The 9h stale window from this incident is **not current**. At 2026-08-16 ~13:40 UTC: - Oracle `last_update_sec=1786881183` (**11:53:03 UTC**, tx [`C903047F…`](https://finder.terra.money/columbus-5/tx/C903047F96FAA558462481D2A515B3F7911AD56DB30E669FF40128746E7748F9)), age **~1.8 h** / max **6 h** - Window + oracle **unpaused**; [`/ust1`](https://dex.cl8y.com/ust1) shows **Oracle Fresh** - DEX `UST1_OPS_STRICT_STALE=1` health script **PASS** Mint/redeem should be possible again on the window. **Leave this issue open** until a Venus-flat stretch shows the new heartbeat (`event=oracle_heartbeat` after 4h) keeping age inside 6h — first eligible include expected ~16:40 UTC today if Venus stays unchanged. Details on [#32](https://gitlab.com/PlasticDigits/ust1-window/-/issues/32).

PlasticDigits commented 2026-08-17 00:52:20 +00:00 (Migrated from gitlab.com)

• Copy link
• Copy Markdown

Status check (on-chain + Coolify logs) — 2026-08-17 ~00:51 UTC

The #31 stale window is cleared. Mint/redeem should work: oracle age ~12 min / 6 h, both oracle and window unpaused. Logs and columbus-5 LCD agree on the same three includes.

Live LCD (effective_swap / oracle State)

Field	Value
Oracle	terra1fmht0t6svq3n24zx03nkfja0m40zhfyyxkdcvlrkl6u7gfe6aagq4gch8n
rate	1225686890014493288 ≈ 1.22569
last_update_sec	1786927136 = 2026-08-17 00:38:56 UTC
Age / max_oracle_age_sec	~733 s / 21600 s (~5.8 h headroom; stale-at ~06:38 UTC)
Oracle paused	false
Window paused	false
day_baseline_rate	1225685240641120292 (utc_day_id=20682)

Log ↔ chain match (operator terra1hm3ph…)

Coolify tick_kind="rate_change" confirms (CheckTx → DeliverTx code=0 → oracle_event_confirm → oracle_state_confirm) match LCD wasm action=update_rate on this contract:

UTC	Tx	Log proposed	On-chain update_rate.rate	DeliverTx
2026-08-16 16:38:40	017F5DAD…	1225677131057590071	same	code 0, height 29976290
2026-08-16 20:38:49	EC2E1D8C…	1225685240641120292	same	code 0, height 29978733
2026-08-17 00:38:56	8329B298…	1225686890014493288	same (current State.rate)	code 0, height 29981174

Inter-tick gaps 14409 s and 14407 s — just over the 4h throttle (INV-ORACLE-THROTTLE-001 / min_interval: 14400). Current State.rate equals the last confirmed tx.

What the logs show besides those includes

• Startup knobs match the invariant: poll_interval_secs=3600, max_silence_since_broadcast_secs=21600, window_default_max_oracle_age_secs=21600.
• Equal Venus vs on-chain → no rate change from BSC (cheap skip; no tx). After C903047F (1225669001749054409) this held until Venus moved.
• Venus moved inside 4h after EC2E1D8C → three UpdateTooSoon { min_interval: 14400 } skips for proposed 1225686890014493288 vs current 1225685240641120292. Next poll after throttle submitted 8329B298. Policy is doing the right thing, not stalling.
• network min gas price probe failed / LCD 501 Not Implemented is warn-only; fallback TERRA_GAS_PRICE=28.325 still included all three txs.
• Confirm path is the #32 / INV-ORACLE-LIVENESS-001 one: wasm events first, then LCD State match. No lagged-State fail-close in this stretch.

Still open on this issue

No tick_kind="heartbeat" / event=oracle_heartbeat yet — Venus kept inching up, so every include was a real rate_change. Heartbeat is the leftover from the last note: prove a same-rate UpdateRate after 4h on a flat Venus stretch so age cannot drift to 6h again. Leave open for that; the mint-block from this incident is not current.

## Status check (on-chain + Coolify logs) — 2026-08-17 ~00:51 UTC The #31 stale window is **cleared**. Mint/redeem should work: oracle age **~12 min / 6 h**, both oracle and window **unpaused**. Logs and columbus-5 LCD agree on the same three includes. ### Live LCD (`effective_swap` / oracle `State`) | Field | Value | |-------|--------| | Oracle | `terra1fmht0t6svq3n24zx03nkfja0m40zhfyyxkdcvlrkl6u7gfe6aagq4gch8n` | | `rate` | `1225686890014493288` ≈ 1.22569 | | `last_update_sec` | `1786927136` = **2026-08-17 00:38:56 UTC** | | Age / `max_oracle_age_sec` | **~733 s / 21600 s** (~5.8 h headroom; stale-at ~06:38 UTC) | | Oracle `paused` | `false` | | Window `paused` | `false` | | `day_baseline_rate` | `1225685240641120292` (`utc_day_id=20682`) | ### Log ↔ chain match (operator `terra1hm3ph…`) Coolify `tick_kind="rate_change"` confirms (CheckTx → DeliverTx `code=0` → `oracle_event_confirm` → `oracle_state_confirm`) match LCD wasm `action=update_rate` on this contract: | UTC | Tx | Log `proposed` | On-chain `update_rate.rate` | DeliverTx | |-----|----|----------------|-----------------------------|-----------| | 2026-08-16 16:38:40 | [`017F5DAD…`](https://finder.terra.money/columbus-5/tx/017F5DADDF82C9B4524E07F56C7C94D45464CD3628BF4EB081D1605A161A65E7) | `1225677131057590071` | **same** | code 0, height 29976290 | | 2026-08-16 20:38:49 | [`EC2E1D8C…`](https://finder.terra.money/columbus-5/tx/EC2E1D8CBA6E067FFF8D303856AD671F9CC51B82DED0DC5C667822BE1A2DEA03) | `1225685240641120292` | **same** | code 0, height 29978733 | | 2026-08-17 00:38:56 | [`8329B298…`](https://finder.terra.money/columbus-5/tx/8329B29832D31E5A47EBF7950C1CAAE396550F762F6AC6845FD2635441757058) | `1225686890014493288` | **same** (current `State.rate`) | code 0, height 29981174 | Inter-tick gaps **14409 s** and **14407 s** — just over the 4h throttle (`INV-ORACLE-THROTTLE-001` / `min_interval: 14400`). Current `State.rate` equals the last confirmed tx. ### What the logs show besides those includes - Startup knobs match the invariant: `poll_interval_secs=3600`, `max_silence_since_broadcast_secs=21600`, `window_default_max_oracle_age_secs=21600`. - Equal Venus vs on-chain → `no rate change from BSC` (cheap skip; no tx). After `C903047F` (`1225669001749054409`) this held until Venus moved. - Venus moved **inside** 4h after `EC2E1D8C` → three `UpdateTooSoon { min_interval: 14400 }` skips for proposed `1225686890014493288` vs current `1225685240641120292`. Next poll after throttle submitted `8329B298`. Policy is doing the right thing, not stalling. - `network min gas price probe failed` / LCD `501 Not Implemented` is **warn-only**; fallback `TERRA_GAS_PRICE=28.325` still included all three txs. - Confirm path is the #32 / INV-ORACLE-LIVENESS-001 one: wasm events first, then LCD `State` match. No lagged-State fail-close in this stretch. ### Still open on this issue No `tick_kind="heartbeat"` / `event=oracle_heartbeat` yet — Venus kept inching up, so every include was a real `rate_change`. Heartbeat is the leftover from the last note: prove a same-rate `UpdateRate` after 4h on a flat Venus stretch so age cannot drift to 6h again. Leave open for that; the mint-block from this incident is not current.

PlasticDigits (Migrated from gitlab.com) closed this issue 2026-08-24 00:50:53 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

issue/50

renovate/configure

No results found.

Labels

Clear labels   

enhancement

  

governance

  

QA

  

testing

No labels

enhancement

governance

QA

testing

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

code/ust1-window#31

No description provided.