feat(oracle-service): quiet expected gas-probe WARN + duplicate Venus normalize INFO #29

New issue

Closed

opened 2026-08-09 06:42:23 +00:00 by PlasticDigits · 2 comments

PlasticDigits commented

2026-08-09 06:42:23 +00:00

(Migrated from gitlab.com)

Summary

Bundle two low-severity oracle-service log hygiene fixes observed in production Coolify logs:

Item	Symptom
Gas-price probe WARN	Every successful `UpdateRate` submit emits `warn!` `network min gas price probe failed; using configured TERRA_GAS_PRICE floor` with `error=node config query 501 Not Implemented`
Duplicate BSC normalize INFO	Every poll tick emits the same `normalized Venus exchangeRateStored → oracle RATE_SCALE units` line twice (identical `exchange_rate_stored` / `oracle_r`)

Both are expected consequences of current code paths (LCD without node-config API; 2-of-N EVM RPC consensus each calling bsc::read_exchange_rate_stored). They do not indicate failed updates in the reviewed log slice, but they drown signal and can train operators to ignore real warn!/error! lines.

Current codebase

Gas-price probe (INV-ORACLE-GAS-001 / audit M-14)

oracle-service/src/terra_tx.rs query_network_min_gas_price GETs {lcd}/cosmos/base/node/v1beta1/config and parses minimum_gas_price.
On non-success HTTP status (including 501 Not Implemented on the production LCD), it returns Err; resolve_gas_price then warn!s and falls back to the configured TERRA_GAS_PRICE floor.
resolve_gas_price is invoked on every fee computation path before broadcast, so a permanently unsupported LCD endpoint produces a WARN on every submit (and potentially every fee path that probes).
Fallback behavior itself is correct per INV-ORACLE-GAS-001: when the probe fails, use configured floor; when it works, max(configured, network_min).
Live config in logs: configured=28.325 (uluna/gas units as used by the service).

Duplicate Venus normalize INFO

oracle-service/src/main.rs run_once calls evm_rpc::run_with_evm_rpc_rate_consensus, which queries at least two BSC RPC URLs in parallel (tokio::join! on urls[0] / urls[1], optional third for ties).
Each successful read goes through bsc::read_exchange_rate_stored, which unconditionally tracing::info!s the full normalization payload (exchange_rate_stored, decimals, oracle_r).
Consensus agreement already has its own sparse logs in evm_rpc.rs (exact match → silent success; within 0.01% → one info; tie-break → warn). The per-provider normalize info! therefore duplicates on every healthy tick.

Policy skip lines use outcome="failed" for normal UpdateTooSoon — separate alerting concern; do not expand this issue into policy-log redesign unless a one-line doc note is needed.

Why this is needed

Signal-to-noise: Production logs already rely on structured tracing for ops (no Prometheus). Recurring expected WARNs and duplicated INFO lines train pager blindness and make real DeliverTx / liveness / RPC-mismatch WARN/ERROR harder to spot.
Gas probe semantics: A 501 / missing endpoint is a known LCD capability gap, not a transient outage. Logging it at warn every submit overstates severity relative to a previously working probe that suddenly fails (which should stay loud).
Gas safety still matters: Operators must retain a clear path when the configured floor is below a real network minimum (underpriced txs / mempool rejection). Quieting expected 501 must not hide “probe worked and network min rose” or “probe used to work and now fails.”
BSC normalize: One INFO per tick (or debug per provider + one tick summary) is enough to audit Venus → RATE_SCALE conversion; two identical lines add no consensus evidence (consensus already logs disagreements).

Constraints / guardrails

Do not weaken INV-ORACLE-GAS-001: fee must still be max(configured, network_min) when the probe succeeds; configured floor when it does not.
Do not weaken EVM consensus: still require ≥2 URLs, 0.01% agreement, optional third tie-break (evm_rpc.rs). Logging changes must not skip RPCs or accept a single provider.
HTTPS / redaction: keep existing LCD URL redaction habits; do not log secrets or full mnemonic material.
No false “healthy probe”: treating 501 as success-with-None is OK only if docs/skill state that the LCD lacks the endpoint; do not claim adaptive network gas when the probe never runs.
Severity ladder (recommended):
- Expected permanent unsupported (e.g. 501 / clear “not implemented”) → at most once per process (or debug thereafter), not warn every submit.
- Transient / unexpected probe failure after prior success, parse anomalies, or network min above configured → keep warn (or escalate).
- Successful probe that raises fee above configured → optional info for ops visibility.
Normalize logging: prefer one tick-level INFO (or demote per-provider lines to debug with rpc_url redacted/host-only). Keep enough fields to verify oracle_r math in incidents.
Scope: oracle-service + docs/skill touch-ups only; no on-chain policy, poll/silence defaults, or contract changes.
Compat: env knobs (TERRA_GAS_PRICE, BSC_RPC_URLS, etc.) remain; optional new env only if clearly documented (prefer code behavior over new knobs).

Relevant files

Path	Role
`oracle-service/src/terra_tx.rs`	`query_network_min_gas_price`, `resolve_gas_price`, `effective_gas_price`, fee path
`oracle-service/src/bsc.rs`	Venus `exchangeRateStored` read + normalize `info!`
`oracle-service/src/evm_rpc.rs`	Multi-RPC consensus; existing consensus log lines
`oracle-service/src/main.rs`	`run_once` tick wiring
`oracle-service/src/config.rs`	`TERRA_GAS_PRICE` / RPC env
`skills/audit-hardening-bundle/SKILL.md`	Documents INV-ORACLE-GAS-001
`docs/DEPLOYMENT.md`	Operator env / LCD notes
`README.md`	Observability summary (if log expectations change)

Recommended direction

A) Gas-price probe WARN

Classify probe outcomes:
- Unsupported LCD (501, and optionally other documented “endpoint missing” statuses) → return Ok(None) (or a dedicated result) and log once at process start / first fee resolve (info or single warn), then stay quiet while still using configured floor.
- Transport / 5xx / timeout / parse failure → keep warn (ideally rate-limited or “state changed” so flaps are visible).
- Success with Some(network_min) → effective_gas_price; if network_min > configured, consider a single clear info! that the fee was raised.
Add wiremock coverage for 501 → no recurring warn, 200 + min gas → max(), and 500 → warn + floor.
Document in DEPLOYMENT.md / audit-hardening skill that columbus-5 LCD may not implement node config; operators must keep TERRA_GAS_PRICE ≥ live min gas.

B) Duplicate BSC normalize lines

Demote the per-call normalize log in bsc::read_exchange_rate_stored to debug! or remove it and emit a single info! in run_once / consensus after a rate is accepted (include oracle_r; optional note that N providers agreed).
Preserve WARN/ERROR paths for RPC mismatch and single-provider failure unchanged.
Unit/integration tests that drive consensus with two mock reads should assert log intent via behavior (returned rate) and, if the crate already asserts tracing, that INFO normalize is not duplicated—otherwise document manual log check in test plan.

Acceptance criteria

On an LCD that returns 501 for /cosmos/base/node/v1beta1/config, a sequence of multiple successful UpdateRate submits does not emit a gas-probe warn! on every submit (at most once per process lifetime, or debug-only thereafter).
When the probe succeeds, fee is still max(configured, network_min) (INV-ORACLE-GAS-001 unchanged).
Unexpected probe failures (e.g. 500 / transport) still surface at warn (not silently ignored forever without any operator signal).
A healthy consensus tick with two agreeing RPCs emits at most one INFO-level Venus normalize / accepted-rate line (not two identical normalize infos).
Consensus still requires two agreeing providers within 0.01%; mismatch / third-provider tie-break behavior unchanged.
skills/audit-hardening-bundle/SKILL.md and docs/DEPLOYMENT.md note LCD 501 / configured floor expectations and the quieter log behavior.
cargo test -p ust1-oracle-service passes; no new clippy warnings in touched files.

Test plan (all paths)

Gas probe

Path	Expect
LCD 200 + `minimum_gas_price` below configured	fee = configured; no warn spam
LCD 200 + `minimum_gas_price` above configured	fee = network min; INV-ORACLE-GAS-001
LCD 200 + missing/unparsable `minimum_gas_price`	floor = configured; defined warn/info once or documented
LCD 501 Not Implemented	configured floor; no per-submit warn loop
LCD 500 / connection error	warn (visible); configured floor; broadcast can still proceed
Probe success then later failure	operator-visible warn on transition (if state-tracking implemented)
Existing `effective_gas_price_*` unit tests	still green

BSC normalize / consensus

Path	Expect
Two URLs agree exactly	one accepted rate; ≤1 INFO normalize/summary
Two URLs agree within 0.01% but not equal	accept; existing consensus info OK; no duplicate normalize INFO pair
Two disagree, third ties with first/second	existing warn + correct winner; no extra normalize INFO spam requirement beyond “not 3× identical info”
Only one URL succeeds, no third	error (unchained)
Both fail	error
`debug` level enabled	per-provider detail still available if demoted to debug

Regression

DeliverTx confirm + State check + liveness record path unchanged (INV-ORACLE-LIVENESS-001).
Policy skip / equal-rate / pause ticks unchanged aside from log volume.

Test plan (attack / hack / abuse vectors)

Vector	Expectation
Malicious / compromised LCD returns huge `minimum_gas_price`	fee rises; operator can see elevated fee via info/logs; service must not panic; consider cap only if already discussed elsewhere—do not silently ignore a successful probe
Malicious LCD returns 501 forever while network min rises above `TERRA_GAS_PRICE`	txs may fail CheckTx; must remain detectable via broadcast/confirm failures (not hidden by quieting 501); docs require monitoring floor
LCD returns 200 with garbage gas string	treat as probe failure / `None`; no underpriced “0” fee from parse bugs
BSC RPC A lies low, B lies high beyond 0.01%	consensus rejects; no UpdateRate on disagreed rates
BSC RPC A and B collude within 0.01% on wrong rate	out of scope for logging fix; existing consensus trust model unchanged—do not reduce to 1-of-N to “fix” duplicate logs
Log injection via weird numeric display	keep tracing fields typed (`%` / structured); no unsanitized multi-line RPC bodies at INFO
Log volume DoS / disk fill	quieter steady-state reduces self-DoS; still bound tick rate by `POLL_INTERVAL_SECS`
Downgrade adaptive gas by forcing 501 (path preference if multiple LCDs later)	single-LCD today; document that unsupported probe ⇒ configured floor only

Verification criteria

Unit/wiremock: new or extended tests in terra_tx cover 501 quieting + success max(); evm_rpc / bsc tests still pass with logging change.
Local/manual: run oracle-service against a mock or staging LCD that 501s the config endpoint; submit or dry-run fee resolve ≥3 times; confirm ≤1 gas-probe warn.
Local/manual: one run_once with two mock RPCs returning the same mantissa; log capture shows a single normalize/summary INFO.
Production follow-up: after deploy, Coolify log sample over ≥1 successful update + several no-op ticks shows no recurring gas-probe WARN storm and no twin normalize INFOs; real WARN/ERROR (DeliverTx fail, silence) still appear when injected/forced in staging.
Docs: DEPLOYMENT / skill mention 501 behavior and that TERRA_GAS_PRICE remains the safety floor when adaptive probe is unavailable.

## Summary Bundle two low-severity **oracle-service log hygiene** fixes observed in production Coolify logs: | Item | Symptom | |------|---------| | **Gas-price probe WARN** | Every successful `UpdateRate` submit emits `warn!` `network min gas price probe failed; using configured TERRA_GAS_PRICE floor` with `error=node config query 501 Not Implemented` | | **Duplicate BSC normalize INFO** | Every poll tick emits the same `normalized Venus exchangeRateStored → oracle RATE_SCALE units` line **twice** (identical `exchange_rate_stored` / `oracle_r`) | Both are expected consequences of current code paths (LCD without node-config API; 2-of-N EVM RPC consensus each calling `bsc::read_exchange_rate_stored`). They do **not** indicate failed updates in the reviewed log slice, but they drown signal and can train operators to ignore real `warn!`/`error!` lines. --- ## Current codebase ### Gas-price probe (INV-ORACLE-GAS-001 / audit M-14) - `oracle-service/src/terra_tx.rs` `query_network_min_gas_price` GETs `{lcd}/cosmos/base/node/v1beta1/config` and parses `minimum_gas_price`. - On non-success HTTP status (including **501 Not Implemented** on the production LCD), it returns `Err`; `resolve_gas_price` then `warn!`s and falls back to the configured `TERRA_GAS_PRICE` floor. - `resolve_gas_price` is invoked on every fee computation path before broadcast, so a permanently unsupported LCD endpoint produces a **WARN on every submit** (and potentially every fee path that probes). - Fallback behavior itself is correct per **INV-ORACLE-GAS-001**: when the probe fails, use configured floor; when it works, `max(configured, network_min)`. - Live config in logs: `configured=28.325` (uluna/gas units as used by the service). ### Duplicate Venus normalize INFO - `oracle-service/src/main.rs` `run_once` calls `evm_rpc::run_with_evm_rpc_rate_consensus`, which queries **at least two** BSC RPC URLs in parallel (`tokio::join!` on `urls[0]` / `urls[1]`, optional third for ties). - Each successful read goes through `bsc::read_exchange_rate_stored`, which unconditionally `tracing::info!`s the full normalization payload (`exchange_rate_stored`, decimals, `oracle_r`). - Consensus agreement already has its own sparse logs in `evm_rpc.rs` (exact match → silent success; within 0.01% → one info; tie-break → warn). The per-provider normalize `info!` therefore duplicates on every healthy tick. ### Related (out of scope unless trivial) - Policy skip lines use `outcome="failed"` for normal `UpdateTooSoon` — separate alerting concern; do **not** expand this issue into policy-log redesign unless a one-line doc note is needed. --- ## Why this is needed 1. **Signal-to-noise:** Production logs already rely on structured `tracing` for ops (no Prometheus). Recurring expected WARNs and duplicated INFO lines train pager blindness and make real DeliverTx / liveness / RPC-mismatch WARN/ERROR harder to spot. 2. **Gas probe semantics:** A **501 / missing endpoint** is a known LCD capability gap, not a transient outage. Logging it at `warn` every submit overstates severity relative to a previously working probe that suddenly fails (which *should* stay loud). 3. **Gas safety still matters:** Operators must retain a clear path when the configured floor is below a *real* network minimum (underpriced txs / mempool rejection). Quieting expected 501 must not hide “probe worked and network min rose” or “probe used to work and now fails.” 4. **BSC normalize:** One INFO per tick (or debug per provider + one tick summary) is enough to audit Venus → `RATE_SCALE` conversion; two identical lines add no consensus evidence (consensus already logs disagreements). --- ## Constraints / guardrails 1. **Do not weaken INV-ORACLE-GAS-001:** fee must still be `max(configured, network_min)` when the probe succeeds; configured floor when it does not. 2. **Do not weaken EVM consensus:** still require ≥2 URLs, 0.01% agreement, optional third tie-break (`evm_rpc.rs`). Logging changes must not skip RPCs or accept a single provider. 3. **HTTPS / redaction:** keep existing LCD URL redaction habits; do not log secrets or full mnemonic material. 4. **No false “healthy probe”:** treating 501 as success-with-`None` is OK only if docs/skill state that the LCD lacks the endpoint; do not claim adaptive network gas when the probe never runs. 5. **Severity ladder (recommended):** - Expected permanent unsupported (e.g. **501** / clear “not implemented”) → at most **once per process** (or `debug` thereafter), not `warn` every submit. - Transient / unexpected probe failure after prior success, parse anomalies, or network min **above** configured → keep **warn** (or escalate). - Successful probe that raises fee above configured → optional info for ops visibility. 6. **Normalize logging:** prefer **one** tick-level INFO (or demote per-provider lines to `debug` with `rpc_url` redacted/host-only). Keep enough fields to verify `oracle_r` math in incidents. 7. **Scope:** oracle-service + docs/skill touch-ups only; no on-chain policy, poll/silence defaults, or contract changes. 8. **Compat:** env knobs (`TERRA_GAS_PRICE`, `BSC_RPC_URLS`, etc.) remain; optional new env only if clearly documented (prefer code behavior over new knobs). --- ## Relevant files | Path | Role | |------|------| | `oracle-service/src/terra_tx.rs` | `query_network_min_gas_price`, `resolve_gas_price`, `effective_gas_price`, fee path | | `oracle-service/src/bsc.rs` | Venus `exchangeRateStored` read + normalize `info!` | | `oracle-service/src/evm_rpc.rs` | Multi-RPC consensus; existing consensus log lines | | `oracle-service/src/main.rs` | `run_once` tick wiring | | `oracle-service/src/config.rs` | `TERRA_GAS_PRICE` / RPC env | | `skills/audit-hardening-bundle/SKILL.md` | Documents **INV-ORACLE-GAS-001** | | `docs/DEPLOYMENT.md` | Operator env / LCD notes | | `README.md` | Observability summary (if log expectations change) | --- ## Recommended direction ### A) Gas-price probe WARN 1. Classify probe outcomes: - **Unsupported LCD** (`501`, and optionally other documented “endpoint missing” statuses) → return `Ok(None)` (or a dedicated result) and log **once** at process start / first fee resolve (`info` or single `warn`), then stay quiet while still using configured floor. - **Transport / 5xx / timeout / parse failure** → keep `warn` (ideally rate-limited or “state changed” so flaps are visible). - **Success with `Some(network_min)`** → `effective_gas_price`; if `network_min > configured`, consider a single clear `info!` that the fee was raised. 2. Add wiremock coverage for **501 → no recurring warn**, **200 + min gas → max()**, and **500 → warn + floor**. 3. Document in `DEPLOYMENT.md` / audit-hardening skill that columbus-5 LCD may not implement node config; operators must keep `TERRA_GAS_PRICE` ≥ live min gas. ### B) Duplicate BSC normalize lines 1. Demote the per-call normalize log in `bsc::read_exchange_rate_stored` to `debug!` **or** remove it and emit a single `info!` in `run_once` / consensus after a rate is accepted (include `oracle_r`; optional note that N providers agreed). 2. Preserve WARN/ERROR paths for RPC mismatch and single-provider failure unchanged. 3. Unit/integration tests that drive consensus with two mock reads should assert log intent via behavior (returned rate) and, if the crate already asserts tracing, that INFO normalize is not duplicated—otherwise document manual log check in test plan. --- ## Acceptance criteria - [ ] On an LCD that returns **501** for `/cosmos/base/node/v1beta1/config`, a sequence of multiple successful `UpdateRate` submits does **not** emit a gas-probe `warn!` on every submit (at most once per process lifetime, or debug-only thereafter). - [ ] When the probe **succeeds**, fee is still `max(configured, network_min)` (INV-ORACLE-GAS-001 unchanged). - [ ] Unexpected probe failures (e.g. 500 / transport) still surface at **warn** (not silently ignored forever without any operator signal). - [ ] A healthy consensus tick with two agreeing RPCs emits **at most one** INFO-level Venus normalize / accepted-rate line (not two identical normalize infos). - [ ] Consensus still requires two agreeing providers within 0.01%; mismatch / third-provider tie-break behavior unchanged. - [ ] `skills/audit-hardening-bundle/SKILL.md` and `docs/DEPLOYMENT.md` note LCD 501 / configured floor expectations and the quieter log behavior. - [ ] `cargo test -p ust1-oracle-service` passes; no new clippy warnings in touched files. --- ## Test plan (all paths) ### Gas probe | Path | Expect | |------|--------| | LCD 200 + `minimum_gas_price` below configured | fee = configured; no warn spam | | LCD 200 + `minimum_gas_price` above configured | fee = network min; INV-ORACLE-GAS-001 | | LCD 200 + missing/unparsable `minimum_gas_price` | floor = configured; defined warn/info once or documented | | LCD **501 Not Implemented** | configured floor; **no per-submit warn loop** | | LCD 500 / connection error | warn (visible); configured floor; broadcast can still proceed | | Probe success then later failure | operator-visible warn on transition (if state-tracking implemented) | | Existing `effective_gas_price_*` unit tests | still green | ### BSC normalize / consensus | Path | Expect | |------|--------| | Two URLs agree exactly | one accepted rate; ≤1 INFO normalize/summary | | Two URLs agree within 0.01% but not equal | accept; existing consensus info OK; no duplicate normalize INFO pair | | Two disagree, third ties with first/second | existing warn + correct winner; no extra normalize INFO spam requirement beyond “not 3× identical info” | | Only one URL succeeds, no third | error (unchained) | | Both fail | error | | `debug` level enabled | per-provider detail still available if demoted to debug | ### Regression - [ ] DeliverTx confirm + State check + liveness record path unchanged (INV-ORACLE-LIVENESS-001). - [ ] Policy skip / equal-rate / pause ticks unchanged aside from log volume. --- ## Test plan (attack / hack / abuse vectors) | Vector | Expectation | |--------|-------------| | **Malicious / compromised LCD** returns huge `minimum_gas_price` | fee rises; operator can see elevated fee via info/logs; service must not panic; consider cap only if already discussed elsewhere—**do not silently ignore** a successful probe | | **Malicious LCD** returns 501 forever while network min rises above `TERRA_GAS_PRICE` | txs may fail CheckTx; must remain detectable via broadcast/confirm failures (not hidden by quieting 501); docs require monitoring floor | | **LCD returns 200 with garbage** gas string | treat as probe failure / `None`; no underpriced “0” fee from parse bugs | | **BSC RPC A lies low, B lies high** beyond 0.01% | consensus **rejects**; no UpdateRate on disagreed rates | | **BSC RPC A and B collude** within 0.01% on wrong rate | out of scope for logging fix; existing consensus trust model unchanged—do not reduce to 1-of-N to “fix” duplicate logs | | **Log injection** via weird numeric display | keep tracing fields typed (`%` / structured); no unsanitized multi-line RPC bodies at INFO | | **Log volume DoS / disk fill** | quieter steady-state reduces self-DoS; still bound tick rate by `POLL_INTERVAL_SECS` | | **Downgrade adaptive gas by forcing 501** (path preference if multiple LCDs later) | single-LCD today; document that unsupported probe ⇒ configured floor only | --- ## Verification criteria 1. **Unit/wiremock:** new or extended tests in `terra_tx` cover 501 quieting + success max(); `evm_rpc` / `bsc` tests still pass with logging change. 2. **Local/manual:** run oracle-service against a mock or staging LCD that 501s the config endpoint; submit or dry-run fee resolve ≥3 times; confirm ≤1 gas-probe warn. 3. **Local/manual:** one `run_once` with two mock RPCs returning the same mantissa; log capture shows a single normalize/summary INFO. 4. **Production follow-up:** after deploy, Coolify log sample over ≥1 successful update + several no-op ticks shows no recurring gas-probe WARN storm and no twin normalize INFOs; real WARN/ERROR (DeliverTx fail, silence) still appear when injected/forced in staging. 5. **Docs:** DEPLOYMENT / skill mention 501 behavior and that `TERRA_GAS_PRICE` remains the safety floor when adaptive probe is unavailable.

PlasticDigits commented

2026-08-16 12:58:09 +00:00

(Migrated from gitlab.com)