Fail over provider heads below canonical progress

PlasticDigits commented

2026-10-14 12:33:38 +00:00

Owner

A cached provider head can be fresh and schema-valid while lying below the already committed canonical cursor. The worker selected it and returned503 instead of trying a healthy alternative. Production sampling reproduced3/45 failures with a sole incoherentLCD, then5/45 after adding a coherent primary whose cached lower head hid the fallback.

Select Terra/BSC heads against the cursor after any verified rewind (BSC retains15 confirmations), and fail over lower candidates before selecting a head. Ask caches to revalidate mutable LCD latest/transaction-search responses. Physical manifests, ordered execution sets, canonical hashes, origin freshness, atomic accounting and genuine all-provider failure refusals remain enforced. Document the independently verified primary/fallback and the unsuccessful live samples rather than accepting intermittent green.

Validation: full fresh SCRAM PostgreSQL release11 on Rust1.88.0:74 unit/supervisor tests,118 integration tests,245 schema-valid API responses and146 frontend tests, production build, both nginx checks and Gitleaks. Actual HTTP-to-database regressions cover both-chain fallback, unchanged state when all heads are below the cursor, genuine rewind using its lower ancestor, simulated cache revalidation and atomic rejection of truly incoherent origin responses. Independent Binodes adapter/head/hash/search capability checks pass; live post-deployment stability remains pending.

Related #36 #38 #39 #41 #42 #43. Invariants and third-party agent guidance are crosslinked in docs/LEDGER_INVARIANTS.md, docs/OPS.md and skills/AGENTS_LEDGER_HARDENING.md. Full evidence: docs/verification/2026-10-14-issues-36-43.md. Issues36–43 remain open until final live criteria pass.

A cached provider head can be fresh and schema-valid while lying below the already committed canonical cursor. The worker selected it and returned503 instead of trying a healthy alternative. Production sampling reproduced3/45 failures with a sole incoherentLCD, then5/45 after adding a coherent primary whose cached lower head hid the fallback. Select Terra/BSC heads against the cursor after any verified rewind (BSC retains15 confirmations), and fail over lower candidates before selecting a head. Ask caches to revalidate mutable LCD latest/transaction-search responses. Physical manifests, ordered execution sets, canonical hashes, origin freshness, atomic accounting and genuine all-provider failure refusals remain enforced. Document the independently verified primary/fallback and the unsuccessful live samples rather than accepting intermittent green. Validation: full fresh SCRAM PostgreSQL release11 on Rust1.88.0:74 unit/supervisor tests,118 integration tests,245 schema-valid API responses and146 frontend tests, production build, both nginx checks and Gitleaks. Actual HTTP-to-database regressions cover both-chain fallback, unchanged state when all heads are below the cursor, genuine rewind using its lower ancestor, simulated cache revalidation and atomic rejection of truly incoherent origin responses. Independent Binodes adapter/head/hash/search capability checks pass; live post-deployment stability remains pending. Related #36 #38 #39 #41 #42 #43. Invariants and third-party agent guidance are crosslinked in docs/LEDGER_INVARIANTS.md, docs/OPS.md and skills/AGENTS_LEDGER_HARDENING.md. Full evidence: docs/verification/2026-10-14-issues-36-43.md. Issues36–43 remain open until final live criteria pass.

PlasticDigits added 2 commits

2026-10-14 12:33:38 +00:00

fix(ledger): fail over cached heads against canonical progress 4d476ab295

Select fresh provider heads above the post-rewind cursor and preserve BSC confirmation depth. Revalidate mutable LCD requests at gateways while retaining complete manifest and canonical checks. Cover lower-head failover, all-behind atomic rejection, real rewind and cached versus inconsistent origin responses through Postgres.

docs: record head failover verification and provider reliability loop

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

83d294993e

PlasticDigits merged commit 3631b4aa07 into main

2026-10-14 12:37:40 +00:00

PlasticDigits referenced this pull request

2026-10-14 14:19:41 +00:00

fix(ledger): validate complete Terra execution responses and BSC provider events #36

PlasticDigits referenced this pull request

2026-10-14 14:19:47 +00:00

fix(ledger): make transfer accounting exactly once across registration and concurrent writers #37