fix(voting): preserve ledger continuity and require fresh canonical dual-chain snapshots #38
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Scope and source
Priority: P0. Gap coverage: GAP-004, GAP-008, GAP-010. Derived from GAP_1791945837.md, reviewed code commit
f8dbe93df953a78b17cb3304b717dcf84c4cd517. Currente476a68da09645c2308cbf47b90211fcaffcb7afadds only the report, so implementation findings remain applicable. Report evidence is local reproduction/source inspection unless explicitly proposed; no production exploit or current live-readiness claim.Coverage and sequencing: gap backlog tracker. Workstream 03. Dependencies: 01, 02.
Current codebase
A cursor more than 10,000 blocks behind silently jumps to tip while retaining old balances. Opening accepts missing/malformed tips as zero;
{0,0}opening was reproduced. Terra rewind checks one block, stores height/hash separately and ignores hash-fetch failure; BSC indexes latest head without automatic canonicality/finality checks. Frozen proposals have no orphan-validity metadata.Why this implementation is needed
Outages and forks can retain spent power or freeze zero/stale weights while appearing current. Automatic starts would amplify these errors.
Relevant files
ledger/src/ingest.rsledger/src/db.rsledger/src/lcd.rsledger/src/bsc.rsoperator-voting/src/api.rsoperator-voting/src/db.rsledger/migrations/20260825000001_ledger.sqldocs/LEDGER_INVARIANTS.mddocs/OPERATOR_VOTING.mddocs/OPS.mdRecommended solution
Persist continuity epochs, observed timestamps, canonical block hashes and degraded reasons. Distinguish first boot without electorate from lost registered-wallet history. For unavailable forward history, establish an explicit new live-balance baseline/eligibility epoch, preserving prior frozen proposals; never pretend skipped history was indexed. Track recent canonical hashes and find a common ancestor within a documented horizon; stop/reconcile beyond it. Define bounded BSC finality and freshness criteria for both chains. Opening atomically selects validated checkpoints and refuses absent/zero/stale/degraded/orphaned tips with a retryable response. Document disclosure/review for orphaned frozen advisory polls without rewriting votes.
Constraints and guardrails
Preserve standalone offchain advisory voting, no new smart contracts or archive-node dependency, BSC as a core electorate, raw 18-decimal integer balances and separate Terra/BSC address identities. Keep Legal fail-closed transactional UI and the proven DEX Terra/Bridge EVM connectors; no browser BSC balance RPC, seed/key prompt or embedded admin/model credential. API must retain a restricted role separate from the ledger writer. Freeze at actual opening, never at draft creation or a client-supplied height. Agents own design, implementation, review and evidence under repository guidance; do not shift approvals/tests to the human or invent PASS evidence.
Acceptance criteria
Lost continuity cannot be hidden by a cursor jump. Recovery is durable and restart-safe, and preserves already-frozen power. Manual and future scheduled opening fail closed unless both chains are fresh, complete and canonical under the recorded policy. Height/hash and selected freeze metadata remain consistent.
Test plan: supported paths, failure paths and concurrency
Fresh boot with/without electorate; short/long outage with transfers in skipped interval; durable recovery/restart and later eligible snapshots. Missing, zero, malformed, stale or degraded Terra/BSC independently. One-block/multi-block forks, beyond-horizon forks, hash-fetch failure, reorg immediately before freeze and crash between progress writes. Concurrent opening and reconciliation must select one consistent state.
Test plan: attack, hack and abuse vectors
Stale holder drains balance during outage; provider supplies old tip or divergent hash; attacker triggers freeze during reconciliation or races a fork; resubmitted freeze tries to alter existing snapshots. Verify stale power never enters a new trustworthy snapshot and incident metadata cannot relabel old votes.
Verification and completion criteria
Attach gap recovery and reorg scenario evidence, documented epoch/finality/freshness rules, migration/rollback results and atomic freeze assertions for both chains. Update the documented jump policy and prove restored readiness matches reconciled accounting, not merely current cursor height.
Record commit and pinned toolchain, commands/results and sanitized artifacts. Relevant Rust changes require
cargo test --workspace --libplus isolated Postgres integration (explicit guarded LEDGER_TEST_DATABASE_URL); frontend changes requirenpm testand production build plus relevant browser cases. Run secret scanning for new fixtures/config. Exercise fresh/upgrade migrations when schema/grants change; report skipped/live-only checks explicitly.Implementation status after main merge PR #53, commit
cd07f00f32f04ad02f8b760ed34588fd9645919e:Services are paused while existing storage is audited and the corrected pinned release is prepared. No production wallet transaction or hardware-wallet/Legal QA is claimed. This is an interim report; the issue remains open and the final comment will map the original criteria to actual live results and any unverified cases.
The bounded read-only deployment
o47uvv9xxkwhmmf6xcutmkfycompleted from main1562434c4237367f828ed71748ef71f116a5ff82. Production inspection confirmed15registrations plus2BSCtransfers/2balance checkpoints under the legacyvotingschema; corresponding canonicalpublicelectorate/history tables are empty. The new public migration tracker contains11migrations while the legacy tracker contains5. This explains the undefinedbaseline_heightfailures and proves that switching pool paths alone would omit the existing electorate.Accounting remains paused in read-only inspection mode. A forward import is being implemented and tested to preserve legacy registrations/history and frozen weights, retain the original tables for audit, reject conflicting data, and require explicit canonical baseline recovery. No issue is closed and no healthy voting deployment is claimed yet. The full latest isolated suite passed73unit/supervisor tests,92integration tests,245validated API responses,146frontend tests, production build/nginx/Gitleaks.
Implemented through PR53, 54, 55, 56, 57, 58, and PR59. Final code release
5cd9363(PR59; CI passed) is deployed across all three Coolify services, each finished and healthy. This draft records completed issue criteria for the parent to post after publishing the final evidence artifact.Verification checklist:
Lost registered continuity is durable degraded state, never a silent cursor jump. Explicit canonical live-baseline recovery creates epochs without rewriting old frozen power.
Indexed Terra passes one-block, multi-block and full retained-horizon fork scenarios; BSC passes multi-block/common-ancestor, unavailable-ancestor and unfinalized-baseline fork cases. Missing/malformed/stale/zero heads, hash failures and atomic freeze/reconciliation races are covered. This does not claim every chain/scenario cross-product as a separately executed fixture.
A holder who drains during an outage retains old frozen 1000 but has reconciled new eligible weight 0. Old orphaned advisory polls receive incident metadata without rewritten votes or misleading validity.
Opening freezes at the actual transaction using consistent certified height/hash/epoch/policy metadata; absent, stale, degraded, orphaned or unverified state fails closed. The same invariant applies to future scheduled callers.
Fresh/populated/legacy-namespace upgrades and conflicting-migration rollback are tested; original frozen rows, electorate, source tables and tracker survive. Final runtime configuration confirms inspection/recovery-only flags are false.
Earlier production readiness correctly returned 503 for canonical BSC lag 278 even with healthy liveness, Terra and intents. The final post-throughput windows above demonstrate sustained dual-chain readiness and live registered-balance admission, while final inventory verifies preserved imported accounting.
Pinned Rust1.88.0 complete fresh disposable release13 passed: 74 unit/supervisor tests, 121 integration tests (66 ledger hardening), 245 schema-validated API responses and 146 frontend tests, with production build/browser, nginx and Gitleaks checks. PR59 CI passed. Commands and results are recorded in the release report.
All three Coolify deployments of
5cd9363finished healthy. Private readiness was green in 240 + 240 + 600 samples (1080 total), with maximum Terra lag 2 and BSC lag 17; both checkpoints stayed complete/canonical and intents/baselines healthy. The final two public rounds passed 45/45 each through 14:07:21 UTC. Live smoke (13 checks), four canonical checks, both original expired-action exact receipt replays, both-chain offchain registration/receipt checks and runtime configuration passed. Grafana/Loki has no post-activation worker warnings; the direct Coolify RPC warning at 13:56:21 belongs to the old container before 13:58:22 activation. See final live evidence.Invariants, recovery/finality/freshness/rollback policy and third-party agent guidance are crosslinked in LEDGER_INVARIANTS.md, OPS.md and AGENTS_LEDGER_HARDENING.md.
A fresh-looking head below the verified post-rewind cursor tries another provider (BSC requires cursor plus 15); all-behind responses leave exact checkpoint/freshness/accounting unchanged. A genuine canonical rewind uses its new ancestor floor. A controlled 90ms-RPC/450ms-block HTTP/Postgres test retains actual final BSC tips with lag 16, 16, 17, 17, 16 over five polls; the policy remains 15 confirmations plus 2 additional blocks, 120-second freshness and the documented canonical horizon. This deterministic timing proof is complemented by the final live windows above.
Regression evidence:
indexed_terra_one_multi_and_beyond_retained_horizon_forks_preserve_frozen_power,multi_block_bsc_fork_rewinds_to_common_ancestor_and_orphans_poll,freeze_racing_reconciliation_and_beyond_horizon_fork_uses_atomic_source_state,drained_holder_missing_terra_interval_requires_recovery_before_new_freeze,drained_holder_missing_bsc_interval_requires_confirmed_recovery_before_new_freeze,indexed_terra_canonical_hash_failure_keeps_checkpoint_then_recovers_same_epoch,cached_lower_head_fails_over_at_canonical_cursor_and_all_lower_publish_nothing_both_chains, andmoving_bsc_head_at_production_latency_commits_true_tip_within_unchanged_freshness_budget. The guarded forward-import/migration suite and original-field live inventory address the additional legacy namespace comment; see the release report.Observed exception: the first public round had one unexplained 503 at 14:00:58.805988 UTC (44/45), recovering by 14:01:01; it fell outside the first private capture window. That earlier round is not the final acceptance window. The subsequent two public rounds and 1080 private samples passed; this is bounded observation evidence, not a promise of permanent availability or a reason to relax fail-closed readiness.
Limits: no blockchain transaction, new contract, archive dependency, chain-identity merge or browser BSC balance RPC. Funded human/hardware/mobile wallet and Legal QA, prolonged external CPU stress and production database downgrade/destructive attacks are not claimed by the isolated fixtures or unfunded live probes. Grafana currently provides Loki only; no Prometheus datasource was available. Legacy hash-only receipts and uncertified polls retain their documented limits.
The body acceptance/completion criteria and current comment requirements are satisfied by the named deterministic regressions, final release checks and recorded live evidence. Closing is supported after the parent publishes the final artifact; no remaining mandatory implementation or verification gap is identified.
Final evidence is published through PR #60. All mandatory body/comment criteria are satisfied by the checklist above; closing this issue.