fix(ledger): isolate chain workers and expose supervised readiness and intent progress #39
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Scope and source
Priority: P1. Gap coverage: GAP-011, GAP-051. Derived from GAP_1791945837.md, reviewed code commit
f8dbe93df953a78b17cb3304b717dcf84c4cd517. Currente476a68da09645c2308cbf47b90211fcaffcb7afadds only the report, so implementation findings remain applicable. Report evidence is local reproduction/source inspection unless explicitly proposed; no production exploit or current live-readiness claim.Coverage and sequencing: gap backlog tracker. Workstream 04. Dependencies: 02, 03.
Current codebase
Terra runs first and can prevent BSC processing; up to 2,000 heights and all pending intents are serial. Early 20-second failures block later holders. API health always reports ok; ledger health masks SQL failures as zero and compares against registration heights instead of network freshness. Spawned workers are unsupervised and lack graceful shutdown.
Why this implementation is needed
A healthy HTTP process can conceal a stalled chain or unbounded registration backlog, making operational status and snapshot readiness misleading.
Relevant files
ledger/src/main.rsledger/src/register.rsledger/src/health.rsledger/src/db.rsledger/src/config.rsoperator-voting/src/api.rsdeploy/docker/ledger.Dockerfiledeploy/docker/operator-voting.Dockerfiledocs/OPS.mdRecommended solution
Run independent bounded Terra/BSC loops and fair indexed intent batches with per-item deadlines, retry backoff/jitter and bounded concurrency. Preserve the single-writer accounting discipline from 02. Supervise exits, define restart/fatal behavior and graceful signal shutdown. Separate liveness from DB/chain/continuity readiness; expose safe last-success times, chain lag, intent backlog/oldest age/error class and scheduler status once available. Alert on stalled workers and feed reliable readiness into opening admission from 03.
Constraints and guardrails
Preserve standalone offchain advisory voting, no new smart contracts or archive-node dependency, BSC as a core electorate, raw 18-decimal integer balances and separate Terra/BSC address identities. Keep Legal fail-closed transactional UI and the proven DEX Terra/Bridge EVM connectors; no browser BSC balance RPC, seed/key prompt or embedded admin/model credential. API must retain a restricted role separate from the ledger writer. Freeze at actual opening, never at draft creation or a client-supplied height. Agents own design, implementation, review and evidence under repository guidance; do not shift approvals/tests to the human or invent PASS evidence.
Acceptance criteria
Terra outage does not stop healthy BSC progress and vice versa. A failing early intent cannot starve later valid registrations. Unavailable DB or stale/dead worker makes readiness fail while liveness accurately reflects process state. Shutdown drains or rolls back bounded work and resumes without duplicated deltas.
Test plan: supported paths, failure paths and concurrency
Both providers healthy; either/both unavailable; recovery/backoff; large catch-up; empty/large intent queues with alternating failures. Force worker panic/exit, DB loss, malformed cursor and signals during RPC/DB commit. Observe last-success/lag/backlog transitions and restart behavior; test bounded fairness under sustained arrivals.
Test plan: attack, hack and abuse vectors
Flood registration intents, slow/hanging endpoint, poisoned early intent and task-crash loop; health scraping must not expose provider URLs, addresses unnecessarily or credentials. Prove bounded CPU/memory/retries and accurate readiness under denial-of-service conditions.
Verification and completion criteria
Attach deterministic worker/fairness tests and liveness/readiness endpoint examples for each failure state, plus supervised restart/shutdown evidence. Demonstrate no false green while a core chain stalls and unchanged exactly-once balances after restart.
Record commit and pinned toolchain, commands/results and sanitized artifacts. Relevant Rust changes require
cargo test --workspace --libplus isolated Postgres integration (explicit guarded LEDGER_TEST_DATABASE_URL); frontend changes requirenpm testand production build plus relevant browser cases. Run secret scanning for new fixtures/config. Exercise fresh/upgrade migrations when schema/grants change; report skipped/live-only checks explicitly.Implementation status after main merge PR #53, commit
cd07f00f32f04ad02f8b760ed34588fd9645919e:Services are paused while existing storage is audited and the corrected pinned release is prepared. No production wallet transaction or hardware-wallet/Legal QA is claimed. This is an interim report; the issue remains open and the final comment will map the original criteria to actual live results and any unverified cases.
Implemented through PR53, 54, 55, 56, 57, 58, and PR59. Final code release
5cd9363(PR59; CI passed) is deployed across all three Coolify services, each finished and healthy. This draft records completed issue criteria for the parent to post after publishing the final evidence artifact.Verification checklist:
5cd9363finished healthy. Private readiness was green in 240 + 240 + 600 samples (1080 total), with maximum Terra lag 2 and BSC lag 17; both checkpoints stayed complete/canonical and intents/baselines healthy. The final two public rounds passed 45/45 each through 14:07:21 UTC. Live smoke (13 checks), four canonical checks, both original expired-action exact receipt replays, both-chain offchain registration/receipt checks and runtime configuration passed. Grafana/Loki has no post-activation worker warnings; the direct Coolify RPC warning at 13:56:21 belongs to the old container before 13:58:22 activation. See final live evidence.Regression evidence:
either_chain_failure_does_not_prevent_sibling_poll,sustained_registration_arrivals_during_poison_timeout_keep_both_chains_progressing,supervised_intent_pass_bounds_sql_row_lock_and_stale_progress,process_database_session_loss_fails_readiness_then_restores_without_new_epoch,process_sigterm_during_pending_bsc_rpc_rolls_back_and_restart_retries_funded_delta,process_sigterm_during_pending_terra_lcd_keeps_bsc_progress_and_retries_funded_delta,process_sigkill_after_event_balance_and_during_commit_recovers_exactly_once_both_chains, andprocess_successful_terra_backlog_repolls_without_interval_sleep_and_keeps_sibling_and_intents_healthy. Panic/fatal-exit supervision is unit-tested; actual process restart and both-chain shutdown are separately exercised. See worker/process regressions.Observed exception: the first public round had one unexplained 503 at 14:00:58.805988 UTC (44/45), recovering by 14:01:01; it fell outside the first private capture window. That earlier round is not the final acceptance window. The subsequent two public rounds and 1080 private samples passed; this is bounded observation evidence, not a promise of permanent availability or a reason to relax fail-closed readiness.
Limits: no blockchain transaction, new contract, archive dependency, chain-identity merge or browser BSC balance RPC. Funded human/hardware/mobile wallet and Legal QA, prolonged external CPU stress and production database downgrade/destructive attacks are not claimed by the isolated fixtures or unfunded live probes. Grafana currently provides Loki only; no Prometheus datasource was available. Legacy hash-only receipts and uncertified polls retain their documented limits.
The body acceptance/completion criteria and current comment requirements are satisfied by the named deterministic regressions, final release checks and recorded live evidence. Closing is supported after the parent publishes the final artifact; no remaining mandatory implementation or verification gap is identified.
Final evidence is published through PR #60. All mandatory body/comment criteria are satisfied by the checklist above; closing this issue.