fix(operator): stop EVM writer RPC/cursor livelock (#138) #160
No reviewers
Labels
No labels
agent:implement
agent:ready
backend
bug
cannot-reproduce
confirmed
desktop
docs
documentation
duplicate
enhancement
feature
frontend
good first issue
help wanted
high-risk
in-review
invalid
mobile
needs-triage
P0-critical
P1-high
P2-medium
P3-low
qa
QA
question
ready
report
responsive
security
security-escalate
smart-contract
solana
tablet
test-pass
ux
wallet-issue
wallet:keplr
wallet:metamask
wallet:station
wallet:walletconnect
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
code/cl8y-bridge-monorepo!160
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/gl-138-evm-writer-livelock"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
eth_getLogsfallback shared by the EVM watcher and writer, so an endpoint that answerseth_blockNumberbut rate-limits logs no longer livelocks the cursor at block 0.Closes #138.
Invariants:
docs/OPERATOR_WRITER_INVARIANTS.md(INV-OP-W1–W10). Agent notes:skills/agent-operator-evm-writer-rpc.md.Test plan
poll_cursor)eth_getLogs, fallback succeeds; all-down errors; logs path does not calleth_blockNumbercargo test --binsinpackages/operator(83 passed) andmultichain-rsrpc_fallback testscargo clippy -- -D warningsfor operator (Makefile equivalent)eth_blockNumberok /eth_getLogs429, second URL serves logs; cursor reaches headRECOMMEND: FIX
Reviewed against #138. The livelock fix is the right design: method-level
eth_getLogsfallback, sticky contiguous cursor, jittered backoff, isolated writer loops, bounded negative verification retry. Fail-closed approval and contract enumeration remain. This is not a BLOCK (no event-range skip, no weakened source verification, no prompt injection).Security
verify_deposit_on_sourceEventPollCursoradvances only on chunk success; sticky first-pollfrometh_blockNumberOK /eth_getLogs429get_logs_with_endpoint_fallback; mock JSON-RPC testsWRITER_MAX_VERIFY_PER_CYCLE. Backoff rotates so the tail of the enumerable set is not permanently starved once the head is suppresseddetect_reset+ cursor reset; enumeration remains durable discoveryinvalidate_for_immediate_retryon newWithdrawSubmitnegative_retry_cache_sizeevictionpoll_configrejects out of rangeevm-<chainId>+ method)tokio::spawnper writer; 5s shutdown waitMedium: path API keys survive sanitization
sanitize_rpc_endpointstrips userinfo and query (?apiKey=) but keeps the path. Common providers put credentials in the path:https://….alchemy.com/v2/<API_KEY>https://….infura.io/v3/<PROJECT_ID>INV-OP-W9 says credentials never appear. Redact the last path segment when it looks like a key, or log
scheme://host[:port]only.Enumeration in
enumerate_and_approvestill logsrpc = %url(raw). Writer poll logs were moved to the sanitized helper; this leftover should usesanitize_rpc_endpointtoo.#138 acceptance criteria
eth_getLogsthen advance cursorpoll_cursortests)from)RUST_LOG=info; first-poll log once)Testing
Strong unit coverage: sticky lookback, partial chunks, recovery order, negative suppress/TTL/size, config bounds, mock RPC primary-429/fallback-OK.
Unchecked vs issue verification (do before calling this done in production):
Gaps
Fix the sanitizer (and enumeration log), run the Anvil/fallback checks, then this is an ACCEPT.
Follow-up (second pass): still RECOMMEND: FIX. INV-OP-W9 is still not closed; additional items:
High —
error = %estill embeds RPC URLs.packages/operator/src/rpc_fallback.rssanitizes therpcfield but logserror = %e. Hyper/reqwestDisplaycommonly includeshttps://host/v2/<key>. The all-endpointseyre!interpolates that last error too. Strip URLs from error Display (or log a classified code only).High — more raw
rpc = %urlsites in the same writer. Besides enumeration: cancel-window warn andverify_evm_deposit_on_chaininfo/warn (writers/evm.rs~184, 934, 975, 997, 1006). Watcher initprimary_rpc = %rpc_urls[0]is pre-existing but the same invariant.Medium — jitter is deterministic (INV-OP-W7). Seeds are
chain_id ^ chunk_start/ hash bytes (writers/evm.rs). Co-scheduled operators retry in lockstep. Mix in process-start entropy. Consensus-head failure seedchain_id ^ Instant::now().elapsed()on a just-created Instant is ~0.Medium — event-poll verifies ignore
WRITER_MAX_VERIFY_PER_CYCLE. Enumeration is capped; the event loop verifies everyWithdrawSubmitin the chunk and invalidates negative-retry. A gas-funded burst in one chunk is uncapped for that cycle. Count event-path verifies toward the cap (or a sibling cap).Medium — full-cache eviction picks earliest
next_retry, which re-queues those hashes as unknown next cycle. Preferinserted_atFIFO under size pressure.Should-fix: empty/wrong-chain fallback
eth_getLogssuccess advances the event cursor with no runtimeeth_chainIdcheck. Enumeration is the backstop; do not treat empty fallback logs as observed without a verified endpoint.Fail-closed
verify_deposit_on_sourceis unchanged. Failed log ranges are still not skipped. A writer-level mock livelock test (primary blockNumber OK, logs 429, fallback + cursor + single first-poll) is still missing.added 1 commit
0d58eacd- fix(operator): close INV-OP-W9 sanitizer and remaining writer review items (#138)Compare with previous version
Review follow-up (INV-OP-W9 and remaining !160 items)
Addressed both review passes on this MR. Fail-closed
verify_deposit_on_sourceis unchanged. Failed log ranges are still not skipped. Commit0d58eac.Review items completed
sanitize_rpc_endpointscheme://host[:port]only. Alchemy/v2/<key>and Infura/v3/<id>are dropped.error = %eembeds RPC URLssanitize_rpc_errorstrips URL tokens from hyper/reqwest Display. Operator fallback, writer, watcher, Terra verify, and Solana verify logs uselog_rpc/log_rpc_error. All-endpointseyre!interpolates the sanitized string.rpc = %urlin writer/watcherverify_evm_deposit_on_chain, watcherprimary_rpc, Terraevm_rpc, Solana verify.process_jitter_entropy()(pid + ASLR + start nanos) is mixed into everyjittered_exponential_backoffseed. Consensus-head failure no longer seeds fromInstant::now().elapsed()(~0).WRITER_MAX_VERIFY_PER_CYCLECycleVerifyBudgetis shared by enumeration and event poll, reset eachprocess_pendingcycle. Event path acquires a slot before invalidate/verify.next_retryinserted_at(FIFO) under size pressure.eth_getLogs(including empty) requireseth_chainIdmatch (confirm_rpc_chain_id) before the caller may treat the range as observed.writer_livelock_primary_blocknumber_ok_logs_429_fallback_advances_cursor_onceplusempty_wrong_chain_fallback_logs_do_not_succeed.Docs/invariants/skill:
docs/OPERATOR_WRITER_INVARIANTS.md(INV-OP-W3, W4, W7, W9),docs/operator.md,docs/testing.md,docs/deployment-guide.md,packages/operator/.env.example,skills/agent-operator-evm-writer-rpc.md.Tests run (twice: after implement, after clippy)
cd packages/operator && cargo test --bins— 89 passed (was 83)cd packages/multichain-rs && cargo test rpc_fallback— 19 passedcd packages/operator && cargo clippy -- -D warnings— cleanChecklist for reviewers
sanitize_rpc_endpoint("https://eth-mainnet.g.alchemy.com/v2/SECRET")==https://eth-mainnet.g.alchemy.com(path key gone)sanitize_rpc_erroron a hypererror sending request for url (https://host/v2/KEY)does not containKEYeth_getLogs429 + fallback success: cursor advances; first-poll info log only once (writer_livelock_*test)eth_chainIddoes not succeed (cursor not advanced)WithdrawSubmitburst cannot exceedWRITER_MAX_VERIFY_PER_CYCLEwhen combined with enumeration in the same cycleinserted_at, not soonestnext_retryWRITER_MAX_VERIFY_PER_CYCLE/WRITER_POLL_INTERVAL_MSin production env match documented boundssrcChain→ no approve)Not done (infra / out of scope)
No further follow-up from this pass beyond those infra soaks when a stack is available.
mentioned in commit
6b40cc5819mentioned in issue #138
mentioned in issue #140