bug(operator): prevent EVM writer RPC/cursor livelock and stale-withdrawal retry amplification #138

New issue

Closed

opened 2026-08-31 04:29:42 +00:00 by PlasticDigits · 12 comments

PlasticDigits commented 2026-08-31 04:29:42 +00:00 (Migrated from gitlab.com)

Copy link

Summary

The EVM writer can enter a persistent polling livelock when an RPC endpoint answers eth_blockNumber but rejects or rate-limits eth_getLogs. If the first lookback chunk fails, the writer keeps last_polled_block == 0, repeats the entire first-poll lookback on every writer cycle, and re-enumerates/re-verifies the same unapproved withdrawals. This creates avoidable CPU, network, RPC, and log load and provides a resource-amplification path for stale or attacker-created pending withdrawals.

This issue bundles the tightly related operator-side work: method-level RPC fallback, cursor-safe retry/backoff, bounded negative-result retry scheduling, polling configuration, observability, and log-volume reduction.

Prior context: #115 covered broader operator RPC hardening, but not this residual method-level fallback/cursor livelock.

Companion contract/state work for terminal Terra withdrawal history is tracked in #139.

---

Current codebase

Writer scheduling

• packages/operator/src/writers/mod.rs::WriterManager::run uses a hard-coded five-second interval.
• POLL_INTERVAL_MS configures other loops but does not configure this writer-manager interval.
• Writers are processed sequentially: primary EVM, Terra, then each multi-EVM writer. One degraded chain can lengthen and amplify the shared cycle.

EVM event polling and cursor behavior

• packages/operator/src/writers/evm.rs::poll_and_approve chooses one provider by trying get_block_number() against the configured RPC URLs.
• Once an endpoint answers eth_blockNumber, the writer uses that same provider for every WithdrawSubmit eth_getLogs chunk. A method-specific failure does not try the remaining RPC URLs.
• The chunk loop correctly stops on failure so it does not skip an unobserved range.
• last_polled_block advances only to last_successful_block. On an initial first-chunk failure, both remain zero.
• With last_polled_block == 0, every later cycle logs another first poll and computes a fresh current_block - EVM_POLL_LOOKBACK_BLOCKS range. A provider that consistently accepts block-number requests but rejects log queries therefore causes an indefinite retry loop.
• The EVM watcher already has get_logs_with_fallback; the writer has separate, weaker behavior.

Enumeration retry amplification

• EvmWriter::process_pending runs enumerate_and_approve() before event polling because contract enumeration is the primary safety path.
• enumerate_and_approve() calls getPendingWithdrawHashes(), then getPendingWithdraw() and source-chain verification for every unapproved hash not present in approved_hashes.
• Only successfully approved hashes are cached. A valid late deposit, stale submission, hash mismatch, unavailable source, or attacker-created unapproved withdrawal is re-queried and re-verified every cycle without per-hash backoff.
• new_to_process currently counts every still-unapproved hash attempted in that cycle; it does not mean the hash is newly discovered.

Logging and observability

• packages/operator/src/main.rs::init_logging defaults to info,cl8y_operator=debug when RUST_LOG is absent.
• Repeated first-poll, enumeration, routing, and negative-verification messages can produce high log volume while the service is making no progress.
• Existing metrics/health do not clearly expose a stalled writer cursor, per-method endpoint failures, suppressed retries, or time since successful writer progress.

---

Why a new implementation is needed

RPC endpoints commonly apply different limits to eth_blockNumber, eth_call, and eth_getLogs. Selecting an endpoint with one method is not proof that another method will work. The current behavior can consume a significant fraction of a core indefinitely, inflate RPC/network/log costs, and obscure real bridge activity.

The enumeration path also turns every unapproved destination-chain entry into recurring source-chain work. Because withdrawal submission is externally reachable, an attacker willing to pay transaction gas can create persistent work amplification. A degraded RPC plus a growing pending set can delay legitimate approvals and other chains handled by the same writer manager.

This is primarily an availability/resource-exhaustion issue. The fix must retain the bridge's fail-closed approval behavior and must never skip an event range merely to restore progress.

---

Constraints and guardrails

1. Fail closed: never approve a withdrawal unless source-chain verification succeeds against the configured chain and bridge.
2. No cursor skip: never advance beyond a failed/unobserved eth_getLogs range. Partial success may advance only through the last contiguous successful chunk.
3. Late-deposit liveness: a negative verification cache must expire and retry. It must not permanently suppress a legitimate deposit that becomes visible later.
4. Bounded state: retry/cache state must have a maximum size and TTL; attacker-controlled hashes must not create unbounded memory growth.
5. Chain isolation: one degraded EVM chain or endpoint must not stall healthy EVM, Terra, or Solana writer paths.
6. Endpoint validation: fallback must not weaken chain/bridge identity checks. Retain or strengthen the chain-ID and canonical configuration guardrails discussed in #115.
7. Reorg safety: chain reset and finality behavior must continue to retry from a safe range.
8. Restart safety: a restart must not create a tight first-poll loop or skip pending work.
9. No secret leakage: metrics and logs must not expose RPC credentials, URL query tokens, private keys, database URLs, or signed payloads.
10. Enumeration remains a safety net: do not remove contract enumeration without an equivalent durable discovery guarantee.
11. Retry discipline: use capped exponential backoff with jitter; do not synchronize many operators into a thundering herd.
12. Configuration validation: reject zero, overflow-prone, or unreasonable interval/lookback/chunk/cache values with documented bounds.

---

Relevant files

Area	Files
Writer orchestration and interval	packages/operator/src/writers/mod.rs
EVM enumeration, event polling, cursor, source verification	packages/operator/src/writers/evm.rs
Existing watcher fallback pattern	packages/operator/src/watchers/evm.rs
Shared RPC fallback/error classification	packages/operator/src/rpc_fallback.rs, packages/operator/src/writers/retry.rs, packages/multichain-rs/src/
Config/env validation	packages/operator/src/config.rs, packages/operator/.env.example
Logging defaults	packages/operator/src/main.rs
Metrics/health/liveness	packages/operator/src/metrics.rs, packages/operator/src/api.rs, packages/operator/src/liveness.rs
EVM writer tests	tests in packages/operator/src/writers/evm.rs and operator integration/E2E suites

---

Recommended direction

1. Extract a reusable EVM RPC operation helper that retries the actual operation against configured endpoints. For each eth_getLogs chunk, try the next validated endpoint on transient transport, HTTP, rate-limit, and retryable JSON-RPC errors (including provider-specific limit errors).
2. Preserve contiguous cursor semantics. Record the first failed range and retry it later; advance only through successful chunks.
3. When all endpoints fail a range, apply per-chain capped exponential backoff with jitter and expose the degraded state. Avoid repeatedly calling eth_blockNumber and rebuilding the same first-poll range at the normal cycle rate.
4. Isolate per-chain writer scheduling so a degraded chain does not sleep/block the shared manager. Keep shutdown cancellation responsive.
5. Add a bounded retry schedule for negative withdrawal verification, keyed by destination chain + hash and invalidated by relevant state changes. Suggested progression: short retry for recent entries, then progressively longer delays with a maximum delay/TTL. Newly observed event/state transitions may trigger an immediate retry.
6. Rename/count metrics accurately (attempted_unapproved, newly_discovered, negative_retry_suppressed) instead of describing every attempt as new.
7. Make writer interval and retry limits configurable with safe defaults. Parse lookback/chunk/cache settings once at startup rather than per poll.
8. Reduce steady-state logging: state transitions and summaries at info, repeated negatives/routing at sampled debug/trace, and default production logging at info. Add structured metrics for diagnosis.
9. Reuse the watcher's fallback implementation or move both watcher and writer onto one tested shared helper so behavior cannot drift again.

---

Acceptance criteria

• If the primary RPC answers eth_blockNumber but returns a retryable error for eth_getLogs, the same chunk is attempted against a configured fallback and the cursor advances after success.
• If every endpoint fails, the cursor remains at the last contiguous successful block and retries are capped/backed off with jitter rather than running every normal writer cycle.
• Recovery scans the exact failed range before later ranges; no WithdrawSubmit event is skipped.
• A failure after one or more successful chunks advances only through the final successful chunk and retries the first failed chunk.
• Repeated initial-chunk failure no longer produces an unbounded first-poll loop at the base interval.
• A degraded chain does not block healthy chain writers or shutdown handling.
• Repeated source verification failures for the same hash follow a bounded retry schedule; a later-valid deposit is eventually retried and approved.
• Retry/cache memory is bounded and expired/terminal entries are evicted.
• Enumeration still discovers withdrawals missed by event polling and remains fail closed.
• Production-default logs are summary/state-change oriented; repeated no-progress cycles do not emit per-hash/per-entry log floods.
• Metrics expose per-chain cursor progress, last successful poll, method-level RPC failures/fallbacks, backoff state, pending attempts, and suppressed retries without exposing secrets.
• Writer interval/lookback/chunk/backoff/cache configuration is documented and validated.

---

Test plan: functional and failure paths

1. Primary healthy: block number, every log chunk, enumeration, and verification succeed; cursor reaches the finalized head exactly once.
2. Method-specific fallback: primary block number succeeds, primary logs fail, fallback logs succeed; confirm same filter/range and one cursor advance.
3. Transport/HTTP/JSON-RPC matrix: timeout, connection reset, 429, 5xx, retryable JSON-RPC limit error, malformed response, and non-retryable error.
4. All endpoints down: cursor unchanged, capped jittered backoff applied, health/metrics degraded, no tight loop.
5. Partial chunks: chunk 1 succeeds and chunk 2 fails; cursor stops at chunk 1 and next poll begins at chunk 2.
6. Recovery: failed endpoint(s) recover; failed range is processed before new head ranges.
7. No new blocks: no log or enumeration storm beyond the explicitly chosen enumeration schedule.
8. First start/restart: configured lookback is scanned once; process restart remains safe and makes bounded progress.
9. Chain reset/reorg: cursor resets to the safe finalized range and cache behavior does not approve orphaned data.
10. Multiple chains: one chain is slow/failing while other EVM, Terra, and Solana paths continue on schedule.
11. Shutdown during retry: shutdown completes promptly without waiting for a full maximum backoff.
12. Negative then positive: verification returns false, is suppressed according to schedule, then returns true and is promptly approved on an eligible retry/state trigger.
13. Terminal states: approved, cancelled, and executed hashes stop consuming negative-verification work and are evicted.
14. Configuration: default, minimum, maximum, invalid, zero, and overflow-edge values for interval/lookback/chunk/backoff/cache.
15. Regression/E2E: execute representative Terra→EVM, EVM→EVM, and Solana→EVM withdrawals with enumeration and event discovery independently exercised.

---

Test plan: attack, hack, and abuse vectors

Vector	Expected result
Attacker submits many withdrawals with no matching source deposit	Per-cycle RPC work and memory remain bounded; entries are retried with capped backoff; legitimate entries are not starved
Endpoint answers cheap methods but rate-limits only eth_getLogs	Method-level fallback/backoff prevents a cursor-zero hot loop
Endpoints alternate failures to induce retry storms	Jitter/circuit state limits calls and avoids synchronized retries
Malicious/wrong-chain fallback returns plausible data	Chain identity/config validation rejects it; approval remains fail closed
RPC returns oversized/malformed/error bodies	Body/log size is bounded, parser fails safely, no panic or secret dump
Extreme env values request huge ranges or zero-delay polling	Startup validation rejects or clamps according to documented safe bounds
Reorg presents then removes a withdrawal event	Finality/cursor rules prevent approval based solely on orphaned data; enumeration/source verification remain authoritative
Hash churn attempts to exhaust negative cache	Cache has strict size/TTL eviction and metrics; no unbounded allocation
Crafted strings attempt log injection or credential disclosure	Structured/sanitized logs contain no credentials and remain parseable
Many operators restart together after outage	Jitter avoids thundering-herd retry synchronization

---

Verification criteria

• Automated tests assert exact RPC call counts/ranges and cursor values for every success/failure transition above.
• A controlled integration test uses mock RPC endpoints where the primary accepts eth_blockNumber and rejects eth_getLogs; the fallback succeeds and the writer reaches head without repeating first poll.
• A soak test with a fixed set of unverifiable hashes shows bounded RPC calls, bounded memory/cache size, low no-progress log volume, and continued processing on healthy chains.
• Metrics demonstrate increasing cursor/last-success timestamps after recovery and a stable retry rate during outage.
• Existing operator unit, integration, and cross-chain E2E suites pass, including source-verification fail-closed tests.
• Review confirms no event-range skip, no weakened source verification, no new secret-bearing logs, and no unbounded attacker-controlled state.

## Summary The EVM writer can enter a persistent polling livelock when an RPC endpoint answers `eth_blockNumber` but rejects or rate-limits `eth_getLogs`. If the first lookback chunk fails, the writer keeps `last_polled_block == 0`, repeats the entire first-poll lookback on every writer cycle, and re-enumerates/re-verifies the same unapproved withdrawals. This creates avoidable CPU, network, RPC, and log load and provides a resource-amplification path for stale or attacker-created pending withdrawals. This issue bundles the tightly related operator-side work: method-level RPC fallback, cursor-safe retry/backoff, bounded negative-result retry scheduling, polling configuration, observability, and log-volume reduction. Prior context: #115 covered broader operator RPC hardening, but not this residual method-level fallback/cursor livelock. Companion contract/state work for terminal Terra withdrawal history is tracked in #139. --- ## Current codebase ### Writer scheduling - `packages/operator/src/writers/mod.rs::WriterManager::run` uses a hard-coded five-second interval. - `POLL_INTERVAL_MS` configures other loops but does not configure this writer-manager interval. - Writers are processed sequentially: primary EVM, Terra, then each multi-EVM writer. One degraded chain can lengthen and amplify the shared cycle. ### EVM event polling and cursor behavior - `packages/operator/src/writers/evm.rs::poll_and_approve` chooses one provider by trying `get_block_number()` against the configured RPC URLs. - Once an endpoint answers `eth_blockNumber`, the writer uses that same provider for every `WithdrawSubmit` `eth_getLogs` chunk. A method-specific failure does not try the remaining RPC URLs. - The chunk loop correctly stops on failure so it does not skip an unobserved range. - `last_polled_block` advances only to `last_successful_block`. On an initial first-chunk failure, both remain zero. - With `last_polled_block == 0`, every later cycle logs another first poll and computes a fresh `current_block - EVM_POLL_LOOKBACK_BLOCKS` range. A provider that consistently accepts block-number requests but rejects log queries therefore causes an indefinite retry loop. - The EVM watcher already has `get_logs_with_fallback`; the writer has separate, weaker behavior. ### Enumeration retry amplification - `EvmWriter::process_pending` runs `enumerate_and_approve()` before event polling because contract enumeration is the primary safety path. - `enumerate_and_approve()` calls `getPendingWithdrawHashes()`, then `getPendingWithdraw()` and source-chain verification for every unapproved hash not present in `approved_hashes`. - Only successfully approved hashes are cached. A valid late deposit, stale submission, hash mismatch, unavailable source, or attacker-created unapproved withdrawal is re-queried and re-verified every cycle without per-hash backoff. - `new_to_process` currently counts every still-unapproved hash attempted in that cycle; it does not mean the hash is newly discovered. ### Logging and observability - `packages/operator/src/main.rs::init_logging` defaults to `info,cl8y_operator=debug` when `RUST_LOG` is absent. - Repeated first-poll, enumeration, routing, and negative-verification messages can produce high log volume while the service is making no progress. - Existing metrics/health do not clearly expose a stalled writer cursor, per-method endpoint failures, suppressed retries, or time since successful writer progress. --- ## Why a new implementation is needed RPC endpoints commonly apply different limits to `eth_blockNumber`, `eth_call`, and `eth_getLogs`. Selecting an endpoint with one method is not proof that another method will work. The current behavior can consume a significant fraction of a core indefinitely, inflate RPC/network/log costs, and obscure real bridge activity. The enumeration path also turns every unapproved destination-chain entry into recurring source-chain work. Because withdrawal submission is externally reachable, an attacker willing to pay transaction gas can create persistent work amplification. A degraded RPC plus a growing pending set can delay legitimate approvals and other chains handled by the same writer manager. This is primarily an availability/resource-exhaustion issue. The fix must retain the bridge's fail-closed approval behavior and must never skip an event range merely to restore progress. --- ## Constraints and guardrails 1. **Fail closed:** never approve a withdrawal unless source-chain verification succeeds against the configured chain and bridge. 2. **No cursor skip:** never advance beyond a failed/unobserved `eth_getLogs` range. Partial success may advance only through the last contiguous successful chunk. 3. **Late-deposit liveness:** a negative verification cache must expire and retry. It must not permanently suppress a legitimate deposit that becomes visible later. 4. **Bounded state:** retry/cache state must have a maximum size and TTL; attacker-controlled hashes must not create unbounded memory growth. 5. **Chain isolation:** one degraded EVM chain or endpoint must not stall healthy EVM, Terra, or Solana writer paths. 6. **Endpoint validation:** fallback must not weaken chain/bridge identity checks. Retain or strengthen the chain-ID and canonical configuration guardrails discussed in #115. 7. **Reorg safety:** chain reset and finality behavior must continue to retry from a safe range. 8. **Restart safety:** a restart must not create a tight first-poll loop or skip pending work. 9. **No secret leakage:** metrics and logs must not expose RPC credentials, URL query tokens, private keys, database URLs, or signed payloads. 10. **Enumeration remains a safety net:** do not remove contract enumeration without an equivalent durable discovery guarantee. 11. **Retry discipline:** use capped exponential backoff with jitter; do not synchronize many operators into a thundering herd. 12. **Configuration validation:** reject zero, overflow-prone, or unreasonable interval/lookback/chunk/cache values with documented bounds. --- ## Relevant files | Area | Files | |---|---| | Writer orchestration and interval | `packages/operator/src/writers/mod.rs` | | EVM enumeration, event polling, cursor, source verification | `packages/operator/src/writers/evm.rs` | | Existing watcher fallback pattern | `packages/operator/src/watchers/evm.rs` | | Shared RPC fallback/error classification | `packages/operator/src/rpc_fallback.rs`, `packages/operator/src/writers/retry.rs`, `packages/multichain-rs/src/` | | Config/env validation | `packages/operator/src/config.rs`, `packages/operator/.env.example` | | Logging defaults | `packages/operator/src/main.rs` | | Metrics/health/liveness | `packages/operator/src/metrics.rs`, `packages/operator/src/api.rs`, `packages/operator/src/liveness.rs` | | EVM writer tests | tests in `packages/operator/src/writers/evm.rs` and operator integration/E2E suites | --- ## Recommended direction 1. Extract a reusable EVM RPC operation helper that retries the **actual operation** against configured endpoints. For each `eth_getLogs` chunk, try the next validated endpoint on transient transport, HTTP, rate-limit, and retryable JSON-RPC errors (including provider-specific limit errors). 2. Preserve contiguous cursor semantics. Record the first failed range and retry it later; advance only through successful chunks. 3. When all endpoints fail a range, apply per-chain capped exponential backoff with jitter and expose the degraded state. Avoid repeatedly calling `eth_blockNumber` and rebuilding the same first-poll range at the normal cycle rate. 4. Isolate per-chain writer scheduling so a degraded chain does not sleep/block the shared manager. Keep shutdown cancellation responsive. 5. Add a bounded retry schedule for negative withdrawal verification, keyed by destination chain + hash and invalidated by relevant state changes. Suggested progression: short retry for recent entries, then progressively longer delays with a maximum delay/TTL. Newly observed event/state transitions may trigger an immediate retry. 6. Rename/count metrics accurately (`attempted_unapproved`, `newly_discovered`, `negative_retry_suppressed`) instead of describing every attempt as new. 7. Make writer interval and retry limits configurable with safe defaults. Parse lookback/chunk/cache settings once at startup rather than per poll. 8. Reduce steady-state logging: state transitions and summaries at `info`, repeated negatives/routing at sampled `debug`/`trace`, and default production logging at `info`. Add structured metrics for diagnosis. 9. Reuse the watcher's fallback implementation or move both watcher and writer onto one tested shared helper so behavior cannot drift again. --- ## Acceptance criteria - [ ] If the primary RPC answers `eth_blockNumber` but returns a retryable error for `eth_getLogs`, the same chunk is attempted against a configured fallback and the cursor advances after success. - [ ] If every endpoint fails, the cursor remains at the last contiguous successful block and retries are capped/backed off with jitter rather than running every normal writer cycle. - [ ] Recovery scans the exact failed range before later ranges; no `WithdrawSubmit` event is skipped. - [ ] A failure after one or more successful chunks advances only through the final successful chunk and retries the first failed chunk. - [ ] Repeated initial-chunk failure no longer produces an unbounded first-poll loop at the base interval. - [ ] A degraded chain does not block healthy chain writers or shutdown handling. - [ ] Repeated source verification failures for the same hash follow a bounded retry schedule; a later-valid deposit is eventually retried and approved. - [ ] Retry/cache memory is bounded and expired/terminal entries are evicted. - [ ] Enumeration still discovers withdrawals missed by event polling and remains fail closed. - [ ] Production-default logs are summary/state-change oriented; repeated no-progress cycles do not emit per-hash/per-entry log floods. - [ ] Metrics expose per-chain cursor progress, last successful poll, method-level RPC failures/fallbacks, backoff state, pending attempts, and suppressed retries without exposing secrets. - [ ] Writer interval/lookback/chunk/backoff/cache configuration is documented and validated. --- ## Test plan: functional and failure paths 1. **Primary healthy:** block number, every log chunk, enumeration, and verification succeed; cursor reaches the finalized head exactly once. 2. **Method-specific fallback:** primary block number succeeds, primary logs fail, fallback logs succeed; confirm same filter/range and one cursor advance. 3. **Transport/HTTP/JSON-RPC matrix:** timeout, connection reset, 429, 5xx, retryable JSON-RPC limit error, malformed response, and non-retryable error. 4. **All endpoints down:** cursor unchanged, capped jittered backoff applied, health/metrics degraded, no tight loop. 5. **Partial chunks:** chunk 1 succeeds and chunk 2 fails; cursor stops at chunk 1 and next poll begins at chunk 2. 6. **Recovery:** failed endpoint(s) recover; failed range is processed before new head ranges. 7. **No new blocks:** no log or enumeration storm beyond the explicitly chosen enumeration schedule. 8. **First start/restart:** configured lookback is scanned once; process restart remains safe and makes bounded progress. 9. **Chain reset/reorg:** cursor resets to the safe finalized range and cache behavior does not approve orphaned data. 10. **Multiple chains:** one chain is slow/failing while other EVM, Terra, and Solana paths continue on schedule. 11. **Shutdown during retry:** shutdown completes promptly without waiting for a full maximum backoff. 12. **Negative then positive:** verification returns false, is suppressed according to schedule, then returns true and is promptly approved on an eligible retry/state trigger. 13. **Terminal states:** approved, cancelled, and executed hashes stop consuming negative-verification work and are evicted. 14. **Configuration:** default, minimum, maximum, invalid, zero, and overflow-edge values for interval/lookback/chunk/backoff/cache. 15. **Regression/E2E:** execute representative Terra→EVM, EVM→EVM, and Solana→EVM withdrawals with enumeration and event discovery independently exercised. --- ## Test plan: attack, hack, and abuse vectors | Vector | Expected result | |---|---| | Attacker submits many withdrawals with no matching source deposit | Per-cycle RPC work and memory remain bounded; entries are retried with capped backoff; legitimate entries are not starved | | Endpoint answers cheap methods but rate-limits only `eth_getLogs` | Method-level fallback/backoff prevents a cursor-zero hot loop | | Endpoints alternate failures to induce retry storms | Jitter/circuit state limits calls and avoids synchronized retries | | Malicious/wrong-chain fallback returns plausible data | Chain identity/config validation rejects it; approval remains fail closed | | RPC returns oversized/malformed/error bodies | Body/log size is bounded, parser fails safely, no panic or secret dump | | Extreme env values request huge ranges or zero-delay polling | Startup validation rejects or clamps according to documented safe bounds | | Reorg presents then removes a withdrawal event | Finality/cursor rules prevent approval based solely on orphaned data; enumeration/source verification remain authoritative | | Hash churn attempts to exhaust negative cache | Cache has strict size/TTL eviction and metrics; no unbounded allocation | | Crafted strings attempt log injection or credential disclosure | Structured/sanitized logs contain no credentials and remain parseable | | Many operators restart together after outage | Jitter avoids thundering-herd retry synchronization | --- ## Verification criteria - Automated tests assert exact RPC call counts/ranges and cursor values for every success/failure transition above. - A controlled integration test uses mock RPC endpoints where the primary accepts `eth_blockNumber` and rejects `eth_getLogs`; the fallback succeeds and the writer reaches head without repeating first poll. - A soak test with a fixed set of unverifiable hashes shows bounded RPC calls, bounded memory/cache size, low no-progress log volume, and continued processing on healthy chains. - Metrics demonstrate increasing cursor/last-success timestamps after recovery and a stable retry rate during outage. - Existing operator unit, integration, and cross-chain E2E suites pass, including source-verification fail-closed tests. - Review confirms no event-range skip, no weakened source verification, no new secret-bearing logs, and no unbounded attacker-controlled state.

PlasticDigits commented 2026-08-31 04:30:17 +00:00 (Migrated from gitlab.com)

Copy link

changed the description

changed the description

PlasticDigits commented 2026-08-31 04:30:20 +00:00 (Migrated from gitlab.com)

Copy link

mentioned in issue #139

mentioned in issue #139

PlasticDigits commented 2026-08-31 04:30:45 +00:00 (Migrated from gitlab.com)

Copy link

marked as related to #139

marked as related to #139

PlasticDigits commented 2026-08-31 05:57:49 +00:00 (Migrated from gitlab.com)

Copy link

mentioned in merge request !158

mentioned in merge request !158

PlasticDigits commented 2026-08-31 06:02:49 +00:00 (Migrated from gitlab.com)

Copy link

mentioned in commit fd4edddbd9

mentioned in commit fd4edddbd997600f9bf322552a1a7ea07b23f9f1

PlasticDigits commented 2026-08-31 06:02:52 +00:00 (Migrated from gitlab.com)

Copy link

mentioned in merge request !160

mentioned in merge request !160

PlasticDigits commented 2026-08-31 11:24:18 +00:00 (Migrated from gitlab.com)

Copy link

mentioned in commit 0d58eacd72

mentioned in commit 0d58eacd72b04517cec9907deb675c3b6fc02a6b

PlasticDigits commented 2026-08-31 12:11:24 +00:00 (Migrated from gitlab.com)

Copy link

mentioned in commit 6b40cc5819

mentioned in commit 6b40cc5819cfd0b0d147c4f524d2fd309ee6dc1d

PlasticDigits (Migrated from gitlab.com) closed this issue 2026-08-31 12:11:24 +00:00

PlasticDigits commented 2026-08-31 12:39:10 +00:00 (Migrated from gitlab.com)

Copy link

Merge status (!160 landed on main; this issue auto-closed on Closes)

Code AC is met: method-level eth_getLogs fallback, sticky cursor, jittered backoff, isolated writer loops, bounded negative retry, INV-OP-W9 sanitizer (scheme://host + sanitize_rpc_error).

Remaining infra soaks (not in this issue after close)

Tracked in a new follow-up issue (linked): operator restart vs Anvil, real-provider fallback soak, regression E2E Terra→EVM / EVM→EVM / Solana→EVM.

Unapproved on-chain set growth remains #139 / contract-side, not retry amplification.

## Merge status (!160 landed on `main`; this issue auto-closed on Closes) Code AC is met: method-level `eth_getLogs` fallback, sticky cursor, jittered backoff, isolated writer loops, bounded negative retry, INV-OP-W9 sanitizer (`scheme://host` + `sanitize_rpc_error`). ### Remaining infra soaks (not in this issue after close) Tracked in a new follow-up issue (linked): operator restart vs Anvil, real-provider fallback soak, regression E2E Terra→EVM / EVM→EVM / Solana→EVM. Unapproved on-chain set growth remains #139 / contract-side, not retry amplification.

PlasticDigits commented 2026-08-31 12:39:32 +00:00 (Migrated from gitlab.com)

Copy link

mentioned in issue #140

mentioned in issue #140

PlasticDigits commented 2026-08-31 12:39:32 +00:00 (Migrated from gitlab.com)

Copy link

marked as related to #140

marked as related to #140

PlasticDigits commented 2026-09-01 07:35:05 +00:00 (Migrated from gitlab.com)

Copy link

mentioned in merge request !163

mentioned in merge request !163

PlasticDigits referenced this issue 2026-09-04 04:03:48 +00:00

bug(operator): MegaETH/LCD rate-limit gaps stall getDeposit and Terra writer approval #164

PlasticDigits referenced this issue 2026-09-11 06:21:18 +00:00

fix(operator): dest-approved Terra→EVM withdrawals stay unexecuted #170

PlasticDigits referenced this issue 2026-09-12 12:38:22 +00:00

security(operator): Terra watcher must wait for confirmation depth #182

PlasticDigits referenced this issue 2026-09-12 12:42:39 +00:00

security(operator): invalidate reorged EVM deposits before dest approve #183

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

issue/165

issue/187

issue/170

issue/195

renovate/configure

review/brouie-solana-e2e-70

Labels

Clear labels   

agent:implement

Queue implement gate

  

agent:ready

  

backend

  

bug

  

cannot-reproduce

  

confirmed

  

desktop

  

docs

  

documentation

  

duplicate

  

enhancement

  

feature

  

frontend

  

good first issue

  

help wanted

  

high-risk

  

in-review

  

invalid

  

mobile

  

needs-triage

  

P0-critical

  

P1-high

  

P2-medium

  

P3-low

  

qa

  

QA

  

question

  

ready

Definition of Ready

  

report

  

responsive

  

security

  

security-escalate

  

smart-contract

  

solana

  

tablet

  

test-pass

  

ux

  

wallet-issue

  

wallet:keplr

  

wallet:metamask

  

wallet:station

  

wallet:walletconnect

  

wontfix

No labels

agent:implement

agent:ready

backend

bug

cannot-reproduce

confirmed

desktop

docs

documentation

duplicate

enhancement

feature

frontend

good first issue

help wanted

high-risk

in-review

invalid

mobile

needs-triage

P0-critical

P1-high

P2-medium

P3-low

qa

QA

question

ready

report

responsive

security

security-escalate

smart-contract

solana

tablet

test-pass

ux

wallet-issue

wallet:keplr

wallet:metamask

wallet:station

wallet:walletconnect

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

code/cl8y-bridge-monorepo#138

No description provided.