Indexer resilience: ingestion task fails silently + RPC calls have no timeouts (launch-day risk)

Brouie commented

2026-05-06 05:53:20 +00:00

(Migrated from gitlab.com)

two tightly-coupled indexer resilience gaps surfaced during a code review pass against 390f9be. both are launch-day failure modes that compound — RPC hang triggers ingestion silent exit, ingestion silent exit means HTTP API serves stale data forever, no path to recover without process restart.

Gap 1 — ingestion task fail-silent in `main.rs:31-35`

spawned task swallows errors with just tracing::error! and exits cleanly:

let ingestion_handle = tokio::spawn(async move {
    if let Err(e) = ingestion::run(&ingest_pool, &ingest_config).await {
        tracing::error!(?e, "ingestion failed");
    }
});

after exit, the HTTP API keeps responding 200 OK to all /v1/* reads, just from frozen state. ingestion_handle is not awaited or supervised. /v1/status exposes max_indexed_block but no client reads it as a liveness check, and there is no ingestion_alive: bool or last_indexed_at_ms to alert against.

failure modes that trip this:

transient RPC error during get_block_by_number (any ? bubble)
decoder bug on a new event variant
DB connection blip
worker panic inside ingestion::run that escapes the spawn closure

risk profile: sale ends, indexer dies mid-finalization, frontend shows stale podiums, nobody notices for hours.

fix shape: either (a) wrap ingestion::run in a backoff-retry loop that re-enters on Err, or (b) propagate the Err to main and let the process supervisor restart the binary. additionally surface ingestion_alive: bool and last_indexed_at_ms: u64 in /v1/status so smoke tests can alert on staleness.

Gap 2 — RPC calls have no per-call or transport-level timeout

chain_timer.rs:138 and ingestion.rs:92 both build providers without timeouts:

let provider = ProviderBuilder::new().on_http(url);

no .timeout(...) on the builder, no tokio::time::timeout(...) wrapper on any of the downstream calls. default reqwest behavior is no timeout. chain_timer::poll_once does 9 sequential eth_calls per tick (saleStart, deadline, timerCapSec, ended, podium(0..3)); a single hung call freezes the snapshot at the previous value. ingestion get_block_by_number(next) with no timeout blocks the entire indexer.

fix shape: wrap each RPC await in tokio::time::timeout(Duration::from_secs(5), ...) (or set a transport-level timeout via reqwest::Client::builder().timeout(...) and pipe through ClientBuilder::on_provider). also: poll_once 9 sequential calls would benefit from a multicall or parallel try_join!, but the timeout fix is the priority.

Why these are coupled

gap 2 alone gives a stalled-but-alive ingestion task — observable through metrics if you look. gap 1 alone gives a clean failure path that should never trigger. together: RPC hangs → first call eventually times out at the OS level (minutes/hours), ingestion ? bubbles → fail-silent log → indexer is now a stale-data zombie with no signal to operators.

closing either alone halves the risk. closing both makes the indexer self-recovering through normal ops.

Priority

gap 1 is the single highest-impact launch-day fix — operator visibility into "is ingestion actually running" goes from zero to definitive. gap 2 is one-day defense-in-depth that mitigates the most common cause of gap 1 firing.

Out of scope

this is distinct from #156 (production config fail-closed), #157 (raw DB error redaction), and the existing #138 children — those covered config + error-message hygiene, not runtime resilience of the ingestion task itself.

cc @PlasticDigits

two tightly-coupled indexer resilience gaps surfaced during a code review pass against `390f9be`. both are launch-day failure modes that compound — RPC hang triggers ingestion silent exit, ingestion silent exit means HTTP API serves stale data forever, no path to recover without process restart. ## Gap 1 — ingestion task fail-silent in `main.rs:31-35` spawned task swallows errors with just `tracing::error!` and exits cleanly: ``` let ingestion_handle = tokio::spawn(async move { if let Err(e) = ingestion::run(&ingest_pool, &ingest_config).await { tracing::error!(?e, "ingestion failed"); } }); ``` after exit, the HTTP API keeps responding 200 OK to all `/v1/*` reads, just from frozen state. `ingestion_handle` is not awaited or supervised. `/v1/status` exposes `max_indexed_block` but no client reads it as a liveness check, and there is no `ingestion_alive: bool` or `last_indexed_at_ms` to alert against. failure modes that trip this: - transient RPC error during `get_block_by_number` (any `?` bubble) - decoder bug on a new event variant - DB connection blip - worker panic inside ingestion::run that escapes the spawn closure risk profile: sale ends, indexer dies mid-finalization, frontend shows stale podiums, nobody notices for hours. fix shape: either (a) wrap ingestion::run in a backoff-retry loop that re-enters on Err, or (b) propagate the Err to main and let the process supervisor restart the binary. additionally surface `ingestion_alive: bool` and `last_indexed_at_ms: u64` in `/v1/status` so smoke tests can alert on staleness. ## Gap 2 — RPC calls have no per-call or transport-level timeout `chain_timer.rs:138` and `ingestion.rs:92` both build providers without timeouts: ``` let provider = ProviderBuilder::new().on_http(url); ``` no `.timeout(...)` on the builder, no `tokio::time::timeout(...)` wrapper on any of the downstream calls. default reqwest behavior is no timeout. `chain_timer::poll_once` does 9 sequential `eth_call`s per tick (saleStart, deadline, timerCapSec, ended, podium(0..3)); a single hung call freezes the snapshot at the previous value. ingestion `get_block_by_number(next)` with no timeout blocks the entire indexer. fix shape: wrap each RPC await in `tokio::time::timeout(Duration::from_secs(5), ...)` (or set a transport-level timeout via `reqwest::Client::builder().timeout(...)` and pipe through `ClientBuilder::on_provider`). also: `poll_once` 9 sequential calls would benefit from a multicall or parallel `try_join!`, but the timeout fix is the priority. ## Why these are coupled gap 2 alone gives a stalled-but-alive ingestion task — observable through metrics if you look. gap 1 alone gives a clean failure path that should never trigger. together: RPC hangs → first call eventually times out at the OS level (minutes/hours), ingestion `?` bubbles → fail-silent log → indexer is now a stale-data zombie with no signal to operators. closing either alone halves the risk. closing both makes the indexer self-recovering through normal ops. ## Priority gap 1 is the single highest-impact launch-day fix — operator visibility into "is ingestion actually running" goes from zero to definitive. gap 2 is one-day defense-in-depth that mitigates the most common cause of gap 1 firing. ## Out of scope this is distinct from #156 (production config fail-closed), #157 (raw DB error redaction), and the existing #138 children — those covered config + error-message hygiene, not runtime resilience of the ingestion task itself. cc @PlasticDigits

PlasticDigits commented

2026-05-06 09:34:21 +00:00

(Migrated from gitlab.com)

mentioned in commit c90bca7149

mentioned in commit c90bca714927e987a7247c7ecbffbf41fcaffe61

PlasticDigits commented

2026-05-06 09:34:21 +00:00

(Migrated from gitlab.com)

mentioned in commit 74967bc5fd

mentioned in commit 74967bc5fd323cf2cc8ffdb3f7d7d2db6ea768ce

PlasticDigits commented

2026-05-06 09:34:40 +00:00

(Migrated from gitlab.com)

Implementation merged (main @ `74967bc`)

What changed

Ingestion supervision: ingestion::run is wrapped in a loop in main.rs with exponential backoff (1s base, 60s cap) on fatal errors so the HTTP API does not keep serving frozen Postgres after a background task exit.
JSON-RPC HTTP timeouts: reqwest::Client is built with INDEXER_RPC_REQUEST_TIMEOUT_SEC (default 5, clamped 1–120) via rpc_http::reqwest_http_provider for ingestion and the chain-timer poller (chain_timer.rs).
Operator visibility: GET /v1/status (schema ≥ 1.16.1) adds ingestion_alive and last_indexed_at_ms (0 if no block committed yet). ingestion_alive is false when ingestion is API-only idle (INGESTION_ENABLED off or no registry addresses).

Docs / invariants

INV-INDEXER-168 in docs/testing/invariants-and-business-logic.md, indexer README, docs/indexer/design.md, skills/README.md, guardrails skill.

Verify (checklist)

cd indexer && cargo clippy --all-targets -- -D warnings && cargo test passes (ran locally before push).
With a real stack: GET /v1/status returns ingestion_alive: true and a recent last_indexed_at_ms while blocks are ingesting; false + 0 in intentional API-only mode.
Induce a transient RPC failure: logs show backoff + retry; /v1/* reads still work but max_indexed_block should stall until recovery (expected).
Optional: set INDEXER_RPC_REQUEST_TIMEOUT_SEC=2 and confirm hung endpoints fail within ~2s in logs (defense-in-depth).

Please verify on your side when convenient.

cc @brouie

## Implementation merged (main @ 74967bc) **What changed** - **Ingestion supervision:** `ingestion::run` is wrapped in a loop in `main.rs` with exponential backoff (1s base, 60s cap) on fatal errors so the HTTP API does not keep serving frozen Postgres after a background task exit. - **JSON-RPC HTTP timeouts:** `reqwest::Client` is built with `INDEXER_RPC_REQUEST_TIMEOUT_SEC` (default **5**, clamped **1–120**) via `rpc_http::reqwest_http_provider` for **ingestion** and the **chain-timer** poller (`chain_timer.rs`). - **Operator visibility:** `GET /v1/status` (schema **≥ 1.16.1**) adds `ingestion_alive` and `last_indexed_at_ms` (0 if no block committed yet). `ingestion_alive` is **false** when ingestion is API-only idle (`INGESTION_ENABLED` off or no registry addresses). **Docs / invariants** - `INV-INDEXER-168` in `docs/testing/invariants-and-business-logic.md`, indexer README, `docs/indexer/design.md`, `skills/README.md`, guardrails skill. **Verify (checklist)** - [ ] `cd indexer && cargo clippy --all-targets -- -D warnings && cargo test` passes (ran locally before push). - [ ] With a real stack: `GET /v1/status` returns `ingestion_alive: true` and a recent `last_indexed_at_ms` while blocks are ingesting; `false` + `0` in intentional API-only mode. - [ ] Induce a transient RPC failure: logs show backoff + retry; `/v1/*` reads still work but `max_indexed_block` should stall until recovery (expected). - [ ] Optional: set `INDEXER_RPC_REQUEST_TIMEOUT_SEC=2` and confirm hung endpoints fail within ~2s in logs (defense-in-depth). Please verify on your side when convenient. cc @brouie

Brouie commented

2026-05-07 02:15:26 +00:00

(Migrated from gitlab.com)

@PlasticDigits — verification PASS on 74967bc / c90bca7.

supervisor loop at indexer/src/main.rs:38-54 — base 1s backoff, 60s cap, doubled per Err return, reset on Ok(()). log lines at :44 ("ingestion::run returned Ok unexpectedly; resetting backoff") and :48 ("ingestion failed; backing off before retry"). all three failure branches in ingestion::run (INGESTION_ENABLED=false, empty addrs, inner-loop ?) flip ingestion_alive to false at ingestion.rs:97/113/232 before returning. atomic stays false during backoff sleep, flips true again at :132 after bootstrap_pointer succeeds.

INDEXER_RPC_REQUEST_TIMEOUT_SEC at config.rs:344-361 — default 5s (DEFAULT_RPC_REQUEST_TIMEOUT_SECS = 5), clamp 1..=120s, lower bound bail! on 0, upper bound min(120) (silent — see follow-up below). wired through rpc_http::reqwest_http_provider at both ingestion.rs:125 and chain_timer.rs:141.

/v1/status body at api.rs:334-344 — ingestion_alive boolean from state.ingestion_alive.load(Acquire), last_indexed_at_ms u64 from IngestionProgress::mark_indexed_now() at ingestion.rs:213-215 (set after tx.commit() per block, initial 0 until first commit).

integration test smoke at integration_stage2.rs:130-131 covers ingestion_alive: false + last_indexed_at_ms: 0 for the harness state. retry/backoff loop itself not exercised in tests — small enough surface to review by inspection, fine.

ready for close on your side.

two follow-up observations from the same review pass — flagging here so you can fold them into a follow-up commit, spawn separate tickets, or accept as out-of-scope, your call:

chain-timer poll task at main.rs:67-69 is unsupervised — tokio::spawn(async move { chain_timer::run_poll_loop(...).await }) with discarded JoinHandle. if run_poll_loop returns or panics, /v1/timecurve/chain-timer silently degrades to 503 with no liveness signal. the ingestion supervisor at :38-54 does not cover it. partially in #168 territory since the original ticket cited "ingestion + chain-timer poller" timeouts together.
INDEXER_RPC_REQUEST_TIMEOUT_SEC upper-bound clamp at config.rs:359 silently truncates values >120 to 120. lower bound bail!s at 0. asymmetric — a tracing::warn! on truncate or symmetric bail! would give operators a heads-up. minor / style.

no panic-safety concern in ingestion::run body (no unwrap/expect/array-index paths surfaced), so the supervisor match covering only Result arms is fine in practice today. worth keeping in mind if ingestion::run ever grows future code.

@PlasticDigits — verification PASS on `74967bc` / `c90bca7`. supervisor loop at `indexer/src/main.rs:38-54` — base 1s backoff, 60s cap, doubled per `Err` return, reset on `Ok(())`. log lines at `:44` ("ingestion::run returned Ok unexpectedly; resetting backoff") and `:48` ("ingestion failed; backing off before retry"). all three failure branches in `ingestion::run` (`INGESTION_ENABLED=false`, empty addrs, inner-loop `?`) flip `ingestion_alive` to false at `ingestion.rs:97/113/232` before returning. atomic stays false during backoff sleep, flips true again at `:132` after `bootstrap_pointer` succeeds. `INDEXER_RPC_REQUEST_TIMEOUT_SEC` at `config.rs:344-361` — default 5s (`DEFAULT_RPC_REQUEST_TIMEOUT_SECS = 5`), clamp `1..=120s`, lower bound `bail!` on 0, upper bound `min(120)` (silent — see follow-up below). wired through `rpc_http::reqwest_http_provider` at both `ingestion.rs:125` and `chain_timer.rs:141`. `/v1/status` body at `api.rs:334-344` — `ingestion_alive` boolean from `state.ingestion_alive.load(Acquire)`, `last_indexed_at_ms` u64 from `IngestionProgress::mark_indexed_now()` at `ingestion.rs:213-215` (set after `tx.commit()` per block, initial 0 until first commit). integration test smoke at `integration_stage2.rs:130-131` covers `ingestion_alive: false` + `last_indexed_at_ms: 0` for the harness state. retry/backoff loop itself not exercised in tests — small enough surface to review by inspection, fine. ready for close on your side. two follow-up observations from the same review pass — flagging here so you can fold them into a follow-up commit, spawn separate tickets, or accept as out-of-scope, your call: 1. chain-timer poll task at `main.rs:67-69` is unsupervised — `tokio::spawn(async move { chain_timer::run_poll_loop(...).await })` with discarded `JoinHandle`. if `run_poll_loop` returns or panics, `/v1/timecurve/chain-timer` silently degrades to 503 with no liveness signal. the ingestion supervisor at `:38-54` does not cover it. partially in #168 territory since the original ticket cited "ingestion + chain-timer poller" timeouts together. 2. `INDEXER_RPC_REQUEST_TIMEOUT_SEC` upper-bound clamp at `config.rs:359` silently truncates values >120 to 120. lower bound `bail!`s at 0. asymmetric — a `tracing::warn!` on truncate or symmetric `bail!` would give operators a heads-up. minor / style. no panic-safety concern in `ingestion::run` body (no `unwrap`/`expect`/`array-index` paths surfaced), so the supervisor `match` covering only `Result` arms is fine in practice today. worth keeping in mind if `ingestion::run` ever grows future code.

Brouie commented

2026-05-07 02:30:30 +00:00

(Migrated from gitlab.com)

@PlasticDigits — runtime checklist walked, all PASS.

cargo clippy --all-targets -- -D warnings && cargo test — clippy clean (0 warnings), tests 40 unit + 2 integration = 42/42 PASS at HEAD 284c15c.
live /v1/status against the anvil-backed stack:

{
  "chain_pointer": { "block_hash": "0x68d548...8d642", "block_number": 7839 },
  "database_connected": true,
  "ingestion_alive": true,
  "last_indexed_at_ms": 1778120767105,
  "max_indexed_block": 7839,
  "schema_version": "1.16.1"
}

ingestion_alive: true flips on after bootstrap_pointer succeeds. last_indexed_at_ms advances per block. max_indexed_block keeps pace with anvil tip.

transient RPC failure induced by spawning a second indexer instance pointed at RPC_URL=http://127.0.0.1:9999 (separate test DB + LISTEN_ADDR=127.0.0.1:3199). log shows backoff/retry working as designed:

T+0.00: ingestion started -> fail (connection refused)
T+2.06: retry -> fail            (1s backoff)
T+4.06: retry -> fail            (2s backoff, doubled)

/v1/* reads stay responsive throughout — GET /v1/status on the failing instance returned ingestion_alive: false, last_indexed_at_ms: 0, database_connected: true, schema 1.16.1. operator visibility into "API up, ingestion dead" works exactly as intended.

optional 2s-timeout induction skipped — INDEXER_RPC_REQUEST_TIMEOUT_SEC defense path is for hung endpoints (accepts connection then never responds), not the connection-refused case I induced. covered by inspection at config.rs:344-361 + wiring at ingestion.rs:125/chain_timer.rs:141. happy to set up a hung-RPC fixture and walk this if you want it for completeness, otherwise treating as out-of-scope per your "optional" tag.

ready for close on your side.

@PlasticDigits — runtime checklist walked, all PASS. 1. `cargo clippy --all-targets -- -D warnings && cargo test` — clippy clean (0 warnings), tests 40 unit + 2 integration = 42/42 PASS at HEAD `284c15c`. 2. live `/v1/status` against the anvil-backed stack: ``` { "chain_pointer": { "block_hash": "0x68d548...8d642", "block_number": 7839 }, "database_connected": true, "ingestion_alive": true, "last_indexed_at_ms": 1778120767105, "max_indexed_block": 7839, "schema_version": "1.16.1" } ``` `ingestion_alive: true` flips on after `bootstrap_pointer` succeeds. `last_indexed_at_ms` advances per block. `max_indexed_block` keeps pace with anvil tip. 3. transient RPC failure induced by spawning a second indexer instance pointed at `RPC_URL=http://127.0.0.1:9999` (separate test DB + `LISTEN_ADDR=127.0.0.1:3199`). log shows backoff/retry working as designed: ``` T+0.00: ingestion started -> fail (connection refused) T+2.06: retry -> fail (1s backoff) T+4.06: retry -> fail (2s backoff, doubled) ``` `/v1/*` reads stay responsive throughout — `GET /v1/status` on the failing instance returned `ingestion_alive: false`, `last_indexed_at_ms: 0`, `database_connected: true`, schema `1.16.1`. operator visibility into "API up, ingestion dead" works exactly as intended. 4. optional 2s-timeout induction skipped — `INDEXER_RPC_REQUEST_TIMEOUT_SEC` defense path is for hung endpoints (accepts connection then never responds), not the connection-refused case I induced. covered by inspection at `config.rs:344-361` + wiring at `ingestion.rs:125`/`chain_timer.rs:141`. happy to set up a hung-RPC fixture and walk this if you want it for completeness, otherwise treating as out-of-scope per your "optional" tag. ready for close on your side.

Brouie commented

2026-05-07 02:48:10 +00:00

(Migrated from gitlab.com)