code/cl8y-dex-terraclassic

Fork 0

Phase 1b: book_snapshot loop — mirror on-chain reserves + resting book into Postgres (freshness contract) #322

New issue

Closed

opened 2026-06-05 08:23:00 +00:00 by Brouie · 12 comments

Brouie commented

2026-06-05 08:23:00 +00:00

(Migrated from gitlab.com)

Summary

Phase 1a (#279, MR !761) landed the mirror schema: the pair_reserves and resting_limit_orders tables, plus the query layer that reads and writes them. Nothing populates those tables yet — they're empty in production.

Phase 1b is the writer: a background loop, modeled on the existing run_oracle_loop / run_tier_sync_loop pattern, that periodically queries each pair's on-chain pool reserves + fee and its resting limit book, and writes them into the Phase 1a tables (upsert_pair_reserves + replace_pair_resting_orders). It also defines and records a freshness contract (snapshot interval + max staleness via snapshot_at / block_height) so Phase 1c can degrade-not-error when the mirror is stale or missing.

This is the hard dependency that #319 (Phase 1c) blocks on: the 0-LCD solver can't read a mirror that nothing fills.

Current codebase

Phase 1a schema + query layer (done, MR !761):

indexer/migrations/20260605010000_pair_reserves.sql — pair_reserves table: PK pair_id, reserve_0/reserve_1 NUMERIC(38,0), fee_bps SMALLINT, nullable block_height BIGINT, snapshot_at TIMESTAMPTZ DEFAULT NOW().
indexer/migrations/20260605010100_resting_limit_orders.sql — resting_limit_orders: PK (pair_id, order_id), side CHECK in ('bid','ask'), price NUMERIC(38,18), remaining NUMERIC(38,0), nullable owner/expires_at/block_height, snapshot_at. Walk index idx_resting_orders_book (pair_id, side, price, order_id).
indexer/src/db/queries/pair_reserves.rs — upsert_pair_reserves(pool, pair_id, reserve_0, reserve_1, fee_bps, block_height) does an INSERT … ON CONFLICT (pair_id) DO UPDATE … snapshot_at = NOW(); get_pair_reserves(pool, pair_id) -> Option<PairReservesRow> (missing snapshot = None, degrade-not-error).
indexer/src/db/queries/resting_orders.rs — replace_pair_resting_orders(pool, pair_id, block_height, &[RestingOrderInput]) deletes the pair's rows and re-inserts the full book in one transaction; get_pair_resting_book(pool, pair_id, side) returns walk order (bids DESC / asks ASC, then FIFO by order_id). RestingOrderInput { order_id, side, price, remaining, owner, expires_at }.
Tests: indexer/tests/db_orderbook_mirror.rs.

Loop / poller precedents to mirror:

indexer/src/indexer/oracle.rs — run_oracle_loop(pool, poll_interval_ms, latest_price) (line 18): builds a client, optional warm-load from DB, then loop { … fetch … write … tokio::time::sleep(interval) }. Interval comes from config (oracle_poll_interval_ms), failures are logged and the last-good state is retained, never propagated as a hard error out of the loop.
indexer/src/indexer/trader_tracker.rs — run_tier_sync_loop(pool, lcd, fee_discount_addr) (line 20): loop { sleep; if let Err(e) = sync_tiers(...).await { tracing::error!(...) } }. sync_tiers (line 39) pulls a row set, then per-row queries an LCD contract (query_contract::<serde_json::Value>) and upserts; a per-row LCD failure is tracing::warn!-ed and skipped, not fatal.
indexer/src/indexer/poller.rs — run_indexer registers each loop via tokio::spawn with cloned pool / lcd / config fields (lines 30-46). This is where the new loop gets wired in.
indexer/src/indexer/mod.rs — submodules are declared here (pub mod oracle; etc.); a new pub mod book_snapshot; goes alongside.

On-chain query shapes the loop reuses (already exercised elsewhere):

Reserves + fee: lcd.query_contract::<PoolResponse>(pair_addr, json!({"pool": {}})) — PoolResponse { assets: [Asset; 2], total_share }, Asset { info, amount } (indexer/src/lcd/types.rs:99-109; live use at indexer/src/api/orderbook_sim.rs:255-256). Fee via json!({"get_fee_config": {}}) -> FeeConfigResponse { fee_config: { fee_bps, treasury } } (indexer/src/lcd/types.rs:111-120; live use in indexer/src/indexer/pair_discovery.rs:108-115). fee_bps casts u16 -> i16 exactly as sync_single_pair already does.
Resting book: walked via json!({"order_book_head": {"side": side_label}}) to get the head order id, then linked limit_order lookups per order — see fetch_limit_book_page in indexer/src/api/limit_book_lcd.rs:108-183 (head at line 129-133, per-order at 159-172). The loop reuses this walk to produce the full per-side book, mapping each order into RestingOrderInput.
Pair set to iterate: get_all_pairs(pool) -> Vec<PairRow> (indexer/src/db/queries/pairs.rs:199).
LcdClient is #[derive(Clone)] (indexer/src/lcd/mod.rs:31) and exposes get_latest_block_height() (indexer/src/lcd/mod.rs:160) for stamping block_height.

Solver side (downstream consumer, for context only — not touched here):

indexer/src/api/route_solver.rs — GET_DEFAULT_MAX_HOPS = 3 (line 31), GET_POOL_ONLY_MAX_HOPS = 4 (line 33).
indexer/src/api/best_execution.rs — solve_global_best_execution over MAX_PATH_CANDIDATES = 5 (line 18), LCD_HYBRID_SIM_BUDGET (line 26). These still hit LCD today; Phase 1c rewires them onto the mirror this loop fills.

Why this is needed

The whole 0-LCD hybrid solver program (#279) rests on the solver reading pool reserves and the resting book from Postgres instead of issuing per-request LCD calls. Phase 1a gave us the tables and accessors; without a writer they stay empty and Phase 1c (#319) has nothing to read. The LCD cost doesn't vanish — it moves out of the hot request path into one bounded background loop that amortizes it across all requests.

The freshness contract is the other half. The mirror is eventually-consistent by construction (snapshot cadence, not block-exact). Phase 1c's degrade-not-error semantics (get_pair_reserves returning None, a stale snapshot_at) only mean something if Phase 1b defines what "stale" is and stamps every row with the data needed to evaluate it.

Constraints and guardrails

The loop is the one allowed LCD user. It IS the mirror writer, so per-pair pool / get_fee_config / book-walk LCD calls are expected and fine. What must stay bounded is total LCD per snapshot cycle — see acceptance below. This is the inverse of the solver-path budgets (LCD_HYBRID_SIM_BUDGET etc.), which Phase 1c drives toward zero.
Atomic per-pair writes only. Use replace_pair_resting_orders (single transaction, delete-then-insert) so a pair's book is never observed half-updated. upsert_pair_reserves is already a single statement.
A per-pair failure must not poison the cycle. Follow the sync_tiers shape: a failed pool/fee/book query for one pair is tracing::warn!-ed and skipped; the loop keeps the last good snapshot for that pair (upsert not run, rows not replaced) and moves on. A missing/failed snapshot for a pair leaves Phase 1c to degrade on it, not error.
Loop errors never propagate. Matching run_oracle_loop / run_tier_sync_loop, the spawned task logs and continues; it does not return an error that would take down the indexer.
Don't touch the append-only logs. limit_order_placements/_cancellations/_fills and the parser are out of scope; this loop only writes the two Phase 1a current-state tables.
Out of scope (explicitly): rewiring the solver to read the mirror, the breaking quote_kind rename, poisoned-mirror fidelity, and concurrent candidate evaluation all belong to Phase 1c (#319). The 4-hop bump (Phase 2) and the path-candidate budget rethink (#286) are out of both.

Recommended direction

New module indexer/src/indexer/book_snapshot.rs, declared in indexer/src/indexer/mod.rs.
pub async fn run_book_snapshot_loop(pool: PgPool, lcd: LcdClient, snapshot_interval_ms: u64) modeled directly on run_oracle_loop: optional initial pass, then loop { snapshot_all_pairs(...).await (logged on error); sleep(interval) }.
async fn snapshot_all_pairs(pool, lcd):
- read the height once up front via lcd.get_latest_block_height() to stamp this cycle's block_height (best-effort; None if it fails, since the column is nullable);
- get_all_pairs(pool), then for each PairRow:
  - pool query -> map assets[0].amount / assets[1].amount to reserve_0 / reserve_1 (the pair's asset_0/asset_1 order; reuse the same asset-order convention pair_discovery established so reserves align with pairs.asset_0_id/asset_1_id); get_fee_config -> fee_bps as i16; call upsert_pair_reserves.
  - walk both book sides (bid, ask) via the order_book_head + linked limit_order pattern from fetch_limit_book_page; collect into Vec<RestingOrderInput> (side = "bid"/"ask", matching the table CHECK); call replace_pair_resting_orders(pool, pair_id, block_height, &orders).
  - per-pair errors -> tracing::warn! + skip.
Add a config field (e.g. book_snapshot_interval_ms, env BOOK_SNAPSHOT_INTERVAL_MS) next to oracle_poll_interval_ms in indexer/src/config.rs, with a sane default. Document the chosen interval as the snapshot cadence half of the freshness contract.
Wire it into poller.rs::run_indexer with a tokio::spawn cloning pool + lcd + the interval, alongside the oracle / tier-sync spawns.
Freshness contract, documented in the module header and a short note in docs/ if there's a fitting runbook:
- Cadence: target snapshot interval (the config default).
- Max staleness: the snapshot_at TTL beyond which Phase 1c should treat a row as stale and degrade (fall back to LCD or mark the quote degraded). Define it as a multiple of the cadence so it tolerates one missed cycle.
- Height lag: block_height is recorded per snapshot so Phase 1c can reason about block-lag, not just wall-clock staleness.
  These three (cadence, TTL, recorded height) are the contract Phase 1c codes against; surface the TTL as a documented constant so 1c imports it rather than re-deriving it.

Acceptance criteria

A book_snapshot background loop exists (module indexer/src/indexer/book_snapshot.rs), modeled on run_oracle_loop / run_tier_sync_loop, and is registered via tokio::spawn in poller.rs::run_indexer.
On each cycle the loop populates both Phase 1a tables for all discovered pairs: pair_reserves via upsert_pair_reserves and resting_limit_orders via replace_pair_resting_orders (both sides).
Reserves map to the correct reserve_0/reserve_1 per the pair's asset_0/asset_1 order; fee_bps is sourced from get_fee_config and stored as i16.
Resting orders are stored in the materialized book with correct side (bid/ask), price, remaining, and optional owner/expires_at, and read back in walk order via get_pair_resting_book.
LCD use is bounded per cycle: a documented, asserted upper bound on LCD calls per snapshot cycle as a function of pair count and book depth (the loop is the allowed LCD writer — the bound just has to exist and be finite).
Every snapshot stamps snapshot_at (via the existing NOW() defaults) and records the cycle's block_height when available.
A freshness contract is defined and documented: snapshot cadence (config interval), max-staleness TTL, and recorded block-height lag — with the TTL exposed as a constant Phase 1c can consume. Degrade-not-error semantics for stale/missing mirror are written down for 1c to implement against.
A per-pair query failure is logged and skipped; it does not abort the cycle, does not propagate an error out of the spawned loop, and leaves that pair's prior snapshot intact.
New config field (book_snapshot_interval_ms / BOOK_SNAPSHOT_INTERVAL_MS) with a default, wired through config.rs.

Test plan

Loop unit test: drive one snapshot_all_pairs pass against a mocked/faked LCD returning a known PoolResponse, FeeConfigResponse, and a small two-sided book; assert pair_reserves and resting_limit_orders end up with the expected rows (extend indexer/tests/db_orderbook_mirror.rs or a sibling test).
Round-trip: after a snapshot pass, get_pair_reserves returns the populated row and get_pair_resting_book(pair, "bid"/"ask") returns the orders in DESC/ASC walk order — reusing the Phase 1a accessors so the writer and reader agree.
Degrade path: a pair whose LCD query fails is skipped; get_pair_reserves for it stays None (or retains its prior row), and the cycle still completes for the other pairs.
Atomicity: replace_pair_resting_orders leaves no partial book on a mid-write failure (transaction rollback) — assert the pair's prior rows are unchanged.
Freshness: assert block_height is stamped when the height query succeeds and None when it doesn't, and that the documented max-staleness TTL constant is exported and non-zero.
LCD bound: a test asserting the per-cycle LCD-call upper bound is finite and matches the documented formula (mirrors the lcd_budget_is_documented_constant style test in best_execution.rs).

Parent program: #279 — 0-LCD hybrid solver (this is Phase 1b).
Depends on: Phase 1a schema + query layer, #279 (MR !761, commits 8ea4bc1 / 5392f96).
Blocks: #319 — Phase 1c (db_orderbook_sim + rewire solver to read the mirror). 1c's degrade-not-error semantics consume this loop's freshness contract.
Context: #286 (route DFS path-candidate budget) and the Phase 2 4-hop bump are out of scope here and in 1c.
Cache-key correctness already shipped: #283 (discount-tier keying, MR !751) and #306 (HTTP cache-tier-isolation test) — not re-done here.

## Summary Phase 1a ([#279](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/279), MR !761) landed the mirror **schema**: the `pair_reserves` and `resting_limit_orders` tables, plus the query layer that reads and writes them. Nothing populates those tables yet — they're empty in production. Phase 1b is the **writer**: a background loop, modeled on the existing `run_oracle_loop` / `run_tier_sync_loop` pattern, that periodically queries each pair's on-chain pool reserves + fee and its resting limit book, and writes them into the Phase 1a tables (`upsert_pair_reserves` + `replace_pair_resting_orders`). It also defines and records a **freshness contract** (snapshot interval + max staleness via `snapshot_at` / `block_height`) so Phase 1c can degrade-not-error when the mirror is stale or missing. This is the hard dependency that [#319](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/319) (Phase 1c) blocks on: the 0-LCD solver can't read a mirror that nothing fills. ## Current codebase **Phase 1a schema + query layer (done, MR !761):** - `indexer/migrations/20260605010000_pair_reserves.sql` — `pair_reserves` table: PK `pair_id`, `reserve_0`/`reserve_1` `NUMERIC(38,0)`, `fee_bps SMALLINT`, nullable `block_height BIGINT`, `snapshot_at TIMESTAMPTZ DEFAULT NOW()`. - `indexer/migrations/20260605010100_resting_limit_orders.sql` — `resting_limit_orders`: PK `(pair_id, order_id)`, `side` CHECK in `('bid','ask')`, `price NUMERIC(38,18)`, `remaining NUMERIC(38,0)`, nullable `owner`/`expires_at`/`block_height`, `snapshot_at`. Walk index `idx_resting_orders_book (pair_id, side, price, order_id)`. - `indexer/src/db/queries/pair_reserves.rs` — `upsert_pair_reserves(pool, pair_id, reserve_0, reserve_1, fee_bps, block_height)` does an `INSERT … ON CONFLICT (pair_id) DO UPDATE … snapshot_at = NOW()`; `get_pair_reserves(pool, pair_id) -> Option<PairReservesRow>` (missing snapshot = `None`, degrade-not-error). - `indexer/src/db/queries/resting_orders.rs` — `replace_pair_resting_orders(pool, pair_id, block_height, &[RestingOrderInput])` deletes the pair's rows and re-inserts the full book in one transaction; `get_pair_resting_book(pool, pair_id, side)` returns walk order (bids DESC / asks ASC, then FIFO by `order_id`). `RestingOrderInput { order_id, side, price, remaining, owner, expires_at }`. - Tests: `indexer/tests/db_orderbook_mirror.rs`. **Loop / poller precedents to mirror:** - `indexer/src/indexer/oracle.rs` — `run_oracle_loop(pool, poll_interval_ms, latest_price)` (line 18): builds a client, optional warm-load from DB, then `loop { … fetch … write … tokio::time::sleep(interval) }`. Interval comes from config (`oracle_poll_interval_ms`), failures are logged and the last-good state is retained, never propagated as a hard error out of the loop. - `indexer/src/indexer/trader_tracker.rs` — `run_tier_sync_loop(pool, lcd, fee_discount_addr)` (line 20): `loop { sleep; if let Err(e) = sync_tiers(...).await { tracing::error!(...) } }`. `sync_tiers` (line 39) pulls a row set, then per-row queries an LCD contract (`query_contract::<serde_json::Value>`) and upserts; a per-row LCD failure is `tracing::warn!`-ed and skipped, not fatal. - `indexer/src/indexer/poller.rs` — `run_indexer` registers each loop via `tokio::spawn` with cloned `pool` / `lcd` / config fields (lines 30-46). This is where the new loop gets wired in. - `indexer/src/indexer/mod.rs` — submodules are declared here (`pub mod oracle;` etc.); a new `pub mod book_snapshot;` goes alongside. **On-chain query shapes the loop reuses (already exercised elsewhere):** - Reserves + fee: `lcd.query_contract::<PoolResponse>(pair_addr, json!({"pool": {}}))` — `PoolResponse { assets: [Asset; 2], total_share }`, `Asset { info, amount }` (`indexer/src/lcd/types.rs:99-109`; live use at `indexer/src/api/orderbook_sim.rs:255-256`). Fee via `json!({"get_fee_config": {}})` -> `FeeConfigResponse { fee_config: { fee_bps, treasury } }` (`indexer/src/lcd/types.rs:111-120`; live use in `indexer/src/indexer/pair_discovery.rs:108-115`). `fee_bps` casts `u16 -> i16` exactly as `sync_single_pair` already does. - Resting book: walked via `json!({"order_book_head": {"side": side_label}})` to get the head order id, then linked `limit_order` lookups per order — see `fetch_limit_book_page` in `indexer/src/api/limit_book_lcd.rs:108-183` (head at line 129-133, per-order at 159-172). The loop reuses this walk to produce the full per-side book, mapping each order into `RestingOrderInput`. - Pair set to iterate: `get_all_pairs(pool) -> Vec<PairRow>` (`indexer/src/db/queries/pairs.rs:199`). - `LcdClient` is `#[derive(Clone)]` (`indexer/src/lcd/mod.rs:31`) and exposes `get_latest_block_height()` (`indexer/src/lcd/mod.rs:160`) for stamping `block_height`. **Solver side (downstream consumer, for context only — not touched here):** - `indexer/src/api/route_solver.rs` — `GET_DEFAULT_MAX_HOPS = 3` (line 31), `GET_POOL_ONLY_MAX_HOPS = 4` (line 33). - `indexer/src/api/best_execution.rs` — `solve_global_best_execution` over `MAX_PATH_CANDIDATES = 5` (line 18), `LCD_HYBRID_SIM_BUDGET` (line 26). These still hit LCD today; Phase 1c rewires them onto the mirror this loop fills. ## Why this is needed The whole 0-LCD hybrid solver program ([#279](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/279)) rests on the solver reading pool reserves and the resting book from Postgres instead of issuing per-request LCD calls. Phase 1a gave us the tables and accessors; without a writer they stay empty and Phase 1c ([#319](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/319)) has nothing to read. The LCD cost doesn't vanish — it moves out of the hot request path into one bounded background loop that amortizes it across all requests. The freshness contract is the other half. The mirror is eventually-consistent by construction (snapshot cadence, not block-exact). Phase 1c's degrade-not-error semantics (`get_pair_reserves` returning `None`, a stale `snapshot_at`) only mean something if Phase 1b defines what "stale" is and stamps every row with the data needed to evaluate it. ## Constraints and guardrails - **The loop is the one allowed LCD user.** It IS the mirror writer, so per-pair `pool` / `get_fee_config` / book-walk LCD calls are expected and fine. What must stay bounded is total LCD per snapshot cycle — see acceptance below. This is the inverse of the solver-path budgets (`LCD_HYBRID_SIM_BUDGET` etc.), which Phase 1c drives toward zero. - **Atomic per-pair writes only.** Use `replace_pair_resting_orders` (single transaction, delete-then-insert) so a pair's book is never observed half-updated. `upsert_pair_reserves` is already a single statement. - **A per-pair failure must not poison the cycle.** Follow the `sync_tiers` shape: a failed pool/fee/book query for one pair is `tracing::warn!`-ed and skipped; the loop keeps the last good snapshot for that pair (`upsert` not run, rows not replaced) and moves on. A missing/failed snapshot for a pair leaves Phase 1c to degrade on it, not error. - **Loop errors never propagate.** Matching `run_oracle_loop` / `run_tier_sync_loop`, the spawned task logs and continues; it does not return an error that would take down the indexer. - **Don't touch the append-only logs.** `limit_order_placements/_cancellations/_fills` and the parser are out of scope; this loop only writes the two Phase 1a current-state tables. - **Out of scope (explicitly):** rewiring the solver to read the mirror, the breaking `quote_kind` rename, poisoned-mirror fidelity, and concurrent candidate evaluation all belong to Phase 1c ([#319](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/319)). The 4-hop bump (Phase 2) and the path-candidate budget rethink ([#286](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/286)) are out of both. ## Recommended direction 1. New module `indexer/src/indexer/book_snapshot.rs`, declared in `indexer/src/indexer/mod.rs`. 2. `pub async fn run_book_snapshot_loop(pool: PgPool, lcd: LcdClient, snapshot_interval_ms: u64)` modeled directly on `run_oracle_loop`: optional initial pass, then `loop { snapshot_all_pairs(...).await (logged on error); sleep(interval) }`. 3. `async fn snapshot_all_pairs(pool, lcd)`: - read the height once up front via `lcd.get_latest_block_height()` to stamp this cycle's `block_height` (best-effort; `None` if it fails, since the column is nullable); - `get_all_pairs(pool)`, then for each `PairRow`: - `pool` query -> map `assets[0].amount` / `assets[1].amount` to `reserve_0` / `reserve_1` (the pair's `asset_0`/`asset_1` order; reuse the same asset-order convention `pair_discovery` established so reserves align with `pairs.asset_0_id`/`asset_1_id`); `get_fee_config` -> `fee_bps as i16`; call `upsert_pair_reserves`. - walk both book sides (`bid`, `ask`) via the `order_book_head` + linked `limit_order` pattern from `fetch_limit_book_page`; collect into `Vec<RestingOrderInput>` (`side` = `"bid"`/`"ask"`, matching the table CHECK); call `replace_pair_resting_orders(pool, pair_id, block_height, &orders)`. - per-pair errors -> `tracing::warn!` + skip. 4. Add a config field (e.g. `book_snapshot_interval_ms`, env `BOOK_SNAPSHOT_INTERVAL_MS`) next to `oracle_poll_interval_ms` in `indexer/src/config.rs`, with a sane default. Document the chosen interval as the snapshot cadence half of the freshness contract. 5. Wire it into `poller.rs::run_indexer` with a `tokio::spawn` cloning `pool` + `lcd` + the interval, alongside the oracle / tier-sync spawns. 6. **Freshness contract**, documented in the module header and a short note in `docs/` if there's a fitting runbook: - **Cadence:** target snapshot interval (the config default). - **Max staleness:** the `snapshot_at` TTL beyond which Phase 1c should treat a row as stale and degrade (fall back to LCD or mark the quote degraded). Define it as a multiple of the cadence so it tolerates one missed cycle. - **Height lag:** `block_height` is recorded per snapshot so Phase 1c can reason about block-lag, not just wall-clock staleness. These three (cadence, TTL, recorded height) are the contract Phase 1c codes against; surface the TTL as a documented constant so 1c imports it rather than re-deriving it. ## Acceptance criteria - [ ] A `book_snapshot` background loop exists (module `indexer/src/indexer/book_snapshot.rs`), modeled on `run_oracle_loop` / `run_tier_sync_loop`, and is registered via `tokio::spawn` in `poller.rs::run_indexer`. - [ ] On each cycle the loop populates **both** Phase 1a tables for **all** discovered pairs: `pair_reserves` via `upsert_pair_reserves` and `resting_limit_orders` via `replace_pair_resting_orders` (both sides). - [ ] Reserves map to the correct `reserve_0`/`reserve_1` per the pair's `asset_0`/`asset_1` order; `fee_bps` is sourced from `get_fee_config` and stored as `i16`. - [ ] Resting orders are stored in the materialized book with correct `side` (`bid`/`ask`), `price`, `remaining`, and optional `owner`/`expires_at`, and read back in walk order via `get_pair_resting_book`. - [ ] LCD use is **bounded per cycle**: a documented, asserted upper bound on LCD calls per snapshot cycle as a function of pair count and book depth (the loop is the allowed LCD writer — the bound just has to exist and be finite). - [ ] Every snapshot stamps `snapshot_at` (via the existing `NOW()` defaults) and records the cycle's `block_height` when available. - [ ] A **freshness contract** is defined and documented: snapshot cadence (config interval), max-staleness TTL, and recorded block-height lag — with the TTL exposed as a constant Phase 1c can consume. Degrade-not-error semantics for stale/missing mirror are written down for 1c to implement against. - [ ] A per-pair query failure is logged and skipped; it does not abort the cycle, does not propagate an error out of the spawned loop, and leaves that pair's prior snapshot intact. - [ ] New config field (`book_snapshot_interval_ms` / `BOOK_SNAPSHOT_INTERVAL_MS`) with a default, wired through `config.rs`. ## Test plan - **Loop unit test:** drive one `snapshot_all_pairs` pass against a mocked/faked LCD returning a known `PoolResponse`, `FeeConfigResponse`, and a small two-sided book; assert `pair_reserves` and `resting_limit_orders` end up with the expected rows (extend `indexer/tests/db_orderbook_mirror.rs` or a sibling test). - **Round-trip:** after a snapshot pass, `get_pair_reserves` returns the populated row and `get_pair_resting_book(pair, "bid"/"ask")` returns the orders in DESC/ASC walk order — reusing the Phase 1a accessors so the writer and reader agree. - **Degrade path:** a pair whose LCD query fails is skipped; `get_pair_reserves` for it stays `None` (or retains its prior row), and the cycle still completes for the other pairs. - **Atomicity:** `replace_pair_resting_orders` leaves no partial book on a mid-write failure (transaction rollback) — assert the pair's prior rows are unchanged. - **Freshness:** assert `block_height` is stamped when the height query succeeds and `None` when it doesn't, and that the documented max-staleness TTL constant is exported and non-zero. - **LCD bound:** a test asserting the per-cycle LCD-call upper bound is finite and matches the documented formula (mirrors the `lcd_budget_is_documented_constant` style test in `best_execution.rs`). ## Related - Parent program: [#279](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/279) — 0-LCD hybrid solver (this is Phase 1b). - Depends on: Phase 1a schema + query layer, [#279](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/279) (MR !761, commits 8ea4bc1 / 5392f96). - Blocks: [#319](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/319) — Phase 1c (`db_orderbook_sim` + rewire solver to read the mirror). 1c's degrade-not-error semantics consume this loop's freshness contract. - Context: [#286](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/286) (route DFS path-candidate budget) and the Phase 2 4-hop bump are out of scope here and in 1c. - Cache-key correctness already shipped: [#283](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/283) (discount-tier keying, MR !751) and [#306](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/306) (HTTP cache-tier-isolation test) — not re-done here.

Brouie commented

2026-06-05 08:23:46 +00:00

(Migrated from gitlab.com)

mentioned in issue #279

Brouie commented

2026-06-05 08:23:47 +00:00

(Migrated from gitlab.com)

mentioned in issue #319

Brouie commented

2026-06-05 08:24:27 +00:00

(Migrated from gitlab.com)

marked as related to #319

ghost1 commented

2026-06-05 10:14:16 +00:00

(Migrated from gitlab.com)

mentioned in commit 6c5a09acee

mentioned in commit 6c5a09acee8cc131c6d1025bb49c66449bfa3a59

PlasticDigits commented

2026-06-05 10:14:37 +00:00

(Migrated from gitlab.com)

mentioned in merge request !793

PlasticDigits commented

2026-06-05 10:14:47 +00:00

(Migrated from gitlab.com)

Implementation complete — MR !793

Opened !793 for #322.

What shipped

book_snapshot background loop (indexer/src/indexer/book_snapshot.rs) spawned from poller.rs
Populates pair_reserves + resting_limit_orders each cycle for all indexed pairs
Freshness contract: BOOK_SNAPSHOT_MAX_STALENESS_MS (20s at default 10s cadence), block-height stamping, degrade-not-error semantics documented for Phase 1c
Config: BOOK_SNAPSHOT_INTERVAL_MS (default 10_000)
LCD budget: book_snapshot_lcd_budget(pair_count, total_resting_orders)

Verification (all PASS)

Item	Command
Unit tests	`cd indexer && cargo test --lib book_snapshot`
Snapshot round-trip	`cargo test --test book_snapshot_loop -j 1 -- --test-threads=1`
Atomicity rollback	`cargo test --test db_orderbook_mirror resting_book_replace_rolls_back -j 1 -- --test-threads=1`

Issue left open pending MR review/merge.

## Implementation complete — MR !793 Opened [!793](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/merge_requests/75) for [#322](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/322). ### What shipped - `book_snapshot` background loop (`indexer/src/indexer/book_snapshot.rs`) spawned from `poller.rs` - Populates `pair_reserves` + `resting_limit_orders` each cycle for all indexed pairs - Freshness contract: `BOOK_SNAPSHOT_MAX_STALENESS_MS` (20s at default 10s cadence), block-height stamping, degrade-not-error semantics documented for Phase 1c - Config: `BOOK_SNAPSHOT_INTERVAL_MS` (default 10_000) - LCD budget: `book_snapshot_lcd_budget(pair_count, total_resting_orders)` ### Verification (all PASS) | Item | Command | |------|---------| | Unit tests | `cd indexer && cargo test --lib book_snapshot` | | Snapshot round-trip | `cargo test --test book_snapshot_loop -j 1 -- --test-threads=1` | | Atomicity rollback | `cargo test --test db_orderbook_mirror resting_book_replace_rolls_back -j 1 -- --test-threads=1` | Issue left **open** pending MR review/merge.

PlasticDigits commented

2026-06-05 10:39:55 +00:00

(Migrated from gitlab.com)

mentioned in commit 74c0449118

mentioned in commit 74c0449118fc30040f3764f8a444d8ffcc1eafa5

PlasticDigits commented

2026-06-05 11:16:52 +00:00

(Migrated from gitlab.com)

Verification complete — #322

Independent QA pass after MR !793 merge (74c0449). No repo changes required.

Acceptance criteria

Criterion	Result	Evidence
`book_snapshot` loop module + `tokio::spawn` in `poller.rs`	PASS	`indexer/src/indexer/book_snapshot.rs`, `poller.rs` L50–55, `mod.rs`
Each cycle fills `pair_reserves` + `resting_limit_orders` for all pairs	PASS	`snapshot_all_pairs` → `snapshot_single_pair`; integration `snapshot_populates_reserves_and_resting_book`
`reserve_0`/`reserve_1` + `fee_bps` from LCD	PASS	`pool` + `get_fee_config` in `snapshot_single_pair`; asserts in `book_snapshot_loop.rs`
Resting book walk order (bid DESC / ask ASC, FIFO)	PASS	`get_pair_resting_book` asserts `vec![101,102,100]` bids, `vec![201,202]` asks
Bounded LCD per cycle	PASS	`book_snapshot_lcd_budget()` + `BOOK_SNAPSHOT_LCD_*` constants; `lcd_budget_constant_matches_formula`
`snapshot_at` + `block_height` stamping	PASS	DB defaults; `block_height: Some(12345)` / `None` tests
Freshness contract (cadence, TTL, height lag)	PASS	Module header + `docs/runbooks/book-snapshot-mirror.md` + `BOOK_SNAPSHOT_MAX_STALENESS_MS`
Per-pair failure → warn, skip, keep prior snapshot	PASS	`snapshot_skips_failed_pair_and_keeps_prior_snapshot`
Config `BOOK_SNAPSHOT_INTERVAL_MS` (default 10_000)	PASS	`indexer/src/config.rs`

Test plan

Test	Result	Command
Unit (LCD budget + TTL)	PASS	`cd indexer && cargo test --lib book_snapshot::tests` → 2 passed
Snapshot round-trip	PASS	`cargo test --test book_snapshot_loop -j 1 -- --test-threads=1` → 4 passed
Phase 1a accessors / atomicity	PASS	`cargo test --test db_orderbook_mirror -j 1 -- --test-threads=1` → 4 passed
Degrade path (failed pair skipped)	PASS	`snapshot_skips_failed_pair_and_keeps_prior_snapshot`
Block height optional	PASS	`snapshot_block_height_none_when_lcd_height_fails`

Manual / code review

Check	Result	Notes
Loop errors do not propagate out of spawn	PASS	`run_book_snapshot_loop` logs cycle errors and sleeps; per-pair `warn!` + continue
Atomic per-pair write	PASS	Single tx: `upsert_pair_reserves` + `replace_pair_resting_orders_in_tx` then commit
Append-only order logs untouched	PASS	Only writes `pair_reserves` / `resting_limit_orders`
Docs / invariants cross-link	PASS	`docs/indexer-invariants.md`, runbook linked

Environment: Postgres via docker compose up -d postgres; TEST_DATABASE_URL from indexer/.env (synced by setup-postgres-dev-databases.sh).

Closing as verified — implementation merged on main.

## Verification complete — [#322](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/322) Independent QA pass after MR !793 merge (`74c0449`). No repo changes required. ### Acceptance criteria | Criterion | Result | Evidence | |-----------|--------|----------| | `book_snapshot` loop module + `tokio::spawn` in `poller.rs` | **PASS** | `indexer/src/indexer/book_snapshot.rs`, `poller.rs` L50–55, `mod.rs` | | Each cycle fills `pair_reserves` + `resting_limit_orders` for all pairs | **PASS** | `snapshot_all_pairs` → `snapshot_single_pair`; integration `snapshot_populates_reserves_and_resting_book` | | `reserve_0`/`reserve_1` + `fee_bps` from LCD | **PASS** | `pool` + `get_fee_config` in `snapshot_single_pair`; asserts in `book_snapshot_loop.rs` | | Resting book walk order (bid DESC / ask ASC, FIFO) | **PASS** | `get_pair_resting_book` asserts `vec![101,102,100]` bids, `vec![201,202]` asks | | Bounded LCD per cycle | **PASS** | `book_snapshot_lcd_budget()` + `BOOK_SNAPSHOT_LCD_*` constants; `lcd_budget_constant_matches_formula` | | `snapshot_at` + `block_height` stamping | **PASS** | DB defaults; `block_height: Some(12345)` / `None` tests | | Freshness contract (cadence, TTL, height lag) | **PASS** | Module header + `docs/runbooks/book-snapshot-mirror.md` + `BOOK_SNAPSHOT_MAX_STALENESS_MS` | | Per-pair failure → warn, skip, keep prior snapshot | **PASS** | `snapshot_skips_failed_pair_and_keeps_prior_snapshot` | | Config `BOOK_SNAPSHOT_INTERVAL_MS` (default 10_000) | **PASS** | `indexer/src/config.rs` | ### Test plan | Test | Result | Command | |------|--------|---------| | Unit (LCD budget + TTL) | **PASS** | `cd indexer && cargo test --lib book_snapshot::tests` → 2 passed | | Snapshot round-trip | **PASS** | `cargo test --test book_snapshot_loop -j 1 -- --test-threads=1` → 4 passed | | Phase 1a accessors / atomicity | **PASS** | `cargo test --test db_orderbook_mirror -j 1 -- --test-threads=1` → 4 passed | | Degrade path (failed pair skipped) | **PASS** | `snapshot_skips_failed_pair_and_keeps_prior_snapshot` | | Block height optional | **PASS** | `snapshot_block_height_none_when_lcd_height_fails` | ### Manual / code review | Check | Result | Notes | |-------|--------|-------| | Loop errors do not propagate out of spawn | **PASS** | `run_book_snapshot_loop` logs cycle errors and sleeps; per-pair `warn!` + continue | | Atomic per-pair write | **PASS** | Single tx: `upsert_pair_reserves` + `replace_pair_resting_orders_in_tx` then commit | | Append-only order logs untouched | **PASS** | Only writes `pair_reserves` / `resting_limit_orders` | | Docs / invariants cross-link | **PASS** | `docs/indexer-invariants.md`, runbook linked | **Environment:** Postgres via `docker compose up -d postgres`; `TEST_DATABASE_URL` from `indexer/.env` (synced by `setup-postgres-dev-databases.sh`). Closing as verified — implementation merged on `main`.

PlasticDigits (Migrated from gitlab.com) closed this issue

2026-06-05 11:16:55 +00:00

PlasticDigits commented

2026-06-05 11:20:03 +00:00

(Migrated from gitlab.com)

mentioned in merge request !798