Indexer: event-driven fee-tier sync instead of full trader scan #364

New issue

ops-worker · 2026-06-12T05:05:45Z

PlasticDigits commented

2026-06-12 05:05:45 +00:00

(Migrated from gitlab.com)

Parent

Gap analysis follow-up from GitLab #361 — register items M11 and approved recommendation event-driven fee-tier sync.

Current codebase

trader_tracker::run_tier_sync_loop sleeps 600s, then loads all rows from traders and queries get_registration on the fee-discount contract per address (indexer/src/indexer/trader_tracker.rs:20-88).
Spawned from poller.rs at indexer startup; disabled when FEE_DISCOUNT_ADDRESS unset.
Swaps upsert trader volume via update_trader_on_swap (parser.rs) but do not refresh tier on registration changes.
Route solver cache keys on resolved discount tier (#283, #245) — stale tiers cause wrong quoted fees until next full scan.

Why needed

LCD load scales O(unique traders) every 10 minutes. As trader count grows, this becomes a steady LCD tax and increases tier staleness window for route quotes and trader API responses.

Constraints / guardrails

Preserve correct tier for route solver cache keys (no cross-tier cache collisions).
Keep a fallback periodic sync (longer interval) for drift correction, not only event-driven.
Do not add hosted MR CI requirement.
Minimize new LCD calls per block; batch where possible.

Relevant files

indexer/src/indexer/trader_tracker.rs
indexer/src/indexer/parser.rs
indexer/src/indexer/poller.rs
indexer/src/db/queries/traders.rs
indexer/src/api/route_solver.rs
smartcontracts/contracts/fee_discount/ (Register / Deregister / RegisterWallet events)

Recommended direction

Index fee-discount registry txs during block parsing: on Register, Deregister, RegisterWallet, DeregisterWallet, UpdateTier (as applicable), update traders.tier_id/tier_name/registered for affected addresses.
On first swap from a new sender: single get_registration LCD query (lazy hydrate).
Replace 10-min full scan with daily reconciliation (or configurable TIER_SYNC_RECONCILE_INTERVAL) for missed events.
Optional: expose tier_sync_lag_seconds in logs for operators (not Prometheus — out of scope).

Acceptance criteria

Registration change on-chain reflected in traders table within one indexed block (integration test).
Full-table LCD scan no longer runs every 10 minutes by default.
Route solver quotes match on-chain discount tier after registration tx (regression for #245).
Reconciliation job still corrects drift if event parse missed.

Test plan

Path	Command / action
Lib	`cd indexer && cargo test --lib`
Integration	`make test-indexer-integration` — add/update binary for tier sync
Manual LocalTerra	Register wallet on fee-discount contract → verify indexer trader row + route quote tier within 1 block

Attack / abuse test plan

Malformed registry wasm events must not crash indexer; log and continue.
Burst of registration txs in one block → bounded LCD queries (batch or dedupe per address per block).

Verification criteria

Integration test: register tier change → GET /api/v1/traders/{addr} shows new tier without waiting for reconcile loop.
LCD query count during steady-state indexing does not scale linearly with total historical traders on each short interval.

## Parent Gap analysis follow-up from [GitLab #361](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/361) — register items **M11** and approved recommendation *event-driven fee-tier sync*. ## Current codebase - `trader_tracker::run_tier_sync_loop` sleeps **600s**, then loads **all** rows from `traders` and queries `get_registration` on the fee-discount contract per address (`indexer/src/indexer/trader_tracker.rs:20-88`). - Spawned from `poller.rs` at indexer startup; disabled when `FEE_DISCOUNT_ADDRESS` unset. - Swaps upsert trader volume via `update_trader_on_swap` (`parser.rs`) but do **not** refresh tier on registration changes. - Route solver cache keys on resolved discount tier ([#283](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/283), [#245](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/245)) — stale tiers cause wrong quoted fees until next full scan. ## Why needed LCD load scales **O(unique traders)** every 10 minutes. As trader count grows, this becomes a steady LCD tax and increases tier staleness window for route quotes and trader API responses. ## Constraints / guardrails - Preserve correct tier for route solver cache keys (no cross-tier cache collisions). - Keep a **fallback** periodic sync (longer interval) for drift correction, not only event-driven. - Do not add hosted MR CI requirement. - Minimize new LCD calls per block; batch where possible. ## Relevant files - `indexer/src/indexer/trader_tracker.rs` - `indexer/src/indexer/parser.rs` - `indexer/src/indexer/poller.rs` - `indexer/src/db/queries/traders.rs` - `indexer/src/api/route_solver.rs` - `smartcontracts/contracts/fee_discount/` (Register / Deregister / RegisterWallet events) ## Recommended direction 1. **Index fee-discount registry txs** during block parsing: on `Register`, `Deregister`, `RegisterWallet`, `DeregisterWallet`, `UpdateTier` (as applicable), update `traders.tier_id/tier_name/registered` for affected addresses. 2. **On first swap** from a new sender: single `get_registration` LCD query (lazy hydrate). 3. Replace 10-min full scan with **daily** reconciliation (or configurable `TIER_SYNC_RECONCILE_INTERVAL`) for missed events. 4. Optional: expose `tier_sync_lag_seconds` in logs for operators (not Prometheus — out of scope). ## Acceptance criteria - [ ] Registration change on-chain reflected in `traders` table within one indexed block (integration test). - [ ] Full-table LCD scan no longer runs every 10 minutes by default. - [ ] Route solver quotes match on-chain discount tier after registration tx (regression for #245). - [ ] Reconciliation job still corrects drift if event parse missed. ## Test plan | Path | Command / action | |------|------------------| | Lib | `cd indexer && cargo test --lib` | | Integration | `make test-indexer-integration` — add/update binary for tier sync | | Manual LocalTerra | Register wallet on fee-discount contract → verify indexer trader row + route quote tier within 1 block | ## Attack / abuse test plan - Malformed registry wasm events must not crash indexer; log and continue. - Burst of registration txs in one block → bounded LCD queries (batch or dedupe per address per block). ## Verification criteria - Integration test: register tier change → `GET /api/v1/traders/{addr}` shows new tier without waiting for reconcile loop. - LCD query count during steady-state indexing does not scale linearly with total historical traders on each short interval.

PlasticDigits commented

2026-06-12 05:06:04 +00:00

(Migrated from gitlab.com)

mentioned in issue #361