fix(indexer): set-based CG/CMC aggregator queries + top-100 pagination (#288) #781

Merged

PlasticDigits merged 1 commit from cursor/gitlab-issue-workflow-6fa7 into main

2026-06-05 09:57:37 +00:00

PlasticDigits commented

2026-06-05 09:53:09 +00:00

(Migrated from gitlab.com)

Summary

Closes the remaining AC on https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/288.

The 60s TTL cache (already on main) bounded how often the per-pair N+1 fanout ran; this MR replaces that fanout with O(1) set-based queries and adds option (b) pagination per stakeholder decision: default top 100 pairs by 24h quote volume, with optional limit/offset.

Changes

get_24h_stats_all_pairs / get_24h_hybrid_breakdown_all_pairs — grouped queries over swap_events
aggregator_snapshot — shared loader: rank pairs by 24h quote volume, apply pagination
/cg/tickers, /cmc/summary, /cmc/ticker — accept limit (default 100) and offset; cache keyed per endpoint + pagination
Docs: CG_CMC_COMPLIANCE.md, indexer-invariants.md

Acceptance checklist

Criterion	Verification	Result
O(1) queries not O(pairs)	cargo test --test api_aggregator_batch batch_stats_match_per_pair_queries	PASS
Concurrent requests do not exhaust pool	60s cache + set-based queries; live burst test SKIP (no live stack)	SKIP
Latency flat as pair count grows	Set-based + default limit 100; live scale test SKIP	SKIP
Option (b) pagination default top 100	cargo test --test api_aggregator_batch	PASS

Verification

cd indexer && cargo test --test api_aggregator_batch --test api_cg --test api_cmc --test api_consolidated_reporting -j 1 -- --test-threads=1

Note

Medium Risk
Public listing endpoints change default payload size and ordering (top 100 by volume); integrators expecting the full pair list must paginate, though behavior is read-only and covered by tests.

Overview
Replaces per-pair N+1 24h stats on CoinGecko/CoinMarketCap listing routes with set-based DB loads and shared aggregator_snapshot ranking, so aggregator crawlers no longer fan out one query per pair.

/cg/tickers, /cmc/summary, and /cmc/ticker now accept limit (default 100, max 1000) and offset (max 10_000 → 400). Results are ordered by 24h quote volume descending, then sliced. The existing 60s response cache is keyed per endpoint + (limit, offset) instead of one global key per route.

New query helpers get_24h_stats_all_pairs and get_24h_hybrid_breakdown_all_pairs back the loader; extensions_from_breakdown builds cl8y_extensions from the batch hybrid map. Docs and indexer-invariants record the new contract; api_aggregator_batch and consolidated reporting tests assert batch parity, pagination, and volume sort.

^{Reviewed by Cursor Bugbot for commit 8f72ce6c61. Bugbot is set up for automated code reviews on this repo. Configure here.}

## Summary Closes the remaining AC on https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/288. The 60s TTL cache (already on main) bounded how often the per-pair N+1 fanout ran; this MR replaces that fanout with **O(1) set-based queries** and adds **option (b) pagination** per stakeholder decision: default **top 100 pairs by 24h quote volume**, with optional limit/offset. ### Changes - get_24h_stats_all_pairs / get_24h_hybrid_breakdown_all_pairs — grouped queries over swap_events - aggregator_snapshot — shared loader: rank pairs by 24h quote volume, apply pagination - /cg/tickers, /cmc/summary, /cmc/ticker — accept limit (default 100) and offset; cache keyed per endpoint + pagination - Docs: CG_CMC_COMPLIANCE.md, indexer-invariants.md ## Acceptance checklist | Criterion | Verification | Result | |-----------|--------------|--------| | O(1) queries not O(pairs) | cargo test --test api_aggregator_batch batch_stats_match_per_pair_queries | PASS | | Concurrent requests do not exhaust pool | 60s cache + set-based queries; live burst test SKIP (no live stack) | SKIP | | Latency flat as pair count grows | Set-based + default limit 100; live scale test SKIP | SKIP | | Option (b) pagination default top 100 | cargo test --test api_aggregator_batch | PASS | ## Verification cd indexer && cargo test --test api_aggregator_batch --test api_cg --test api_cmc --test api_consolidated_reporting -j 1 -- --test-threads=1  --- > [!NOTE] > **Medium Risk** > Public listing endpoints change default payload size and ordering (top 100 by volume); integrators expecting the full pair list must paginate, though behavior is read-only and covered by tests. > > **Overview** > Replaces **per-pair N+1** 24h stats on CoinGecko/CoinMarketCap listing routes with **set-based** DB loads and shared **`aggregator_snapshot`** ranking, so aggregator crawlers no longer fan out one query per pair. > > **`/cg/tickers`**, **`/cmc/summary`**, and **`/cmc/ticker`** now accept **`limit`** (default **100**, max 1000) and **`offset`** (max 10_000 → **400**). Results are ordered by **24h quote volume** descending, then sliced. The existing **60s** response cache is keyed per **endpoint + `(limit, offset)`** instead of one global key per route. > > New query helpers **`get_24h_stats_all_pairs`** and **`get_24h_hybrid_breakdown_all_pairs`** back the loader; **`extensions_from_breakdown`** builds `cl8y_extensions` from the batch hybrid map. Docs and **`indexer-invariants`** record the new contract; **`api_aggregator_batch`** and consolidated reporting tests assert batch parity, pagination, and volume sort. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 8f72ce6c61c36d07da87513acefc601dea1c089f. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

PlasticDigits commented

2026-06-05 09:53:17 +00:00

(Migrated from gitlab.com)

mentioned in issue #288

ghost1 commented

2026-06-05 09:53:17 +00:00

(Migrated from gitlab.com)

changed the description

ghost1 commented

2026-06-05 09:53:26 +00:00

(Migrated from gitlab.com)

changed the description

PlasticDigits commented

2026-06-05 09:54:53 +00:00

(Migrated from gitlab.com)

Security review

Commit reviewed: 8f72ce6c61c36d07da87513acefc601dea1c089f
Scope: Set-based CG/CMC aggregator queries (get_24h_stats_all_pairs, get_24h_hybrid_breakdown_all_pairs), shared aggregator_snapshot loader, pagination on /cg/tickers, /cmc/summary, /cmc/ticker, and per-endpoint cache keys.

Outcome: FINDINGS: 0 medium+

Summary

No medium, high, or critical vulnerabilities identified in this diff.

Reviewed attack surfaces:

SQL injection — New grouped queries in swap_events.rs use parameterized $1 binds only; limit/offset never reach SQL.
Input validation — parse_aggregator_list_query clamps limit to [1, 1000], bounds offset to [0, 10_000] (400 above max), and rejects excessive offset. Matches existing /cg/pairs pagination pattern.
Authn/authz — Endpoints remain public read-only market data; no permission boundary changes.
Secret leakage / logging — Errors still routed through internal_err() (generic 500 body).
DoS — This MR reduces per-request DB fanout from O(pairs) N+1 to O(1) set-based queries. Per-pagination cache keys expand the cache keyspace vs. the prior single-key design, but abuse is bounded by existing global rate limiting (default 60 RPS) and the 60s TTL with eviction on insert. Not a medium+ finding given controls and substantially lower per-miss cost.
XSS / SSRF / deserialization — Not applicable to these changes.

Inline threads: none (no findings to anchor).

Automated security review (Cursor Cloud Agent)

## Security review **Commit reviewed:** `8f72ce6c61c36d07da87513acefc601dea1c089f` **Scope:** Set-based CG/CMC aggregator queries (`get_24h_stats_all_pairs`, `get_24h_hybrid_breakdown_all_pairs`), shared `aggregator_snapshot` loader, pagination on `/cg/tickers`, `/cmc/summary`, `/cmc/ticker`, and per-endpoint cache keys. **Outcome:** `FINDINGS: 0` medium+ ### Summary No medium, high, or critical vulnerabilities identified in this diff. **Reviewed attack surfaces:** - **SQL injection** — New grouped queries in `swap_events.rs` use parameterized `$1` binds only; `limit`/`offset` never reach SQL. - **Input validation** — `parse_aggregator_list_query` clamps `limit` to `[1, 1000]`, bounds `offset` to `[0, 10_000]` (400 above max), and rejects excessive offset. Matches existing `/cg/pairs` pagination pattern. - **Authn/authz** — Endpoints remain public read-only market data; no permission boundary changes. - **Secret leakage / logging** — Errors still routed through `internal_err()` (generic 500 body). - **DoS** — This MR reduces per-request DB fanout from O(pairs) N+1 to O(1) set-based queries. Per-pagination cache keys expand the cache keyspace vs. the prior single-key design, but abuse is bounded by existing global rate limiting (default 60 RPS) and the 60s TTL with eviction on insert. Not a medium+ finding given controls and substantially lower per-miss cost. - **XSS / SSRF / deserialization** — Not applicable to these changes. **Inline threads:** none (no findings to anchor). --- *Automated security review (Cursor Cloud Agent)*

PlasticDigits commented

2026-06-05 09:57:37 +00:00

(Migrated from gitlab.com)

mentioned in commit 706856fd17

mentioned in commit 706856fd17eec6ff6ba1f498b995cbab8a94f9ce