feat: Charts overview 24h volume USD-only (fix $0 / 10,000,000T) and audit other stats

PlasticDigits commented

2026-08-17 10:29:09 +00:00

(Migrated from gitlab.com)

Summary

On Charts & Analytics (/charts) the overview strip still shows a meaningless raw 24h Volume (10,000,000T) next to 24h Volume (USD) = 0, even when 24h trades exist. Retail should see one 24h volume figure, in USD, with a correct (non-zero when catalog-priced volume exists) number. Also audit the other overview boxes (USTC/USD, 24h Trades, Pairs, Tokens) so they match documented indexer semantics.

Observed (columbus-5 / dex.cl8y.com):

Box	Shown	Problem
24h Volume	`10,000,000T`	`formatNum(total_volume_24h)` on mixed raw `SUM(offer_amount)` — same `T` class as #534 / #544
24h Volume (USD)	`0` (no `$`)	Ingest `volume_usd` is USTC-leg only (X4); `COALESCE(SUM(NULL),0)` → `"0"`; UI treats `"0"` as a real value
USTC / USD	`$0.00487800`	Plausible oracle print — confirm source, empty/stale, formatting
24h Trades	`4`	Confirm = 24h `swap_events` count (L10, not fills)
Pairs	`13`	Confirm = indexed factory pairs
Tokens	`12`	Confirm = unique pair-leg tokens, not every `assets` row

Related (do not treat as done): #544 pair-list / picker USD volume + shared ingest catalog; #540 pair stats strip still formatNum(raw); #522 P522-Q quote USD catalog; #515 oracles; #281 / #333 overview rollup + 60s cache.

This issue uniquely owns the /charts overview strip. Pair-search / /pool badges stay on #544. Pair-level 24h stats (Vol (AAA) / Vol (BBB)) stay on #540 / #544. Share one compute_volume_usd catalog change — do not invent a second USD formula.

Current codebase

Frontend — two volume boxes, raw first

frontend-dapp/src/pages/ChartsPage.tsx loads GET /api/v1/overview via getOverview and renders six StatBoxes:

          <StatBox
            label="24h Volume"
            value={overview ? formatNum(overview.total_volume_24h) : '—'}
            loading={overviewQuery.isLoading}
          />
          <StatBox
            label="24h Volume (USD)"
            value={
              overview?.total_volume_24h_usd != null && overview.total_volume_24h_usd !== ''
                ? formatNum(overview.total_volume_24h_usd, 2)
                : '—'
            }
            loading={overviewQuery.isLoading}
          />
          <StatBox
            label="USTC / USD"
            value={
              overview?.ustc_price_usd != null && overview.ustc_price_usd !== ''
                ? `$${formatNum(overview.ustc_price_usd, 6)}`
                : '—'
            }
            ...
          />
          <StatBox label="24h Trades" value={overview ? overview.total_trades_24h.toLocaleString() : '—'} />
          <StatBox label="Pairs" value={overview ? overview.pair_count.toString() : '—'} />
          <StatBox label="Tokens" value={overview ? overview.token_count.toString() : '—'} />

total_volume_24h is a raw mixed-decimal integer (SUM(offer_amount) across all offer assets). Passing it to formatNum compact-formats ≥1e12 as T. Ten human USTR (18 decimals) is 1e19 raw → 10,000,000T. There is no unit on the label.
total_volume_24h_usd: formatNum(x, 2) with no $. The empty check allows "0" / "0.0" through, so a true zero and a missing-USD rollup both render 0. Contrast USTC/USD, which prefixes $.
ChartsPage.test.tsx mocks both volumes as '0' and never asserts the overview strip labels or values.
Grid is lg:grid-cols-6; dropping one box should reflow (5 boxes).

Indexer — overview JSON + USTC-only `volume_usd`

GET /api/v1/overview (OverviewResponse):

Field	Source	Notes
`total_volume_24h`	`global_stats_24h.total_volume`	`SUM(offer_amount)` raw — not human, not USD
`total_volume_24h_usd`	`global_stats_24h.total_volume_usd`	`COALESCE(SUM(volume_usd), 0)`
`total_trades_24h`	`global_stats_24h.total_trades`	`COUNT(*)` of `swap_events` in 24h
`pair_count`	live `SELECT COUNT(*) FROM pairs`	cache-miss only
`token_count`	`assets::get_all_assets(&pool)?.len()`	loads every `assets` row into memory just to count
`ustc_price_usd`	in-memory `state.oracle_prices.ustc`	#515 USTC feed; `Option` → JSON `null`

Rollup refresh ~5 min (refresh_global_stats); whole response cached 60s (#281 / #333). Live fallback only if OVERVIEW_GLOBAL_STATS_LIVE=1 or rollup total_trades == 0 but recent swaps exist.

Why USD is 0 while raw volume and 4 trades are non-zero: ingest compute_volume_usd returns Some only when offer or ask is USTC (uusd / configured ustc_denom), using a hardcoded 1_000,000 decimals factor and the USTC oracle. Anything else (UST1/USTR, UST1/cLUNC, gem/UST1, …) → None. SUM of NULLs is 0. #522 already has P522-Q in pair_price_usd.rs (UST1=$1; USTC/cUSTC/uusd=#515 USTC; LUNC/cLUNC/uluna=#515 LUNC; USTR=2.5× USTC) for price_usd, but compute_volume_usd does not use it. Documented X4 still says volume_usd is USTC-only (docs/runbooks/indexer-external-oracle.md).

Worked example matching the screenshot: ~10 human USTR offered → raw 1e19 → UI 10,000,000T. USD should be 10 × 2.5 × ~$0.004878 ≈ $0.12, not 0.

Other overview numbers (audit targets)

Box	Current meaning	Risk
USTC / USD	In-memory USTC oracle at request time (not averaged into volume)	Missing → should be `—`, not `$0`. `formatNum(..., 6)` pads `$0.00487800`. Stale feed has no UI TTL.
24h Trades	`COUNT(*)` `swap_events` in the 24h window	Correct if L10 (one row per taker swap). Must not add `limit_order_fills`. Rollup lag ~5 min.
Pairs	`COUNT(*)` from `pairs`	Factory provenance (#311) already gates inserts. API `is_active` is hardcoded `true` — count is “indexed pairs”, not “pairs with liquidity”.
Tokens	`get_all_assets().len()`	Counts all `assets` rows (natives + CW20s ever upserted), not necessarily unique pair legs. LP tokens live on `pairs.lp_token` (usually not `assets`). Loading the full table to count is wasteful and a mild DoS footgun as the catalog grows.

Frontend types: IndexerOverview (total_volume_24h_usd? optional even though the API always sends a string).

Why this is needed

Retail cannot read Charts. 10,000,000T is not a market size; 0 USD next to 4 trades is a lie. Price (USD) on the same page is already oracle-valued (#522); volume still speaks mixed raw integers.
Two volume boxes fight. Product ask: USD only — drop the raw box. Integrators keep total_volume_24h on the JSON.
$0 vs missing vs dust. COALESCE(SUM(NULL),0) plus formatNum("0") cannot distinguish “no priced volume” from “no trades”. After catalog ingest, UST1/USTR (and other P522-Q legs) must contribute; remaining unknown quotes must not display as $0.
Other boxes can silently drift. Token count from a full table scan, pair count vs factory, trade count vs fills, USTC $0.000000 on a missing oracle — all need an explicit retail contract and tests.

Constraints / guardrails

USD is advisory, not settlement (X5 / P522). Label 24h Volume (USD implied) or 24h Volume (USD). Do not imply a peg guarantee. Tooltip ≤ one short sentence if needed (24h volume in USD).
Reuse P522-Q. Same quote_usd_kind / usd_per_human_quote as #544. One ingest implementation shared with #544 — do not fork the formula. Unknown quotes → volume_usd NULL, not $0, not a guessed price.
Human amounts before × USD. Divide by 10^decimals of the priced leg (USTR 18, UST1/cUSTC/cLUNC 6). Do not reuse hardcoded decimals_factor = 1_000_000.
One notional per swap (L10). Consolidated offer_amount/return_amount once. Prefer catalog-known oriented quote (human quote × quote USD). If only the other leg is in the catalog, use that leg. If both known, one side (quote preferred) — never sum both, never add fills/legs.
Classify by factory asset row (symbol and denom/contract), never by UI invert (#524).
Keep API field total_volume_24h. Additive JSON. Charts must not render it. Do not humanize it in the indexer (still raw SUM(offer_amount)).
Missing USD → —, not $0.00, not 0. If total_trades_24h > 0 but catalog USD is 0/empty (all legs unknown or oracle down), show —. True idle DEX (total_trades_24h == 0) may show $0 or — — pick one and test it ($0 only when there were no trades).
Display: $ + compact human USD (formatNum on a human number, 2–3 sigfigs). Never pass raw integers. Reject non-finite / negative. Do not compact-format the USTC spot as T (use a price formatter / formatPairPrice, keep $).
Rollup + cache unchanged (V5). No live 24h SUM(swap_events) on /overview in production. Backfill swap_events.volume_usd then refresh_global_stats (same as #544). Document ~5 min rollup + 60s response cache.
Do not change CG/CMC base_volume / target_volume, candle histogram units, #522 price/price_usd, hybrid columns, or trader leaderboard total_volume (follow-up if it still prints raw).
Pairs / tokens / trades semantics (retail):
- 24h Trades = 24h swap_events rows only (L10).
- Pairs = COUNT(*) indexed factory pairs (current SQL is OK if provenance holds).
- Tokens = unique assets that appear as pairs.asset_0_id or asset_1_id (not get_all_assets().len(), not LP tokens). Use COUNT SQL, do not load the table.
- USTC / USD = #515 USTC feed only (do not show LUNC here). Missing/non-finite → —. Do not use this box as volume.
Copy (#489). Short labels. After dropping the raw box, a single 24h Volume labeled as USD (or 24h Volume (USD)) is enough — do not keep both.
Docs + skill + make verify-issue-<iid> in the same MR. Update X4 together with #544 (catalog, not USTC-only). Overview row in docs/indexer-invariants.md. Charts overview in docs/frontend.md.
Out of scope: pair 24h stats strip (#540/#544); Trade/Pool picker badges (#544); candle volume bars; using USD volume in fee/settlement math.

Relevant files

Area	Path
Charts overview UI	`frontend-dapp/src/pages/ChartsPage.tsx`
Charts tests	`frontend-dapp/src/pages/ChartsPage.test.tsx`
Overview types / client	`frontend-dapp/src/types/index.ts`, `frontend-dapp/src/services/indexer/client.ts`
Compact / USD format	`frontend-dapp/src/utils/formatAmount.ts`
Overview API	`indexer/src/api/overview.rs`
Rollup	`indexer/src/db/queries/volume.rs`, `indexer/src/indexer/volume_aggregator.rs`
Ingest USD	`indexer/src/indexer/parser.rs` (`compute_volume_usd`)
Quote catalog	`indexer/src/indexer/pair_price_usd.rs`
Token count	`indexer/src/db/queries/assets.rs` (`get_all_assets`)
Overview tests	`indexer/tests/api_overview.rs`, `indexer/tests/indexer_overview_global_stats.rs`
Invariants / oracle	`docs/indexer-invariants.md`, `docs/runbooks/indexer-external-oracle.md`, `docs/runbooks/overview-global-stats-brin.md`
Skills	`skills/AGENTS_INDEXER_VOLUME_PAGINATION.md`, `skills/AGENTS_INDEXER_PAIR_PRICE_USD.md`, `skills/AGENTS_INDEXER_EXTERNAL_ORACLE.md`

Recommended direction

Indexer ingest (shared with #544): rewrite compute_volume_usd through P522-Q; humanize with per-asset decimals; backfill; refresh global_stats_24h. Assert a UST1/USTR swap produces non-NULL volume_usd ≈ human USTR × 2.5 × USTC.
Overview token_count: replace get_all_assets().len() with a COUNT(DISTINCT …) over pair legs. Keep pair_count as COUNT(*) FROM pairs. Keep total_trades_24h as swap-row count.
Overview JSON: keep total_volume_24h for integrators. Optionally add nothing new — Charts just stops reading it. If total_volume_24h_usd is "0" because of NULL sum, consider omitting / using JSON null when there is 24h trade activity but no priced USD so the UI can show — without guessing. Prefer an explicit null over "0" for “unpriced”.
Charts UI: remove the raw 24h Volume StatBox. Keep one USD volume box ($ + compact). — when unpriced. $0 (or —) only when total_trades_24h === 0. Prefix USTC with $; missing → —. Add data-testids: charts-overview-volume-usd, charts-overview-ustc-usd, charts-overview-trades, charts-overview-pairs, charts-overview-tokens. Do not leave a testid for the removed raw volume box.
Tests + make verify-issue-<iid>: Vitest overview strip; indexer overview + ingest USD; optional Playwright /charts with mocked overview.

If #544 lands the ingest/backfill first, this issue is frontend + token_count SQL + overview display contract + verify script. If this lands first, #544 must consume the same ingest helper.

Acceptance criteria

C1 /charts overview shows exactly one 24h volume control, in USD ($ + compact human). No raw 24h Volume box, no …T from total_volume_24h.
C2 With 24h swaps whose legs are in P522-Q (e.g. UST1/USTR, UST1/cUSTC), 24h Volume (USD) is a positive compact USD that matches GET /api/v1/overview total_volume_24h_usd (rollup lag ≤ ~5 min OK) — not 0.
C3 Unpriced 24h activity (unknown quote, oracle down) → volume shows —, not $0 / 0. Zero trades → documented $0 or —.
C4 USTC / USD = $ + human USTC spot from overview ustc_price_usd; null/invalid → —; never LUNC; never T.
C5 24h Trades = total_trades_24h = 24h swap_events count (not fills, not LP events).
C6 Pairs = indexed pair count; Tokens = unique pair-leg assets (SQL COUNT, not full table load). Fixture: N pairs / M distinct legs → Tokens = M.
C7 API still returns total_volume_24h (raw) for integrators; Charts does not display it.
C8 Ingest uses P522-Q once (shared with #544); X4 / invariants / frontend docs / skill updated; make verify-issue-<iid> exists and is listed in the Makefile help.
C9 Outage banner (#215) still hides or skeletons the strip when overview+pairs fail; no VITE_INDEXER_URL leak.

Test plan (all paths)

Indexer

ID	Case	Expect
I1	UST1/USTR swap (18/6)	`volume_usd` ≈ human USTR × 2.5 × USTC (or human UST1 × $1); overview SUM includes it after refresh
I2	UST1/cUSTC swap	`volume_usd` ≈ human cUSTC × USTC (or UST1 × $1); one side, not both
I3	USTC-leg swap (legacy)	Still priced; not double-counted vs I1/I2
I4	LUNC/cLUNC-quoted swap	Uses LUNC oracle, not USTC
I5	Unknown gem quote, no catalog base	`volume_usd` NULL; overview USD does not invent a number
I6	Hybrid pool+book one swap	One `volume_usd`; fills not added (L10)
I7	`refresh_global_stats` vs live	`total_volume_24h_usd` / `total_trades_24h` match live after refresh
I8	Rollup zeros + recent swaps	Existing live fallback still works; USD included when priced
I9	`token_count`	Equals distinct pair-leg assets; extra orphan `assets` row does not increment; LP address on `pairs.lp_token` does not increment
I10	`pair_count`	Equals `COUNT(*) FROM pairs`
I11	`ustc_price_usd`	Present when oracle cache set; `null` when unset; never LUNC value
I12	`/overview` 60s cache	Back-to-back identical JSON (existing #281 test)
I13	Backfill idempotent	Second run does not double USD
I14	Overview does not `SUM(swap_events)` on the hot path	Rollup PK read (existing EXPLAIN test)

Frontend (Vitest)

ID	Case	Expect
F1	Overview with `total_volume_24h: '10000000000000000000'`, `total_volume_24h_usd: '1234.56'`, trades 4	No `10,000,000T` / raw volume label; one USD box `~$1.235K` (or `$1,235`); `$` present; no second volume box
F2	`total_volume_24h_usd: '0'` or `'0.00'` and `total_trades_24h > 0`	Volume `—`, not `0`
F3	Both volume fields `'0'`, `total_trades_24h: 0`	Documented empty (`$0` or `—`)
F4	`total_volume_24h_usd` missing/null/empty	`—`
F5	`ustc_price_usd: '0.004878'`	`$` + human (no `T`); `null` → `—`; `''` → `—`
F6	`pair_count` / `token_count` / `total_trades_24h`	Exact locale strings; no compact `T`
F7	Loading	Skeletons; no flash of `0`
F8	Overview+pairs 502	Existing #215 outage banner; no indexer URL
F9	Negative / `NaN` / `1e309` / HTML string in USD field	`—`; no exception; no HTML inject
F10	Grid	Five (or fewer) overview boxes; no empty slot labeled 24h Volume for raw

Manual / LocalTerra / mainnet

ID	Case	Expect
M1	Open `/charts` on dex.cl8y.com (or LocalTerra with UST1/USTR + UST1/cUSTC flow)	Overview USD compact matches `curl $INDEXER/api/v1/overview` `total_volume_24h_usd`; not `0` when trades exist and legs are catalog-priced
M2	Compare 24h Trades to indexer / DB 24h swap count	Equal (within rollup lag)
M3	Pairs vs `/api/v1/pairs?limit=1` `total`	Equal
M4	Tokens vs distinct symbols on pair list	Equal unique pair-leg tokens
M5	USTC / USD vs `GET /api/v1/oracle/price/ustc`	Same order of magnitude; `$` prefix
M6	Narrow / `lg` layout	Strip readable; no overlapping boxes

Test plan (attack, hack, abuse)

Vector	Expectation
A1 Symbol spoof — hostile CW20 `symbol=UST1`/`USTR` not in factory catalog	Must not price as hub; `volume_usd` NULL; overview USD not inflated. Prefer denom/contract allowlist (same as #544 A1).
A2 Double notional — hybrid + fill rows	Overview USD = sum of swap `volume_usd` only. Adding fills must fail the test.
A3 Wash volume — many tiny catalog-priced swaps	Count and USD both rise (same wash surface as today). Do not trust client-supplied overview. Values only from indexer JSON.
A4 JSON injection — `total_volume_24h_usd` = `"><script>` / huge string	Parse as decimal; invalid → `—`. No `dangerouslySetInnerHTML`. Cap display length.
A5 Overflow / tab lock — 38-digit raw still in `total_volume_24h`	Charts must not pass it to `formatNum`. Even if API sends it, unused.
A6 Unit confusion — UI ×1e6 or × `price_usd` again on already-human USD	Tests: overview USD is human dollars; display does not rescale.
A7 Oracle / peg game — thin USTR print, 2.5× USTC, stale cache	Advisory only; stale/missing → NULL/`—`, not last-good forever without documented TTL (#515). Do not use overview USD for settlement/fees.
A8 Cache poison — 60s overview cache serves another tenant’s totals	Cache is process-global public stats (OK). Do not key cache on attacker-controlled query params (overview has none).
A9 token_count DoS — millions of `assets` rows	`COUNT` SQL / pair-leg distinct; no `get_all_assets` full fetch on `/overview`.
A10 Pair-count inflation — unlisted clone emitting swaps	Provenance (#311) still blocks pair insert; overview pair_count must not include skipped clones.
A11 Invert / wrong feed — UI invert or LUNC ticker shown as USTC/USD	USTC box stays USTC. Volume USD does not flip with #524 invert (global, not pair).
A12 Negative / dust — `volume_usd = -1` or `1e-18`	Hide / `—` / clamp; never negative compact.
A13 Replay backfill	Idempotent; overview USD does not double.
A14 Integrator break — removing `total_volume_24h` from JSON	Field remains; only UI drops it. CG/CMC units unchanged.

Verification criteria

Issue is done when:

A reviewer on mainnet or LocalTerra opens /charts and sees one 24h volume figure in USD, matching GET /api/v1/overview total_volume_24h_usd, not 0 when catalog-priced 24h swaps exist, and not 10,000,000T.
USTC / USD, 24h Trades, Pairs, and Tokens match the semantics in C4–C6 (spot-check against oracle + pair list + DB counts).
make verify-issue-<iid> and listed indexer/frontend tests pass in CI.
Docs/skills: Charts overview is USD-only; total_volume_24h remains raw for API clients; X4 catalog (shared with #544); L10 / CG units unchanged; token_count = pair-leg distinct.
Abuse cases A1, A2, A4, A5, A6, A9, A14 have automated coverage; A3/A7/A8/A10–A13 are tested or explicitly waived in the MR with a reason.

Out of scope: Trade/Pool pair-search USD badges (#544); Charts pair stats Vol (token) formatting (#540/#544); candle histogram; trader leaderboard raw volume; inventing USD for faucet gems; settlement/fee math.

## Summary On **Charts & Analytics** (`/charts`) the overview strip still shows a meaningless raw **24h Volume** (`10,000,000T`) next to **24h Volume (USD) = 0**, even when 24h trades exist. Retail should see **one** 24h volume figure, in **USD**, with a correct (non-zero when catalog-priced volume exists) number. Also **audit** the other overview boxes (USTC/USD, 24h Trades, Pairs, Tokens) so they match documented indexer semantics. **Observed (columbus-5 / dex.cl8y.com):** | Box | Shown | Problem | |-----|--------|---------| | 24h Volume | `10,000,000T` | `formatNum(total_volume_24h)` on mixed raw `SUM(offer_amount)` — same `T` class as [#534](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/534) / [#544](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/544) | | 24h Volume (USD) | `0` (no `$`) | Ingest `volume_usd` is USTC-leg only (**X4**); `COALESCE(SUM(NULL),0)` → `"0"`; UI treats `"0"` as a real value | | USTC / USD | `$0.00487800` | Plausible oracle print — confirm source, empty/stale, formatting | | 24h Trades | `4` | Confirm = 24h `swap_events` count (**L10**, not fills) | | Pairs | `13` | Confirm = indexed factory pairs | | Tokens | `12` | Confirm = unique pair-leg tokens, not every `assets` row | Related (do **not** treat as done): [#544](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/544) pair-list / picker USD volume + shared ingest catalog; [#540](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/540) pair **stats** strip still `formatNum(raw)`; [#522](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/522) P522-Q quote USD catalog; [#515](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/515) oracles; [#281](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/281) / [#333](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/333) overview rollup + 60s cache. **This issue uniquely owns the `/charts` overview strip.** Pair-search / `/pool` badges stay on #544. Pair-level 24h stats (`Vol (AAA)` / `Vol (BBB)`) stay on #540 / #544. Share one `compute_volume_usd` catalog change — do not invent a second USD formula. ## Current codebase ### Frontend — two volume boxes, raw first [`frontend-dapp/src/pages/ChartsPage.tsx`](frontend-dapp/src/pages/ChartsPage.tsx) loads `GET /api/v1/overview` via `getOverview` and renders six `StatBox`es: ```253:290:frontend-dapp/src/pages/ChartsPage.tsx <StatBox label="24h Volume" value={overview ? formatNum(overview.total_volume_24h) : '—'} loading={overviewQuery.isLoading} /> <StatBox label="24h Volume (USD)" value={ overview?.total_volume_24h_usd != null && overview.total_volume_24h_usd !== '' ? formatNum(overview.total_volume_24h_usd, 2) : '—' } loading={overviewQuery.isLoading} /> <StatBox label="USTC / USD" value={ overview?.ustc_price_usd != null && overview.ustc_price_usd !== '' ? `$${formatNum(overview.ustc_price_usd, 6)}` : '—' } ... /> <StatBox label="24h Trades" value={overview ? overview.total_trades_24h.toLocaleString() : '—'} /> <StatBox label="Pairs" value={overview ? overview.pair_count.toString() : '—'} /> <StatBox label="Tokens" value={overview ? overview.token_count.toString() : '—'} /> ``` - **`total_volume_24h`** is a **raw mixed-decimal integer** (`SUM(offer_amount)` across all offer assets). Passing it to [`formatNum`](frontend-dapp/src/utils/formatAmount.ts) compact-formats ≥1e12 as `T`. Ten human USTR (18 decimals) is `1e19` raw → **`10,000,000T`**. There is no unit on the label. - **`total_volume_24h_usd`**: `formatNum(x, 2)` with **no `$`**. The empty check allows `"0"` / `"0.0"` through, so a true zero *and* a missing-USD rollup both render **`0`**. Contrast USTC/USD, which prefixes `$`. - [`ChartsPage.test.tsx`](frontend-dapp/src/pages/ChartsPage.test.tsx) mocks both volumes as `'0'` and never asserts the overview strip labels or values. - Grid is `lg:grid-cols-6`; dropping one box should reflow (5 boxes). ### Indexer — overview JSON + USTC-only `volume_usd` [`GET /api/v1/overview`](indexer/src/api/overview.rs) (`OverviewResponse`): | Field | Source | Notes | |-------|--------|--------| | `total_volume_24h` | `global_stats_24h.total_volume` | `SUM(offer_amount)` raw — **not** human, **not** USD | | `total_volume_24h_usd` | `global_stats_24h.total_volume_usd` | `COALESCE(SUM(volume_usd), 0)` | | `total_trades_24h` | `global_stats_24h.total_trades` | `COUNT(*)` of `swap_events` in 24h | | `pair_count` | live `SELECT COUNT(*) FROM pairs` | cache-miss only | | `token_count` | `assets::get_all_assets(&pool)?.len()` | loads **every** `assets` row into memory just to count | | `ustc_price_usd` | in-memory `state.oracle_prices.ustc` | #515 USTC feed; `Option` → JSON `null` | Rollup refresh ~5 min ([`refresh_global_stats`](indexer/src/db/queries/volume.rs)); whole response cached **60s** ([#281](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/281) / [#333](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/work_items/333)). Live fallback only if `OVERVIEW_GLOBAL_STATS_LIVE=1` **or** rollup `total_trades == 0` but recent swaps exist. **Why USD is 0 while raw volume and 4 trades are non-zero:** ingest [`compute_volume_usd`](indexer/src/indexer/parser.rs) returns `Some` only when **offer or ask is USTC** (`uusd` / configured `ustc_denom`), using a **hardcoded `1_000,000` decimals factor** and the USTC oracle. Anything else (UST1/USTR, UST1/cLUNC, gem/UST1, …) → `None`. `SUM` of NULLs is 0. [#522](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/522) already has **P522-Q** in [`pair_price_usd.rs`](indexer/src/indexer/pair_price_usd.rs) (UST1=`$1`; USTC/cUSTC/`uusd`=#515 USTC; LUNC/cLUNC/`uluna`=#515 LUNC; USTR=`2.5×` USTC) for **`price_usd`**, but **`compute_volume_usd` does not use it**. Documented **X4** still says volume_usd is USTC-only ([`docs/runbooks/indexer-external-oracle.md`](docs/runbooks/indexer-external-oracle.md)). Worked example matching the screenshot: ~10 human USTR offered → raw `1e19` → UI `10,000,000T`. USD should be `10 × 2.5 × ~$0.004878 ≈ $0.12`, not `0`. ### Other overview numbers (audit targets) | Box | Current meaning | Risk | |-----|-----------------|------| | **USTC / USD** | In-memory USTC oracle at request time (not averaged into volume) | Missing → should be `—`, not `$0`. `formatNum(..., 6)` pads `$0.00487800`. Stale feed has no UI TTL. | | **24h Trades** | `COUNT(*)` `swap_events` in the 24h window | Correct if **L10** (one row per taker swap). Must **not** add `limit_order_fills`. Rollup lag ~5 min. | | **Pairs** | `COUNT(*)` from `pairs` | Factory provenance ([#311](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/311)) already gates inserts. API `is_active` is hardcoded `true` — count is “indexed pairs”, not “pairs with liquidity”. | | **Tokens** | `get_all_assets().len()` | Counts **all** `assets` rows (natives + CW20s ever upserted), not necessarily unique pair legs. LP tokens live on `pairs.lp_token` (usually not `assets`). Loading the full table to count is wasteful and a mild DoS footgun as the catalog grows. | Frontend types: [`IndexerOverview`](frontend-dapp/src/types/index.ts) (`total_volume_24h_usd?` optional even though the API always sends a string). ## Why this is needed 1. **Retail cannot read Charts.** `10,000,000T` is not a market size; **`0` USD** next to **4 trades** is a lie. Price (USD) on the same page is already oracle-valued ([#522](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/522)); volume still speaks mixed raw integers. 2. **Two volume boxes fight.** Product ask: **USD only** — drop the raw box. Integrators keep `total_volume_24h` on the JSON. 3. **`$0` vs missing vs dust.** `COALESCE(SUM(NULL),0)` plus `formatNum("0")` cannot distinguish “no priced volume” from “no trades”. After catalog ingest, UST1/USTR (and other P522-Q legs) must contribute; remaining unknown quotes must not display as `$0`. 4. **Other boxes can silently drift.** Token count from a full table scan, pair count vs factory, trade count vs fills, USTC `$0.000000` on a missing oracle — all need an explicit retail contract and tests. ## Constraints / guardrails 1. **USD is advisory, not settlement** (**X5** / **P522**). Label **24h Volume** (USD implied) or **24h Volume (USD)**. Do not imply a peg guarantee. Tooltip ≤ one short sentence if needed (`24h volume in USD`). 2. **Reuse P522-Q.** Same `quote_usd_kind` / `usd_per_human_quote` as #544. **One ingest implementation** shared with #544 — do not fork the formula. Unknown quotes → `volume_usd` NULL, not `$0`, not a guessed price. 3. **Human amounts before × USD.** Divide by `10^decimals` of the priced leg (USTR 18, UST1/cUSTC/cLUNC 6). Do **not** reuse hardcoded `decimals_factor = 1_000_000`. 4. **One notional per swap (L10).** Consolidated `offer_amount`/`return_amount` once. Prefer catalog-known **oriented quote** (human quote × quote USD). If only the other leg is in the catalog, use that leg. If both known, **one** side (quote preferred) — never sum both, never add fills/legs. 5. **Classify by factory asset row** (symbol **and** denom/contract), never by UI invert ([#524](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/524)). 6. **Keep API field `total_volume_24h`.** Additive JSON. Charts **must not render it**. Do not humanize it in the indexer (still raw `SUM(offer_amount)`). 7. **Missing USD → `—`, not `$0.00`, not `0`.** If `total_trades_24h > 0` but catalog USD is 0/empty (all legs unknown **or** oracle down), show **—**. True idle DEX (`total_trades_24h == 0`) may show `$0` or `—` — pick one and test it (`$0` only when there were no trades). 8. **Display:** `$` + compact human USD (`formatNum` on a **human** number, 2–3 sigfigs). Never pass raw integers. Reject non-finite / negative. Do not compact-format the USTC spot as `T` (use a price formatter / `formatPairPrice`, keep `$`). 9. **Rollup + cache unchanged (V5).** No live 24h `SUM(swap_events)` on `/overview` in production. Backfill `swap_events.volume_usd` then `refresh_global_stats` (same as #544). Document ~5 min rollup + 60s response cache. 10. **Do not change** CG/CMC `base_volume` / `target_volume`, candle histogram units, #522 `price`/`price_usd`, hybrid columns, or trader leaderboard `total_volume` (follow-up if it still prints raw). 11. **Pairs / tokens / trades semantics (retail):** - **24h Trades** = 24h `swap_events` rows only (L10). - **Pairs** = `COUNT(*)` indexed factory pairs (current SQL is OK if provenance holds). - **Tokens** = **unique assets that appear as `pairs.asset_0_id` or `asset_1_id`** (not `get_all_assets().len()`, not LP tokens). Use `COUNT` SQL, do not load the table. - **USTC / USD** = #515 USTC feed only (do **not** show LUNC here). Missing/non-finite → `—`. Do not use this box as volume. 12. **Copy (#489).** Short labels. After dropping the raw box, a single **24h Volume** labeled as USD (or **24h Volume (USD)**) is enough — do not keep both. 13. **Docs + skill + `make verify-issue-<iid>`** in the same MR. Update **X4** together with #544 (catalog, not USTC-only). Overview row in [`docs/indexer-invariants.md`](docs/indexer-invariants.md). Charts overview in [`docs/frontend.md`](docs/frontend.md). 14. **Out of scope:** pair 24h stats strip (#540/#544); Trade/Pool picker badges (#544); candle volume bars; using USD volume in fee/settlement math. ## Relevant files | Area | Path | |------|------| | Charts overview UI | `frontend-dapp/src/pages/ChartsPage.tsx` | | Charts tests | `frontend-dapp/src/pages/ChartsPage.test.tsx` | | Overview types / client | `frontend-dapp/src/types/index.ts`, `frontend-dapp/src/services/indexer/client.ts` | | Compact / USD format | `frontend-dapp/src/utils/formatAmount.ts` | | Overview API | `indexer/src/api/overview.rs` | | Rollup | `indexer/src/db/queries/volume.rs`, `indexer/src/indexer/volume_aggregator.rs` | | Ingest USD | `indexer/src/indexer/parser.rs` (`compute_volume_usd`) | | Quote catalog | `indexer/src/indexer/pair_price_usd.rs` | | Token count | `indexer/src/db/queries/assets.rs` (`get_all_assets`) | | Overview tests | `indexer/tests/api_overview.rs`, `indexer/tests/indexer_overview_global_stats.rs` | | Invariants / oracle | `docs/indexer-invariants.md`, `docs/runbooks/indexer-external-oracle.md`, `docs/runbooks/overview-global-stats-brin.md` | | Skills | `skills/AGENTS_INDEXER_VOLUME_PAGINATION.md`, `skills/AGENTS_INDEXER_PAIR_PRICE_USD.md`, `skills/AGENTS_INDEXER_EXTERNAL_ORACLE.md` | ## Recommended direction 1. **Indexer ingest (shared with #544):** rewrite `compute_volume_usd` through P522-Q; humanize with per-asset decimals; backfill; refresh `global_stats_24h`. Assert a UST1/USTR swap produces non-NULL `volume_usd` ≈ human USTR × 2.5 × USTC. 2. **Overview `token_count`:** replace `get_all_assets().len()` with a `COUNT(DISTINCT …)` over pair legs. Keep `pair_count` as `COUNT(*) FROM pairs`. Keep `total_trades_24h` as swap-row count. 3. **Overview JSON:** keep `total_volume_24h` for integrators. Optionally add nothing new — Charts just stops reading it. If `total_volume_24h_usd` is `"0"` because of NULL sum, consider omitting / using JSON `null` when there is 24h trade activity but no priced USD so the UI can show `—` without guessing. Prefer an explicit `null` over `"0"` for “unpriced”. 4. **Charts UI:** remove the raw **24h Volume** `StatBox`. Keep one USD volume box (`$` + compact). `—` when unpriced. `$0` (or `—`) only when `total_trades_24h === 0`. Prefix USTC with `$`; missing → `—`. Add `data-testid`s: `charts-overview-volume-usd`, `charts-overview-ustc-usd`, `charts-overview-trades`, `charts-overview-pairs`, `charts-overview-tokens`. Do **not** leave a testid for the removed raw volume box. 5. **Tests + `make verify-issue-<iid>`:** Vitest overview strip; indexer overview + ingest USD; optional Playwright `/charts` with mocked overview. If #544 lands the ingest/backfill first, this issue is **frontend + token_count SQL + overview display contract + verify script**. If this lands first, #544 must consume the same ingest helper. ## Acceptance criteria - [ ] **C1** `/charts` overview shows **exactly one** 24h volume control, in **USD** (`$` + compact human). No raw `24h Volume` box, no `…T` from `total_volume_24h`. - [ ] **C2** With 24h swaps whose legs are in P522-Q (e.g. UST1/USTR, UST1/cUSTC), **24h Volume (USD)** is a **positive** compact USD that matches `GET /api/v1/overview` `total_volume_24h_usd` (rollup lag ≤ ~5 min OK) — **not** `0`. - [ ] **C3** Unpriced 24h activity (unknown quote, oracle down) → volume shows **`—`**, not `$0` / `0`. Zero trades → documented `$0` or `—`. - [ ] **C4** **USTC / USD** = `$` + human USTC spot from overview `ustc_price_usd`; null/invalid → `—`; never LUNC; never `T`. - [ ] **C5** **24h Trades** = `total_trades_24h` = 24h `swap_events` count (not fills, not LP events). - [ ] **C6** **Pairs** = indexed pair count; **Tokens** = unique pair-leg assets (SQL `COUNT`, not full table load). Fixture: N pairs / M distinct legs → Tokens = M. - [ ] **C7** API still returns `total_volume_24h` (raw) for integrators; Charts does not display it. - [ ] **C8** Ingest uses P522-Q once (shared with #544); **X4** / invariants / frontend docs / skill updated; `make verify-issue-<iid>` exists and is listed in the Makefile help. - [ ] **C9** Outage banner ([#215](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/215)) still hides or skeletons the strip when overview+pairs fail; no `VITE_INDEXER_URL` leak. ## Test plan (all paths) ### Indexer | ID | Case | Expect | |----|------|--------| | I1 | UST1/USTR swap (18/6) | `volume_usd` ≈ human USTR × 2.5 × USTC (or human UST1 × $1); overview SUM includes it after refresh | | I2 | UST1/cUSTC swap | `volume_usd` ≈ human cUSTC × USTC (or UST1 × $1); **one** side, not both | | I3 | USTC-leg swap (legacy) | Still priced; not double-counted vs I1/I2 | | I4 | LUNC/cLUNC-quoted swap | Uses LUNC oracle, not USTC | | I5 | Unknown gem quote, no catalog base | `volume_usd` NULL; overview USD does not invent a number | | I6 | Hybrid pool+book one swap | One `volume_usd`; fills not added (**L10**) | | I7 | `refresh_global_stats` vs live | `total_volume_24h_usd` / `total_trades_24h` match live after refresh | | I8 | Rollup zeros + recent swaps | Existing live fallback still works; USD included when priced | | I9 | `token_count` | Equals distinct pair-leg assets; extra orphan `assets` row does **not** increment; LP address on `pairs.lp_token` does **not** increment | | I10 | `pair_count` | Equals `COUNT(*) FROM pairs` | | I11 | `ustc_price_usd` | Present when oracle cache set; `null` when unset; never LUNC value | | I12 | `/overview` 60s cache | Back-to-back identical JSON (existing #281 test) | | I13 | Backfill idempotent | Second run does not double USD | | I14 | Overview does not `SUM(swap_events)` on the hot path | Rollup PK read (existing EXPLAIN test) | ### Frontend (Vitest) | ID | Case | Expect | |----|------|--------| | F1 | Overview with `total_volume_24h: '10000000000000000000'`, `total_volume_24h_usd: '1234.56'`, trades 4 | **No** `10,000,000T` / raw volume label; **one** USD box `~$1.235K` (or `$1,235`); `$` present; no second volume box | | F2 | `total_volume_24h_usd: '0'` or `'0.00'` **and** `total_trades_24h > 0` | Volume **`—`**, not `0` | | F3 | Both volume fields `'0'`, `total_trades_24h: 0` | Documented empty (`$0` or `—`) | | F4 | `total_volume_24h_usd` missing/null/empty | `—` | | F5 | `ustc_price_usd: '0.004878'` | `$` + human (no `T`); `null` → `—`; `''` → `—` | | F6 | `pair_count` / `token_count` / `total_trades_24h` | Exact locale strings; no compact `T` | | F7 | Loading | Skeletons; no flash of `0` | | F8 | Overview+pairs 502 | Existing #215 outage banner; no indexer URL | | F9 | Negative / `NaN` / `1e309` / HTML string in USD field | `—`; no exception; no HTML inject | | F10 | Grid | Five (or fewer) overview boxes; no empty slot labeled 24h Volume for raw | ### Manual / LocalTerra / mainnet | ID | Case | Expect | |----|------|--------| | M1 | Open `/charts` on dex.cl8y.com (or LocalTerra with UST1/USTR + UST1/cUSTC flow) | Overview USD compact matches `curl $INDEXER/api/v1/overview` `total_volume_24h_usd`; not `0` when trades exist and legs are catalog-priced | | M2 | Compare **24h Trades** to indexer / DB 24h swap count | Equal (within rollup lag) | | M3 | **Pairs** vs `/api/v1/pairs?limit=1` `total` | Equal | | M4 | **Tokens** vs distinct symbols on pair list | Equal unique pair-leg tokens | | M5 | **USTC / USD** vs `GET /api/v1/oracle/price/ustc` | Same order of magnitude; `$` prefix | | M6 | Narrow / `lg` layout | Strip readable; no overlapping boxes | ## Test plan (attack, hack, abuse) | Vector | Expectation | |--------|-------------| | **A1 Symbol spoof** — hostile CW20 `symbol=UST1`/`USTR` not in factory catalog | Must not price as hub; `volume_usd` NULL; overview USD not inflated. Prefer denom/contract allowlist (same as #544 A1). | | **A2 Double notional** — hybrid + fill rows | Overview USD = sum of **swap** `volume_usd` only. Adding fills must fail the test. | | **A3 Wash volume** — many tiny catalog-priced swaps | Count and USD both rise (same wash surface as today). Do not trust client-supplied overview. Values only from indexer JSON. | | **A4 JSON injection** — `total_volume_24h_usd` = `"><script>` / huge string | Parse as decimal; invalid → `—`. No `dangerouslySetInnerHTML`. Cap display length. | | **A5 Overflow / tab lock** — 38-digit raw still in `total_volume_24h` | Charts **must not** pass it to `formatNum`. Even if API sends it, unused. | | **A6 Unit confusion** — UI ×1e6 or × `price_usd` again on already-human USD | Tests: overview USD is human dollars; display does not rescale. | | **A7 Oracle / peg game** — thin USTR print, 2.5× USTC, stale cache | Advisory only; stale/missing → NULL/`—`, not last-good forever without documented TTL (#515). Do not use overview USD for settlement/fees. | | **A8 Cache poison** — 60s overview cache serves another tenant’s totals | Cache is process-global public stats (OK). Do not key cache on attacker-controlled query params (overview has none). | | **A9 token_count DoS** — millions of `assets` rows | `COUNT` SQL / pair-leg distinct; **no** `get_all_assets` full fetch on `/overview`. | | **A10 Pair-count inflation** — unlisted clone emitting swaps | Provenance (#311) still blocks pair insert; overview pair_count must not include skipped clones. | | **A11 Invert / wrong feed** — UI invert or LUNC ticker shown as USTC/USD | USTC box stays USTC. Volume USD does not flip with #524 invert (global, not pair). | | **A12 Negative / dust** — `volume_usd = -1` or `1e-18` | Hide / `—` / clamp; never negative compact. | | **A13 Replay backfill** | Idempotent; overview USD does not double. | | **A14 Integrator break** — removing `total_volume_24h` from JSON | Field remains; only UI drops it. CG/CMC units unchanged. | ## Verification criteria Issue is **done** when: 1. A reviewer on mainnet or LocalTerra opens `/charts` and sees **one** 24h volume figure in **USD**, matching `GET /api/v1/overview` `total_volume_24h_usd`, **not** `0` when catalog-priced 24h swaps exist, and **not** `10,000,000T`. 2. **USTC / USD**, **24h Trades**, **Pairs**, and **Tokens** match the semantics in C4–C6 (spot-check against oracle + pair list + DB counts). 3. `make verify-issue-<iid>` and listed indexer/frontend tests pass in CI. 4. Docs/skills: Charts overview is USD-only; `total_volume_24h` remains raw for API clients; **X4** catalog (shared with #544); **L10** / CG units unchanged; token_count = pair-leg distinct. 5. Abuse cases A1, A2, A4, A5, A6, A9, A14 have automated coverage; A3/A7/A8/A10–A13 are tested or explicitly waived in the MR with a reason. **Out of scope:** Trade/Pool pair-search USD badges (#544); Charts pair **stats** `Vol (token)` formatting (#540/#544); candle histogram; trader leaderboard raw volume; inventing USD for faucet gems; settlement/fee math.

PlasticDigits commented

2026-08-17 10:29:10 +00:00

(Migrated from gitlab.com)

marked as related to #544

PlasticDigits commented

2026-08-17 10:29:11 +00:00

(Migrated from gitlab.com)

marked as related to #540

PlasticDigits commented

2026-08-17 10:29:12 +00:00

(Migrated from gitlab.com)

marked as related to #522

PlasticDigits commented

2026-08-17 10:29:13 +00:00

(Migrated from gitlab.com)

marked as related to #515

PlasticDigits commented

2026-08-17 10:29:14 +00:00

(Migrated from gitlab.com)

marked as related to #281

PlasticDigits commented

2026-08-17 10:29:15 +00:00

(Migrated from gitlab.com)

marked as related to #333

PlasticDigits commented

2026-08-17 11:57:44 +00:00

(Migrated from gitlab.com)

mentioned in commit 7393dfed7a

mentioned in commit 7393dfed7af86f6ebc1b79376969d7d7cc50ac67

PlasticDigits commented

2026-08-17 11:58:00 +00:00

(Migrated from gitlab.com)

mentioned in merge request !1076

PlasticDigits commented

2026-08-17 13:32:22 +00:00

(Migrated from gitlab.com)

mentioned in commit 11b1117eda

mentioned in commit 11b1117eda685a7eddcf2a9b81e5e381a31bf1c3

PlasticDigits (Migrated from gitlab.com) closed this issue

2026-08-17 13:32:22 +00:00

PlasticDigits commented

2026-08-17 13:39:57 +00:00

(Migrated from gitlab.com)

mentioned in commit 0536b093eb

mentioned in commit 0536b093ebddb9531371828ef647c4bbc34e002f

PlasticDigits commented

2026-08-17 13:46:07 +00:00

(Migrated from gitlab.com)

mentioned in commit 81b403d55e

mentioned in commit 81b403d55ee8e08f8249e4d109e364a3a0415c52

PlasticDigits commented

2026-08-17 13:48:58 +00:00

(Migrated from gitlab.com)

Merged as !1076 onto main (11b1117e). Follow-up merge !1078 kept this issue’s P522-Q catalog volume_usd, nullable total_volume_24h_usd, and pair-leg token_count (did not revert to USTC-only X4).

Remaining / post-merge:

Deploy: apply indexer/migrations/20260817120000_backfill_swap_volume_usd_catalog.sql (and the later #550 census migration). After backfill, wait for refresh_global_stats (~5 min) plus the 60s overview cache, then confirm /charts 24h USD matches GET /api/v1/overview total_volume_24h_usd.
C9: outage banner coverage exists; there is still no unit test that the overview strip is absent when overview+pairs both fail.
Manual QA from the issue (USTC box not T notation; pair-search USD badges stay #544) is still open.
Charts H2 deep-link RTL logs Query data cannot be undefined for indexer-pair-one (test still passes). Harmless noise, not a product miss.

Tracking: new post-merge issue for deploy + remaining manual QA.

Merged as !1076 onto `main` (`11b1117e`). Follow-up merge !1078 kept this issue’s P522-Q catalog `volume_usd`, nullable `total_volume_24h_usd`, and pair-leg `token_count` (did not revert to USTC-only X4). Remaining / post-merge: - **Deploy:** apply `indexer/migrations/20260817120000_backfill_swap_volume_usd_catalog.sql` (and the later #550 census migration). After backfill, wait for `refresh_global_stats` (~5 min) plus the 60s overview cache, then confirm `/charts` 24h USD matches `GET /api/v1/overview` `total_volume_24h_usd`. - **C9:** outage banner coverage exists; there is still no unit test that the overview strip is absent when overview+pairs both fail. - **Manual QA** from the issue (USTC box not `T` notation; pair-search USD badges stay #544) is still open. - Charts `H2` deep-link RTL logs `Query data cannot be undefined` for `indexer-pair-one` (test still passes). Harmless noise, not a product miss. Tracking: new post-merge issue for deploy + remaining manual QA.

PlasticDigits commented

2026-08-17 13:49:00 +00:00

(Migrated from gitlab.com)

mentioned in issue #547