fix(indexer/listings): reconcile DeFiLlama volume, fees, and TVL with live data #1354

New issue

Open

opened 2026-09-25 05:35:05 +00:00 by PlasticDigits · 0 comments

PlasticDigits commented

2026-09-25 05:35:05 +00:00

Owner

Summary

On 2026-09-25 around 05:25–05:35 UTC, CL8Y DEX's DeFiLlama volume, fee history, and TVL diverged from the live indexer and pool inventory for different reasons. This is a reconciliation issue; keep the already completed listing/adapter tickets #631 and #687 closed.

The three current causes are: (1) one unpriced swap makes an entire UTC day's otherwise priced volume JSON null, and the upstream DEX adapter throws; (2) historical fee corrections do not propagate through both the eight-day indexer Llama rollup and Llama's stored daily chart; (3) Llama prices only LUNC and USTC in the TVL adapter's reserve basket, omitting material CW20 pool legs. There are also missing fee sources and a pair-scoping risk described below.

Reproducible evidence

Compare completed UTC days 2026-08-26 through 2026-09-24, inclusive for flows. Do not compare a UTC-day flow with /overview trailing 24h/30d.

Metric	DeFiLlama stored value	Current indexer value	Observation
DEX volume, 30 completed UTC days	$31,166	$53,177.23 known priced lower bound	Ten of 30 daily Llama responses are `null`; their already priced swaps sum to $22,011.24 in the indexer's UTC protocol series. That accounts for the gap (rounding aside).
Fees, same days	$746.87	$816.74 sum of current `/defillama/daily`; $897.47 in `/protocol/fees/daily`	Two gaps: $69.87 between Llama's stored chart and its current feed; $80.73 between the current feed and the current broader fee series. The latter needs source-level DB reconciliation before attributing every dollar.
TVL, 2026-09-25 snapshot	$84,722.46 at 03:53 UTC	`/overview` around $157–160k about 1–2 hours later	These use different USD methodologies and are not directly equal. Llama's token breakdown contains only LUNC $28,357.89 and USTC $56,364.57; it omits all other CW20 pool legs.

Public evidence: Llama DEX volume, Llama fees, Llama TVL with tokensInUsd, indexer UTC volume series, indexer UTC fee series, indexer overview, 2026-09-24 Llama feed. For the exact 30-day volume window, add the 2026-08-26 daily feed to the protocol series' 2026-08-27–09-24 points and exclude its in-progress 09-25 point.

1. Volume: a single unpriced hop erases the day's priced total

The 2026-09-24 daily response has trade_count: 28 and volume_usd: null. The same UTC day in the protocol series has $558.461005 of priced swaps and 28 trades. A CL8Y-cb/ALPHA trade at 16:56:49 UTC, tx 37A8F6FAB565F532382C1EA1EC349624BB1D0E2316409AF95EBAB53EF294EAAF, has no USD price; it is one hop in a route through UST1/ALPHA. Both CL8Y and ALPHA are outside the swap-volume USD catalog for that pair. The source query in indexer/src/db/queries/defillama.rs stores a positive priced SUM(volume_usd) and unpriced_trade_count; indexer/src/api/defillama.rs then calls daily_usd_field_fail_closed, which makes the whole day null if the count is nonzero. Upstream dexs/cl8y-dex throws on that null. Its arithmetic and start pin otherwise match the feed.

The ten null dates in the 30-day sample are 08-27, 09-01, 09-05, 09-09, 09-11, 09-12, 09-13, 09-17, 09-18, and 09-24. Llama's chart contains the other twenty days and reports their approximately $31,166 sum. This is a lower bound problem as well: the $53,177.23 comparison includes only priced swaps, not a guessed USD value for the unpriced hops.

2. Fees: two stale history layers, plus omitted fee sources

Examples on matching UTC days: 09-10 is $69 stored by Llama, $69.23 on today's Llama daily feed, $92.10 on today's protocol fee series; 09-17 is $115 stored versus $166.95 on both live indexer series. 09-24 agrees at about $2.01. Upstream fees/cl8y-dex maps the current endpoint's seven sources into labeled groups and a residual; the present-day value agrees. The issue is historical state, not an observed double-add in that adapter.

indexer/src/indexer/defillama.rs sets DAILY_LOOKBACK_DAYS = 8, and refresh_defillama_daily only reaggregates today and those eight prior days. protocol_fee_series reaggregates 90 prior days. The completed #1269 multihop fee repair backfills historical protocol_fee_events, and the NULL-only USD backfill can fill older rows later. Old Llama daily fee rollups therefore have a concrete route to remaining stale; Llama's already stored daily chart can lag the corrected endpoint again. Public gem-pair swap tapes show no gem swaps during this 30-day sample, but a production DB comparison by fee source and pair is needed to distinguish every dollar of the $80.73 indexer-series gap from intentional gem exclusions or other differences.

FeeSource::ALL and the live /protocol/fees API still contain only seven sources. Pair-creation treasury fees and community SKU/settings invoices are absent from both the fee ledger and DeFiLlama feed; existing #1209 and #1210 specify that work. fj currently shows both closed, while their latest comments and main still say implementation is outstanding after docs-only auto-close. Their amounts are not included in the $150.60 measured gap because the underlying events have not been reconciled here.

Source-level additional risk: indexer/src/db/queries/defillama.rs excludes a swap_amm fee when any gem-pair swap has the same tx_hash, even though #1269 now stores protocol_fee_events.pair_id. A mixed gem/non-gem router transaction would lose the non-gem fee too. Current 30-day impact is unmeasured; this is not an explanation for the volume nulls.

3. TVL: token price coverage, not a bad factory pin

The upstream TVL adapter queries the configured factory's listed Pool {} reserves, maps cLUNC/cUSTC 1:1 to native uluna/uusd, and adds other CW20s by address. The factory matches the deployment registry. Llama's current token breakdown values only LUNC and USTC. A Llama coins price request likewise returns only the two native prices. Material omitted legs include the UST1/USTR pool (about $42.3k at probe time) and UST1's leg of the UST1/cUSTC pool (pool about $53.1k). Llama has no verified UST1, USTR, or CL8Y CW20 prices. /overview values some pools by a priced leg times two, so its ~$157–160k is an illustrative coverage comparison, not an exact expected Llama TVL.

Recommended fixes and acceptance

Restore publishable priced volume. For the DeFiLlama headline return the existing priced USD sum when positive, even if another swap is unpriced; idle stays "0", all-unpriced activity stays JSON null. Add unpriced_trade_count / priced-trade count or completeness metadata to the daily API and document that the headline is a priced lower bound. Keep gem exclusion, one row per swap hop, and no wrap/window or limit-fill double count. Investigate the CL8Y/ALPHA hop's missing USD mark separately using a verified, time-appropriate price; do not invent a peg. Recompute from adapter start 2026-08-17 and request Llama to reload the affected daily dates. Acceptance: 2026-09-24 publishes at least its existing $558.46 priced sum, and the anchored 30-day chart includes the ten missing dates without reporting unpriced activity as $0.
Reconcile and repair fee history. Run a read-only per-day/per-source/per-pair SQL comparison of protocol_fee_events, defillama_daily_fees, and protocol_daily_fees for 2026-08-17 onward, with the same gem rules. Then reaggregate defillama_daily_* for all affected historical days after #1269/NULL-USD backfills, and ask Llama to reingest those dates. Add a dirty-day queue or explicit backfill command when old fee rows change; the ordinary eight-day loop can remain bounded. Acceptance: every historical day matches the priced ledger after intentional exclusions, and Llama's stored fee chart matches the corrected daily endpoint within display rounding; document any excluded amounts.
Finish missing fee sources through their existing issues. Restore tracking on #1209 and #1210 if still closed by docs-only merges; implement their pinned on-chain fee parsers and wire both /protocol/fees and DeFiLlama. Do not infer gas, spread, tax, or unverified invoices as fees. Add a mixed gem/non-gem same-tx case and scope pair-linked fee exclusion to its own pair_id/event rather than any matching tx hash.
Improve TVL price coverage. Work with DeFiLlama on verifiable Terra Classic CW20 prices for UST1, USTR, CL8Y, and other material factory pool legs; preserve cLUNC/cUSTC's true 1:1 wrap treatment. UST1 is an unstablecoin: do not hardcode $1, and do not invent USTR or gem pegs. Compare factory reserve amounts, valued versus unvalued token balances, and the Llama token breakdown at the same block/time before declaring a TVL target. Keep raw on-chain pool reserves as the TVL source.
Add a small ongoing drift check. Compare completed UTC-day Llama volume/fees with the daily endpoint after its normal delay, and record the priced/unpriced counts and historical corrections. Compare Llama's priced TVL token list with factory pool inventory. Alert on missing dates or material historical drift rather than silently accepting a smaller 7d/30d total.

Related: #631 listing, #687 fee adapter/start repair, #688 post-merge operations, #1269 multihop fee backfill, #1209 and #1210 missing fee sources, #629 price coverage context. In-repo source: scripts/defillama/, indexer/src/api/defillama.rs, indexer/src/db/queries/defillama.rs, indexer/src/indexer/defillama.rs, indexer/src/db/queries/protocol_fee_series.rs.

## Summary On 2026-09-25 around 05:25–05:35 UTC, CL8Y DEX's DeFiLlama volume, fee history, and TVL diverged from the live indexer and pool inventory for different reasons. This is a reconciliation issue; keep the already completed listing/adapter tickets #631 and #687 closed. The three current causes are: (1) one unpriced swap makes an entire UTC day's otherwise priced volume JSON `null`, and the upstream DEX adapter throws; (2) historical fee corrections do not propagate through both the eight-day indexer Llama rollup and Llama's stored daily chart; (3) Llama prices only LUNC and USTC in the TVL adapter's reserve basket, omitting material CW20 pool legs. There are also missing fee sources and a pair-scoping risk described below. ## Reproducible evidence **Compare completed UTC days 2026-08-26 through 2026-09-24, inclusive** for flows. Do not compare a UTC-day flow with `/overview` trailing 24h/30d. | Metric | DeFiLlama stored value | Current indexer value | Observation | | --- | ---: | ---: | --- | | DEX volume, 30 completed UTC days | **$31,166** | **$53,177.23** known priced lower bound | Ten of 30 daily Llama responses are `null`; their already priced swaps sum to **$22,011.24** in the indexer's UTC protocol series. That accounts for the gap (rounding aside). | | Fees, same days | **$746.87** | **$816.74** sum of current `/defillama/daily`; **$897.47** in `/protocol/fees/daily` | Two gaps: **$69.87** between Llama's stored chart and its current feed; **$80.73** between the current feed and the current broader fee series. The latter needs source-level DB reconciliation before attributing every dollar. | | TVL, 2026-09-25 snapshot | **$84,722.46** at 03:53 UTC | `/overview` around **$157–160k** about 1–2 hours later | These use different USD methodologies and are not directly equal. Llama's token breakdown contains **only** LUNC $28,357.89 and USTC $56,364.57; it omits all other CW20 pool legs. | Public evidence: [Llama DEX volume](https://api.llama.fi/summary/dexs/cl8y-dex), [Llama fees](https://api.llama.fi/summary/fees/cl8y-dex), [Llama TVL with `tokensInUsd`](https://api.llama.fi/protocol/cl8y-dex), [indexer UTC volume series](https://indexer.dex.cl8y.com/api/v1/protocol/volume/daily?days=30), [indexer UTC fee series](https://indexer.dex.cl8y.com/api/v1/protocol/fees/daily?grain=daily&limit=45), [indexer overview](https://indexer.dex.cl8y.com/api/v1/overview), [2026-09-24 Llama feed](https://indexer.dex.cl8y.com/api/v1/defillama/daily?timestamp=1790208000). For the exact 30-day volume window, add the 2026-08-26 [daily feed](https://indexer.dex.cl8y.com/api/v1/defillama/daily?timestamp=1787702400) to the protocol series' 2026-08-27–09-24 points and exclude its in-progress 09-25 point. ### 1. Volume: a single unpriced hop erases the day's priced total The [2026-09-24 daily response](https://indexer.dex.cl8y.com/api/v1/defillama/daily?timestamp=1790208000) has `trade_count: 28` and `volume_usd: null`. The [same UTC day in the protocol series](https://indexer.dex.cl8y.com/api/v1/protocol/volume/daily?days=7) has **$558.461005** of priced swaps and 28 trades. A [CL8Y-cb/ALPHA trade](https://indexer.dex.cl8y.com/api/v1/pairs/terra163qm8z5rjgp8av6c6sg673lq2v4kfa0we5uhtzj8alfwhddfhjfs27k40z/trades?limit=100) at 16:56:49 UTC, tx `37A8F6FAB565F532382C1EA1EC349624BB1D0E2316409AF95EBAB53EF294EAAF`, has no USD price; it is one hop in a route through [UST1/ALPHA](https://indexer.dex.cl8y.com/api/v1/pairs/terra1rmdtckz5gd0usja36ydwat6prnmew639ry37yq72xh9ek4s3m83sehn5u6/trades?limit=100). Both CL8Y and ALPHA are outside the swap-volume USD catalog for that pair. The source query in `indexer/src/db/queries/defillama.rs` stores a positive priced `SUM(volume_usd)` **and** `unpriced_trade_count`; `indexer/src/api/defillama.rs` then calls `daily_usd_field_fail_closed`, which makes the whole day `null` if the count is nonzero. [Upstream `dexs/cl8y-dex`](https://github.com/DefiLlama/dimension-adapters/blob/master/dexs/cl8y-dex/index.ts) throws on that `null`. Its arithmetic and start pin otherwise match the feed. The ten null dates in the 30-day sample are 08-27, 09-01, 09-05, 09-09, 09-11, 09-12, 09-13, 09-17, 09-18, and 09-24. Llama's chart contains the other twenty days and reports their approximately $31,166 sum. This is a lower bound problem as well: the $53,177.23 comparison includes only priced swaps, not a guessed USD value for the unpriced hops. ### 2. Fees: two stale history layers, plus omitted fee sources Examples on matching UTC days: 09-10 is **$69** stored by Llama, **$69.23** on today's Llama daily feed, **$92.10** on today's protocol fee series; 09-17 is **$115** stored versus **$166.95** on both live indexer series. 09-24 agrees at about **$2.01**. [Upstream `fees/cl8y-dex`](https://github.com/DefiLlama/dimension-adapters/blob/master/fees/cl8y-dex/index.ts) maps the current endpoint's seven sources into labeled groups and a residual; the present-day value agrees. The issue is historical state, not an observed double-add in that adapter. `indexer/src/indexer/defillama.rs` sets `DAILY_LOOKBACK_DAYS = 8`, and `refresh_defillama_daily` only reaggregates today and those eight prior days. `protocol_fee_series` reaggregates 90 prior days. The completed [#1269 multihop fee repair](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1269) backfills historical `protocol_fee_events`, and the NULL-only USD backfill can fill older rows later. Old Llama daily fee rollups therefore have a concrete route to remaining stale; Llama's already stored daily chart can lag the corrected endpoint again. Public gem-pair swap tapes show no gem swaps during this 30-day sample, but a production DB comparison by fee source and pair is needed to distinguish every dollar of the **$80.73** indexer-series gap from intentional gem exclusions or other differences. `FeeSource::ALL` and the live `/protocol/fees` API still contain only seven sources. Pair-creation treasury fees and community SKU/settings invoices are absent from both the fee ledger and DeFiLlama feed; existing [#1209](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1209) and [#1210](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1210) specify that work. `fj` currently shows both closed, while their latest comments and `main` still say implementation is outstanding after docs-only auto-close. Their amounts are **not** included in the $150.60 measured gap because the underlying events have not been reconciled here. Source-level additional risk: `indexer/src/db/queries/defillama.rs` excludes a `swap_amm` fee when **any** gem-pair swap has the same `tx_hash`, even though #1269 now stores `protocol_fee_events.pair_id`. A mixed gem/non-gem router transaction would lose the non-gem fee too. Current 30-day impact is unmeasured; this is not an explanation for the volume nulls. ### 3. TVL: token price coverage, not a bad factory pin The [upstream TVL adapter](https://github.com/DefiLlama/DefiLlama-Adapters/blob/master/projects/cl8y-dex/index.js) queries the configured factory's listed `Pool {}` reserves, maps cLUNC/cUSTC 1:1 to native `uluna`/`uusd`, and adds other CW20s by address. The factory matches the deployment registry. [Llama's current token breakdown](https://api.llama.fi/protocol/cl8y-dex) values only LUNC and USTC. A [Llama coins price request](https://coins.llama.fi/prices/current/terra:terra1f0eqgy9w7e5e7up97vjudqwx38tesf8ylx75x2lv3nwm0clry0pqmgfy72,terra:terra1vy3kc0swag2rhn7jz6n72jp0l2ns0p6r6ez5grxq5uhj2rvs97fqfsetxv,terra:terra16wtml2q66g82fdkx66tap0qjkahqwp4lwq3ngtygacg5q0kzycgqvhpax3,terra:uluna,terra:uusd) likewise returns only the two native prices. Material omitted legs include the [UST1/USTR pool](https://indexer.dex.cl8y.com/api/v1/pairs/terra16vxrhpvpcucu05y0nr862vf9hnqeh274uaff4s7hz4n0ea74006qf5hgqy) (about **$42.3k** at probe time) and UST1's leg of the [UST1/cUSTC pool](https://indexer.dex.cl8y.com/api/v1/pairs/terra1ceprjsxp86ggftf5e38wwt34l83e5gq7penkdnv4wsatkwcs8v6qccw55f) (pool about **$53.1k**). Llama has no verified UST1, USTR, or CL8Y CW20 prices. `/overview` values some pools by a priced leg times two, so its ~$157–160k is an illustrative coverage comparison, not an exact expected Llama TVL. ## Recommended fixes and acceptance 1. **Restore publishable priced volume.** For the DeFiLlama *headline* return the existing priced USD sum when positive, even if another swap is unpriced; idle stays `"0"`, all-unpriced activity stays JSON `null`. Add `unpriced_trade_count` / priced-trade count or completeness metadata to the daily API and document that the headline is a priced lower bound. Keep gem exclusion, one row per swap hop, and no wrap/window or limit-fill double count. Investigate the CL8Y/ALPHA hop's missing USD mark separately using a verified, time-appropriate price; do not invent a peg. Recompute from adapter start **2026-08-17** and request Llama to reload the affected daily dates. Acceptance: 2026-09-24 publishes at least its existing $558.46 priced sum, and the anchored 30-day chart includes the ten missing dates without reporting unpriced activity as `$0`. 2. **Reconcile and repair fee history.** Run a read-only per-day/per-source/per-pair SQL comparison of `protocol_fee_events`, `defillama_daily_fees`, and `protocol_daily_fees` for 2026-08-17 onward, with the same gem rules. Then reaggregate `defillama_daily_*` for all affected historical days after #1269/NULL-USD backfills, and ask Llama to reingest those dates. Add a dirty-day queue or explicit backfill command when old fee rows change; the ordinary eight-day loop can remain bounded. Acceptance: every historical day matches the priced ledger after intentional exclusions, and Llama's stored fee chart matches the corrected daily endpoint within display rounding; document any excluded amounts. 3. **Finish missing fee sources through their existing issues.** Restore tracking on #1209 and #1210 if still closed by docs-only merges; implement their pinned on-chain fee parsers and wire both `/protocol/fees` and DeFiLlama. Do not infer gas, spread, tax, or unverified invoices as fees. Add a mixed gem/non-gem same-tx case and scope pair-linked fee exclusion to its own `pair_id`/event rather than any matching tx hash. 4. **Improve TVL price coverage.** Work with DeFiLlama on verifiable Terra Classic CW20 prices for UST1, USTR, CL8Y, and other material factory pool legs; preserve cLUNC/cUSTC's true 1:1 wrap treatment. UST1 is an unstablecoin: do **not** hardcode `$1`, and do not invent USTR or gem pegs. Compare factory reserve amounts, valued versus unvalued token balances, and the Llama token breakdown at the same block/time before declaring a TVL target. Keep raw on-chain pool reserves as the TVL source. 5. **Add a small ongoing drift check.** Compare completed UTC-day Llama volume/fees with the daily endpoint after its normal delay, and record the priced/unpriced counts and historical corrections. Compare Llama's priced TVL token list with factory pool inventory. Alert on missing dates or material historical drift rather than silently accepting a smaller 7d/30d total. Related: [#631](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/631) listing, [#687](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/687) fee adapter/start repair, [#688](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/688) post-merge operations, [#1269](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1269) multihop fee backfill, [#1209](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1209) and [#1210](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1210) missing fee sources, [#629](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/629) price coverage context. In-repo source: `scripts/defillama/`, `indexer/src/api/defillama.rs`, `indexer/src/db/queries/defillama.rs`, `indexer/src/indexer/defillama.rs`, `indexer/src/db/queries/protocol_fee_series.rs`.