fix: DeFiLlama fees adapter throws on null — unblock dimension-adapters#8987 #687

Closed
opened 2026-08-27 06:03:00 +00:00 by PlasticDigits · 15 comments
PlasticDigits commented 2026-08-27 06:03:00 +00:00 (Migrated from gitlab.com)

Summary

Llama maintainer review of dimension-adapters#8987 (fees/cl8y-dex): recent UTC days stay null and the adapter throws. Volume for the same days can be a number. This leftover does not reopen closed #631 (listing + daily GET) or #683 (CL8Y / economic fee_usd stamps). Bundle indexer GET contract + vendored / upstream adapter + start-date / 404 backfill into one ticket.

Live probe (2026-08-27): yesterday–6d now return numeric daily_fees_usd after economic stamps; 2026-05-01 still 404; ~2026-08-13 and older 404; 2026-08-17–21 return volume with daily_fees_usd: "0". The throw-on-null path is still in the GitHub PR and in scripts/defillama/fees/index.ts. Any future unpriced fee event (or Llama backfill of a 404 day) fails pnpm test fees cl8y-dex again.


Current codebase

Upstream PR (what Llama runs)

dimension-adapters#8987 adds dexs/cl8y-dex/index.ts and fees/cl8y-dex/index.ts. Both are version 1, pin GET https://indexer.dex.cl8y.com/api/v1/defillama/daily?timestamp=${options.startOfDay}, and throw when the USD field is JSON null:

  • Volume: asNumberOrNull(data?.volume_usd) → throw cl8y-dex dailyVolume unpriced or missing
  • Fees: asNumberOrNull(data?.daily_fees_usd) → throw cl8y-dex dailyFees unpriced or missing

httpGet (axios) also throws on HTTP 404. CI previously failed with Request failed with status code 404 before Coolify shipped the route. The fees review is the remaining merge block: maintainer pnpm test fees cl8y-dex over the last few UTC days hit 200 + daily_fees_usd: null + throw.

Fees mapping in the PR: labeled METRIC.SWAP_FEES / DEPOSIT_WITHDRAW_FEES / MINT_REDEEM_FEES from fees.*, residual daily_fees_usd - labeled, dailyRevenue from daily_revenue_usd (fallback total), dailySupplySideRevenue: 0.

Indexer daily GET (#631)

Layer Path Behavior
GET indexer/src/api/defillama.rs O(1) rollup + 60s cache. Invalid / unaligned / future timestamp → 400. Missing defillama_daily_stats row → 404.
Aggregate fees same file, daily_usd_field_fail_closed(fee_events, fee_unpriced, &fee_usd) Any unpriced_count > 0 across all sources → whole daily_fees_usd / revenue fields are JSON null.
Per-source fee_field Same fail-closed per swap_amm … ust1_redeem. Missing source → "0".
Rollup indexer/src/db/queries/defillama.rs UTC-day SUM(fee_usd) from protocol_fee_events; gem-pair txs excluded for pair sources; wrap/window always included. Inserts all seven FeeSource::ALL rows.
Fail-closed helper indexer/src/indexer/defillama.rs daily_usd_field_fail_closed Idle (activity_count <= 0) → "0". Else if unpriced_count > 0 or priced sum <= 0 → None.
Lookback DAILY_LOOKBACK_DAYS = 8 Refresh today + 8 prior UTC days only. No prune of defillama_daily_*; days never refreshed stay 404.
Adapter start pin ADAPTER_START_UTC_DAY = 1_777_593_600 (2026-05-01) scripts/defillama/gems.js ADAPTER_START. Llama will request every day from that start.

Tests already encode fail-closed: indexer/tests/indexer_defillama.rs unpriced_active_is_null_idle_is_zero asserts daily_fees_usd is JSON null when a swap_amm row has fee_usd = NULL.

Conflict with #683 (closed)

#683 EFee-6: Protocol headlines are partial priced SUM; idle "0"; all activity unpriced → null. Llama was documented as inheriting stamped fee_usd.

Llama GET does not match that. One unpriced CL8Y limit_place / ask-side swap_amm (the original #683 hole) nulls wrap + UST1 window + priced pair commission for the whole UTC day. Volume for the same day can stay a number because swap_events.volume_usd is offer/quote hub catalog, not ask-side CL8Y.

Vendored copies (drift)

Copy Path Drift vs GitHub PR
Volume scripts/defillama/dexs/index.ts Uses mapDaily.js + options.http.get; still throws on null. start is unix ADAPTER_START, PR uses "2026-05-01".
Fees scripts/defillama/fees/index.ts Throws on null. Adds mapped.dailyFees then adds each breakdown label again — double-counts if this file is copied upstream. PR adds labeled METRIC amounts + residual only.
Mapper scripts/defillama/dimensions/mapDaily.js asNumberOrNull treats JSON null as null (correct for L631-7). Unit tests do not cover adapter throw / 404 / partial SUM.

Docs still say #8987 is draft until Coolify ships the route (docs/DEFILLAMA.md, scripts/defillama/README.md). The route is live; the leftover is null/404 contract vs Llama crawler.


Why the new implementation is needed

  1. Llama cannot merge #8987. Maintainer fees test over recent days throws. Listing on defillama.com Dexs/Fees is blocked even though TVL is a separate repo and volume can succeed.
  2. Fail-closed on the headline is the wrong grain. A single unpriced economic commission must not wipe priced wrap/window/hub fees. That contradicts EFee-6 and makes the crawler brittle after every new listed ask asset.
  3. Throw-on-null is a hard fail, not a chart gap. Llama pnpm test / production fetch treat adapter exceptions as adapter breakage. JSON null is our honest “unpriced” signal; the adapter must not turn a partial-price day into an exception once the GET headline is a number.
  4. Start date vs rollup coverage. start: "2026-05-01" + lookback 8 days → Llama historical backfill 404s from May through mid-August. Axios throws. Same class of “adapter broken” as null fees.
  5. Honest zeros vs silent undercount. 2026-08-17–21 currently show volume and daily_fees_usd: "0" (no fee rows / ingest gap). That must stay documented idle-or-pre-ingest zero, not a silent $0 covering unpriced CL8Y. After #683 stamps, refresh those days if events exist.

Constraints and guardrails

  • Do not reopen #631 or #683. Do not change P522-Q volume catalog, hub card (H11), or gem exclude (L631-4 / #562).
  • Do not treat CG liquidity_in_usd / overview.total_volume_24h_usd as Llama volume or TVL (A1 / L631-1).
  • Do not count spread_amount, burn tax, gas, hooks, or community-tax extra-debit as dailyFees (L631-6).
  • Do not set dailySupplySideRevenue to a residual. SSR stays 0. Ignore review bots that reclassify leftover USD as LP commission.
  • Do not invent UST1=$1 or USTR=2.5×USTC. Named wrap substitution only (cLUNC→uluna, cUSTC→uusd).
  • Do not add from/to range dump on GET /defillama/daily. Single UTC-day timestamp only (L631-7 / A12).
  • Do not scan swap_events / protocol_fee_events on GET. Refresh stays the ~5 min volume loop.
  • Do not bind-mount indexer/ into root Docker for cargo.
  • Do not open Llama GitHub PRs from this repo’s CI. Patch the GitHub branch + keep scripts/defillama/ in lockstep.
  • Do not move start earlier than the first day the daily API can return 200 with the documented idle/unpriced contract.
  • Partial priced SUM for the headline (daily_fees_usd / revenue) must match EFee-6. Per-source fees.swap_amm etc. may still be null when that source has unpriced events.
  • Headline null remains valid only when fee activity exists and the priced SUM is empty (all unpriced). Adapter may throw on that (fail closed) or skip the day — pick one, document it, and keep volume/fees consistent.
  • HTTP 404 for days before first rollup: prefer moving adapter start to the first 200 day over fabricating May–August zeros. Optional one-time backfill of defillama_daily_* from that start is OK; do not widen GET into a live scan.
  • CORS stays https://dex.cl8y.com. Llama is server-side. Host pin stays https://indexer.dex.cl8y.com (A18).

Relevant files

Area Path
Daily GET indexer/src/api/defillama.rs
Fail-closed helper indexer/src/indexer/defillama.rs
Fee/volume rollup indexer/src/db/queries/defillama.rs
Asset rollup indexer/src/db/queries/defillama_assets.rs
Migrations indexer/migrations/20260825150000_defillama_daily.sql, 20260825160000_defillama_daily_assets.sql
Indexer tests indexer/tests/indexer_defillama.rs
Economic stamps indexer/src/indexer/economic_usd.rs, indexer/src/indexer/protocol_fees.rs (overview_fee_usd_field)
Vendored volume scripts/defillama/dexs/index.ts
Vendored fees scripts/defillama/fees/index.ts
Mapper + tests scripts/defillama/dimensions/mapDaily.js, mapDaily.test.js
Pins scripts/defillama/gems.js
Docs / skill docs/DEFILLAMA.md, skills/AGENTS_DEFILLAMA.md, docs/indexer-invariants.md (DeFiLlama UTC-day + Economic fee USD)
Verify scripts/qa/verify-issue-631.sh, Makefile verify-issue-631 / verify-issue-683
Upstream https://github.com/DefiLlama/dimension-adapters/pull/8987 (dexs/cl8y-dex, fees/cl8y-dex)

  1. Indexer headline = EFee-6 partial SUM. daily_fees_usd / daily_revenue_usd / daily_protocol_revenue_usd = SUM of priced source amount_usd. JSON null only when total event_count > 0 and priced SUM is 0. Do not OR unpriced_count across sources into a whole-day null while wrap/window/hub sources are priced. Keep per-source fees.* fail-closed.
  2. Refresh after stamps. Re-run refresh_defillama_daily (or a one-shot from adapter start) so Coolify rows pick up #683 NULL-only fee_usd backfill. Days with volume and true zero fee events stay "0".
  3. Adapter start. Set GitHub + scripts/defillama/gems.js ADAPTER_START to the first UTC day that returns 200 (today that is ~2026-08-17, not 2026-05-01). Document in docs/DEFILLAMA.md. Version stays 1 (cannot split hourly).
  4. Adapter errors. Keep throw on HTTP 5xx / malformed JSON. Do not throw on JSON "0". After (1), last-few-days tests must return numeric dailyFees. If a day is still all-unpriced (null), either throw with a stable message or omit the day — do not map unpriced → $0. Catch 404 only if start still precedes coverage; preferred fix is (3), not swallow-404-as-zero.
  5. Sync copies. GitHub PR remains METRIC labeled + residual (no double-count). Fix scripts/defillama/fees/index.ts so it does not addUSDValue(total) then add breakdown again. start string vs unix: one canonical form in both trees.
  6. Docs. #8987 is no longer “draft until Coolify”. State: leftover is null/404 vs crawler; close this issue when pnpm test fees cl8y-dex and pnpm test dexs cl8y-dex succeed for yesterday and start, and the GitHub PR is re-tested.
  7. Verify target. Add make verify-issue-<this-iid> that covers indexer partial-SUM tests + mapper/adapter-start pins + verify-issue-631 / 683 related. Keep VERIFY_ISSUE_631_SKIP_RELATED pattern if needed.

Acceptance criteria

  1. GET /api/v1/defillama/daily?timestamp=<yesterday_00:00_utc> returns 200 with numeric daily_fees_usd when any priced treasury fee exists that day (wrap/window/hub/economic). One unpriced source does not null the headline.
  2. Headline null only when fee events exist and none have fee_usd > 0. Idle (no fee events) stays "0" even if volume is non-zero.
  3. Per-source fees.swap_amm (etc.) may be null independently; SSR remains "0"; daily_revenue_usd equals daily_fees_usd.
  4. Adapter start is the first 200 UTC day. GET for start is 200; GET for start - 86400 is 404 (or start is the genesis row). Llama backfill does not walk May 2026 404s.
  5. GitHub fees/cl8y-dex + in-repo copy: no throw on numeric / "0" days; no double-count of breakdown vs total; METRIC labels ⊆ breakdownMethodology.
  6. pnpm test fees cl8y-dex and pnpm test dexs cl8y-dex (yesterday and start) succeed; output pasted on #8987 (operator).
  7. make verify-issue-<this-iid> green on host Postgres. make verify-issue-631 and make verify-issue-683 stay green.
  8. Docs/skill/invariants updated; #8987 draft language removed.

Test plan (all paths)

Indexer (Postgres) — make setup-indexer-postgres

  • Seed same UTC day: priced swap_amm (hub token) + priced wrap + unpriced CL8Y limit_place (fee_usd NULL) + gem-pair swap_amm (excluded).
  • Refresh daily rollup.
  • Headline daily_fees_usd = priced swap_amm + wrap only (partial SUM). fees.limit_place is null. Gem pair not in any source.
  • Same day, all fee rows unpriced → headline null (not "0"). Volume may still be a number.
  • Idle fee day with swaps → daily_fees_usd "0", volume_usd priced or fail-closed on its own unpriced swaps.
  • Hybrid fill: book_take from limit_order_fills.commission_amount only (not swap-level book_commission_amount); volume still L10 parent swap.
  • Unaligned timestamp / injection / negative / future → 400. Unknown day → 404. Repeat GET within 60s does not rescan protocol_fee_events.
  • After NULL-only fee_usd backfill fixture, refresh flips headline from null to partial SUM; non-null stamps are not rewritten (C568-1).

Mapper / vendored adapters (no Llama SDK)

  • node --test scripts/defillama/dimensions/mapDaily.test.js (and fees start pin).
  • mapFees: "0" → 0; JSON null → null; numeric string → number. SSR 0. Breakdown keys ⊆ methodology.
  • Fixture: daily_fees_usd 10, labeled 10 → no residual. Labeled 8 + total 10 → residual 2 without double-counting 10+8.
  • dailyUrl(ADAPTER_START) hits the first 200 day, not 2026-05-01 unless that day is 200.
  • Static: mapper never reads liquidity_in_usd / total_liquidity_usd.

Upstream clone (operator)

  • pnpm test fees cl8y-dex yesterday UTC → numeric dailyFees, revenue = fees, SSR 0.
  • pnpm test fees cl8y-dex <start> → 200 path, no 404.
  • pnpm test dexs cl8y-dex same days → numeric dailyVolume.
  • Repeat “last few days” (yesterday, −1, −2, −3) — none throw.

Live Coolify

  • Yesterday and start curl as in scripts/defillama/README.md.
  • A mid-lookback day that previously 404s (e.g. 2026-08-13) either stays 404 (if before start) or 200 after backfill — consistent with adapter start.

Test plan (attack, hack, abuse)

ID Vector Expectation
A1 Publish CG liquidity_in_usd as Llama TVL/volume Still forbidden; this ticket does not touch TVL adapter.
A2 Double-count hybrid fills into dailyVolume Unchanged L10; volume tests still exclude limit_order_fills.
A3 Wash gem pairs to inflate Llama fees/volume Gem-pair exclude unchanged on pair-linked sources.
A4 One unpriced CL8Y commission nulls wrap/window headline Must not after this ticket (partial SUM).
A5 Map unpriced headline to $0 so Llama stops throwing Forbidden — silent undercount. null or skip, not zero.
A6 Count spread_amount / community-tax extra-debit as residual dailyFees Residual only from already-ingested treasury sources; no new source invented to “balance” income statement.
A7 Reclassify residual as dailySupplySideRevenue Forbidden. SSR stays 0.
A8 Swallow HTTP 404 as $0 from May 2026 Forbidden. Move start or backfill real rows.
A9 Symbol spoof CL8Y on a native/gem to mint economic fee_usd EFee-1 contract pin; gems stay unpriced.
A10 Compromised indexer inflates daily_fees_usd GET remains public read, rollup-only; TVL stays on-chain. No write methods.
A11 timestamp injection / unaligned / future 400; bind i64; no raw SQL.
A12 from/to range dump Still ignored / 400 without single-day timestamp.
A13 Unbounded lookback refresh DoS Refresh cap stays bounded (8d or documented start-backfill job, not GET).
A14 SSRF / user-supplied indexer URL in adapter Host pin indexer.dex.cl8y.com only.
A15 CORS browser spoof of daily API CORS allowlist unchanged; Llama is server-side.
A16 Double-count labeled METRIC + headline in createBalances In-repo + GitHub fees adapters add labels or residual, not total+labels.
A17 Rewrite historical non-null fee_usd when the mark jumps C568-1 / EFee-5 NULL-only backfill.
A18 Price vFDUSD / gems via CEX or Venus into Llama fees X4 / EFee-3; still unpriced.
A19 Widen GET to live SUM(protocol_fee_events) Forbidden (A11–A12 class).
A20 Adapter version 2 hourly split of the UTC-day rollup Forbidden. Stay version 1.

Verification criteria

Issue is done when all of the following are true:

  1. Coolify GET /api/v1/defillama/daily?timestamp=<yesterday_utc> returns 200 with numeric daily_fees_usd whenever priced fee events exist; maintainer-style “last few days” do not return headline null solely because one source is unpriced.
  2. Adapter start matches the first 200 UTC day; GitHub + scripts/defillama/gems.js agree.
  3. Comment on dimension-adapters#8987 with pnpm test fees cl8y-dex and pnpm test dexs cl8y-dex logs (yesterday + start). Maintainer can re-run without throw.
  4. make verify-issue-<this-iid> passes (make setup-indexer-postgres). Related verify-issue-631 and verify-issue-683 pass.
  5. Docs/skill/invariants describe partial-SUM headline vs per-source fail-closed; #8987 is not documented as waiting on a missing Coolify route.
  6. In-repo fees copy does not double-count; GitHub PR remains METRIC + residual; SSR is 0.

Close is blocked if Llama fees tests still throw on recent days, if unpriced activity is published as $0, if SSR becomes non-zero, if from/to is added, or if TVL/volume start using indexer USD / CG liquidity_in_usd.


## Summary Llama maintainer review of [dimension-adapters#8987](https://github.com/DefiLlama/dimension-adapters/pull/8987) (`fees/cl8y-dex`): **recent UTC days stay `null` and the adapter throws**. Volume for the same days can be a number. This leftover does **not** reopen closed [#631](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/631) (listing + daily GET) or [#683](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/683) (CL8Y / economic `fee_usd` stamps). Bundle indexer GET contract + vendored / upstream adapter + start-date / 404 backfill into **one** ticket. Live probe (2026-08-27): yesterday–6d now return numeric `daily_fees_usd` after economic stamps; **2026-05-01 still 404**; **~2026-08-13 and older 404**; **2026-08-17–21 return volume with `daily_fees_usd: "0"`**. The throw-on-null path is still in the GitHub PR and in `scripts/defillama/fees/index.ts`. Any future unpriced fee event (or Llama backfill of a 404 day) fails `pnpm test fees cl8y-dex` again. --- ## Current codebase ### Upstream PR (what Llama runs) [dimension-adapters#8987](https://github.com/DefiLlama/dimension-adapters/pull/8987) adds `dexs/cl8y-dex/index.ts` and `fees/cl8y-dex/index.ts`. Both are **version 1**, pin `GET https://indexer.dex.cl8y.com/api/v1/defillama/daily?timestamp=${options.startOfDay}`, and **throw** when the USD field is JSON `null`: - Volume: `asNumberOrNull(data?.volume_usd)` → throw `cl8y-dex dailyVolume unpriced or missing` - Fees: `asNumberOrNull(data?.daily_fees_usd)` → throw `cl8y-dex dailyFees unpriced or missing` `httpGet` (axios) also throws on **HTTP 404**. CI previously failed with `Request failed with status code 404` before Coolify shipped the route. The fees review is the remaining merge block: maintainer `pnpm test fees cl8y-dex` over the last few UTC days hit **200 + `daily_fees_usd: null` + throw**. Fees mapping in the PR: labeled `METRIC.SWAP_FEES` / `DEPOSIT_WITHDRAW_FEES` / `MINT_REDEEM_FEES` from `fees.*`, residual `daily_fees_usd - labeled`, `dailyRevenue` from `daily_revenue_usd` (fallback total), `dailySupplySideRevenue: 0`. ### Indexer daily GET (#631) | Layer | Path | Behavior | |-------|------|----------| | GET | [`indexer/src/api/defillama.rs`](indexer/src/api/defillama.rs) | O(1) rollup + 60s cache. Invalid / unaligned / future timestamp → **400**. Missing `defillama_daily_stats` row → **404**. | | Aggregate fees | same file, `daily_usd_field_fail_closed(fee_events, fee_unpriced, &fee_usd)` | **Any** `unpriced_count > 0` across **all** sources → whole `daily_fees_usd` / revenue fields are JSON **`null`**. | | Per-source | `fee_field` | Same fail-closed per `swap_amm` … `ust1_redeem`. Missing source → `"0"`. | | Rollup | [`indexer/src/db/queries/defillama.rs`](indexer/src/db/queries/defillama.rs) | UTC-day `SUM(fee_usd)` from `protocol_fee_events`; gem-pair txs excluded for pair sources; wrap/window always included. Inserts **all seven** `FeeSource::ALL` rows. | | Fail-closed helper | [`indexer/src/indexer/defillama.rs`](indexer/src/indexer/defillama.rs) `daily_usd_field_fail_closed` | Idle (`activity_count <= 0`) → `"0"`. Else if `unpriced_count > 0` **or** priced sum `<= 0` → `None`. | | Lookback | `DAILY_LOOKBACK_DAYS = 8` | Refresh today + 8 prior UTC days only. No prune of `defillama_daily_*`; days never refreshed stay **404**. | | Adapter start pin | `ADAPTER_START_UTC_DAY = 1_777_593_600` (2026-05-01) | [`scripts/defillama/gems.js`](scripts/defillama/gems.js) `ADAPTER_START`. Llama will request every day from that start. | Tests already encode fail-closed: [`indexer/tests/indexer_defillama.rs`](indexer/tests/indexer_defillama.rs) `unpriced_active_is_null_idle_is_zero` asserts `daily_fees_usd` is JSON null when a swap_amm row has `fee_usd = NULL`. ### Conflict with #683 (closed) [#683](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/683) **EFee-6**: Protocol headlines are **partial priced SUM**; idle `"0"`; **all** activity unpriced → `null`. Llama was documented as inheriting stamped `fee_usd`. Llama GET does **not** match that. One unpriced CL8Y `limit_place` / ask-side `swap_amm` (the original #683 hole) nulls **wrap + UST1 window + priced pair commission** for the whole UTC day. Volume for the same day can stay a number because `swap_events.volume_usd` is offer/quote hub catalog, not ask-side CL8Y. ### Vendored copies (drift) | Copy | Path | Drift vs GitHub PR | |------|------|--------------------| | Volume | [`scripts/defillama/dexs/index.ts`](scripts/defillama/dexs/index.ts) | Uses `mapDaily.js` + `options.http.get`; still throws on null. `start` is unix `ADAPTER_START`, PR uses `"2026-05-01"`. | | Fees | [`scripts/defillama/fees/index.ts`](scripts/defillama/fees/index.ts) | Throws on null. **Adds `mapped.dailyFees` then adds each breakdown label again** — double-counts if this file is copied upstream. PR adds labeled METRIC amounts + residual only. | | Mapper | [`scripts/defillama/dimensions/mapDaily.js`](scripts/defillama/dimensions/mapDaily.js) | `asNumberOrNull` treats JSON null as null (correct for L631-7). Unit tests do not cover adapter throw / 404 / partial SUM. | Docs still say #8987 is draft until Coolify ships the route ([`docs/DEFILLAMA.md`](docs/DEFILLAMA.md), [`scripts/defillama/README.md`](scripts/defillama/README.md)). The route is live; the leftover is **null/404 contract vs Llama crawler**. --- ## Why the new implementation is needed 1. **Llama cannot merge #8987.** Maintainer fees test over recent days throws. Listing on defillama.com Dexs/Fees is blocked even though TVL is a separate repo and volume can succeed. 2. **Fail-closed on the headline is the wrong grain.** A single unpriced economic commission must not wipe priced wrap/window/hub fees. That contradicts EFee-6 and makes the crawler brittle after every new listed ask asset. 3. **Throw-on-null is a hard fail**, not a chart gap. Llama `pnpm test` / production fetch treat adapter exceptions as adapter breakage. JSON `null` is our honest “unpriced” signal; the adapter must not turn a partial-price day into an exception once the GET headline is a number. 4. **Start date vs rollup coverage.** `start: "2026-05-01"` + lookback 8 days → Llama historical backfill **404**s from May through mid-August. Axios throws. Same class of “adapter broken” as null fees. 5. **Honest zeros vs silent undercount.** 2026-08-17–21 currently show volume and `daily_fees_usd: "0"` (no fee rows / ingest gap). That must stay **documented idle-or-pre-ingest zero**, not a silent `$0` covering unpriced CL8Y. After #683 stamps, refresh those days if events exist. --- ## Constraints and guardrails - **Do not** reopen #631 or #683. Do not change P522-Q volume catalog, hub card (**H11**), or gem exclude (**L631-4** / #562). - **Do not** treat CG `liquidity_in_usd` / `overview.total_volume_24h_usd` as Llama volume or TVL (**A1** / **L631-1**). - **Do not** count `spread_amount`, burn tax, gas, hooks, or community-tax extra-debit as `dailyFees` (**L631-6**). - **Do not** set `dailySupplySideRevenue` to a residual. SSR stays **`0`**. Ignore review bots that reclassify leftover USD as LP commission. - **Do not** invent UST1=`$1` or USTR=2.5×USTC. Named wrap substitution only (cLUNC→`uluna`, cUSTC→`uusd`). - **Do not** add `from`/`to` range dump on `GET /defillama/daily`. Single UTC-day `timestamp` only (**L631-7** / A12). - **Do not** scan `swap_events` / `protocol_fee_events` on GET. Refresh stays the ~5 min volume loop. - **Do not** bind-mount `indexer/` into root Docker for cargo. - **Do not** open Llama GitHub PRs from this repo’s CI. Patch the GitHub branch + keep `scripts/defillama/` in lockstep. - **Do not** move `start` earlier than the first day the daily API can return **200** with the documented idle/unpriced contract. - Partial priced SUM for the **headline** (`daily_fees_usd` / revenue) must match EFee-6. Per-source `fees.swap_amm` etc. may still be `null` when that source has unpriced events. - Headline `null` remains valid only when fee **activity exists and the priced SUM is empty** (all unpriced). Adapter may throw on that (fail closed) **or** skip the day — pick one, document it, and keep volume/fees consistent. - HTTP **404** for days before first rollup: prefer **moving adapter `start`** to the first 200 day over fabricating May–August zeros. Optional one-time backfill of `defillama_daily_*` from that start is OK; do not widen GET into a live scan. - CORS stays `https://dex.cl8y.com`. Llama is server-side. Host pin stays `https://indexer.dex.cl8y.com` (**A18**). --- ## Relevant files | Area | Path | |------|------| | Daily GET | `indexer/src/api/defillama.rs` | | Fail-closed helper | `indexer/src/indexer/defillama.rs` | | Fee/volume rollup | `indexer/src/db/queries/defillama.rs` | | Asset rollup | `indexer/src/db/queries/defillama_assets.rs` | | Migrations | `indexer/migrations/20260825150000_defillama_daily.sql`, `20260825160000_defillama_daily_assets.sql` | | Indexer tests | `indexer/tests/indexer_defillama.rs` | | Economic stamps | `indexer/src/indexer/economic_usd.rs`, `indexer/src/indexer/protocol_fees.rs` (`overview_fee_usd_field`) | | Vendored volume | `scripts/defillama/dexs/index.ts` | | Vendored fees | `scripts/defillama/fees/index.ts` | | Mapper + tests | `scripts/defillama/dimensions/mapDaily.js`, `mapDaily.test.js` | | Pins | `scripts/defillama/gems.js` | | Docs / skill | `docs/DEFILLAMA.md`, `skills/AGENTS_DEFILLAMA.md`, `docs/indexer-invariants.md` (DeFiLlama UTC-day + Economic fee USD) | | Verify | `scripts/qa/verify-issue-631.sh`, `Makefile` `verify-issue-631` / `verify-issue-683` | | Upstream | https://github.com/DefiLlama/dimension-adapters/pull/8987 (`dexs/cl8y-dex`, `fees/cl8y-dex`) | --- ## Recommended direction 1. **Indexer headline = EFee-6 partial SUM.** `daily_fees_usd` / `daily_revenue_usd` / `daily_protocol_revenue_usd` = SUM of priced source `amount_usd`. JSON `null` only when total `event_count > 0` and priced SUM is 0. Do **not** OR `unpriced_count` across sources into a whole-day null while wrap/window/hub sources are priced. Keep per-source `fees.*` fail-closed. 2. **Refresh after stamps.** Re-run `refresh_defillama_daily` (or a one-shot from adapter start) so Coolify rows pick up #683 NULL-only `fee_usd` backfill. Days with volume and true zero fee events stay `"0"`. 3. **Adapter `start`.** Set GitHub + `scripts/defillama/gems.js` `ADAPTER_START` to the first UTC day that returns **200** (today that is ~2026-08-17, not 2026-05-01). Document in `docs/DEFILLAMA.md`. Version stays **1** (cannot split hourly). 4. **Adapter errors.** Keep throw on HTTP 5xx / malformed JSON. Do not throw on JSON `"0"`. After (1), last-few-days tests must return numeric `dailyFees`. If a day is still all-unpriced (`null`), either throw with a stable message or omit the day — do **not** map unpriced → `$0`. Catch **404** only if `start` still precedes coverage; preferred fix is (3), not swallow-404-as-zero. 5. **Sync copies.** GitHub PR remains METRIC labeled + residual (no double-count). Fix `scripts/defillama/fees/index.ts` so it does **not** `addUSDValue(total)` then add breakdown again. `start` string vs unix: one canonical form in both trees. 6. **Docs.** #8987 is no longer “draft until Coolify”. State: leftover is null/404 vs crawler; close this issue when `pnpm test fees cl8y-dex` and `pnpm test dexs cl8y-dex` succeed for yesterday and `start`, and the GitHub PR is re-tested. 7. **Verify target.** Add `make verify-issue-<this-iid>` that covers indexer partial-SUM tests + mapper/adapter-start pins + `verify-issue-631` / `683` related. Keep `VERIFY_ISSUE_631_SKIP_RELATED` pattern if needed. --- ## Acceptance criteria 1. `GET /api/v1/defillama/daily?timestamp=<yesterday_00:00_utc>` returns **200** with numeric `daily_fees_usd` when any priced treasury fee exists that day (wrap/window/hub/economic). One unpriced source does **not** null the headline. 2. Headline `null` only when fee events exist and **none** have `fee_usd > 0`. Idle (no fee events) stays `"0"` even if volume is non-zero. 3. Per-source `fees.swap_amm` (etc.) may be `null` independently; SSR remains `"0"`; `daily_revenue_usd` equals `daily_fees_usd`. 4. Adapter `start` is the first 200 UTC day. `GET` for `start` is 200; `GET` for `start - 86400` is 404 (or start is the genesis row). Llama backfill does not walk May 2026 404s. 5. GitHub `fees/cl8y-dex` + in-repo copy: no throw on numeric / `"0"` days; no double-count of breakdown vs total; METRIC labels ⊆ `breakdownMethodology`. 6. `pnpm test fees cl8y-dex` and `pnpm test dexs cl8y-dex` (yesterday and `start`) succeed; output pasted on #8987 (operator). 7. `make verify-issue-<this-iid>` green on host Postgres. `make verify-issue-631` and `make verify-issue-683` stay green. 8. Docs/skill/invariants updated; #8987 draft language removed. --- ## Test plan (all paths) ### Indexer (Postgres) — `make setup-indexer-postgres` - Seed same UTC day: priced `swap_amm` (hub token) + priced `wrap` + **unpriced** CL8Y `limit_place` (`fee_usd` NULL) + gem-pair `swap_amm` (excluded). - Refresh daily rollup. - Headline `daily_fees_usd` = priced swap_amm + wrap **only** (partial SUM). `fees.limit_place` is `null`. Gem pair not in any source. - Same day, all fee rows unpriced → headline `null` (not `"0"`). Volume may still be a number. - Idle fee day with swaps → `daily_fees_usd` `"0"`, `volume_usd` priced or fail-closed on its own unpriced swaps. - Hybrid fill: `book_take` from `limit_order_fills.commission_amount` only (not swap-level `book_commission_amount`); volume still L10 parent swap. - Unaligned timestamp / injection / negative / future → **400**. Unknown day → **404**. Repeat GET within 60s does not rescan `protocol_fee_events`. - After NULL-only `fee_usd` backfill fixture, refresh flips headline from `null` to partial SUM; non-null stamps are not rewritten (**C568-1**). ### Mapper / vendored adapters (no Llama SDK) - `node --test scripts/defillama/dimensions/mapDaily.test.js` (and fees start pin). - `mapFees`: `"0"` → 0; JSON null → null; numeric string → number. SSR 0. Breakdown keys ⊆ methodology. - Fixture: `daily_fees_usd` 10, labeled 10 → no residual. Labeled 8 + total 10 → residual 2 **without** double-counting 10+8. - `dailyUrl(ADAPTER_START)` hits the **first 200 day**, not 2026-05-01 unless that day is 200. - Static: mapper never reads `liquidity_in_usd` / `total_liquidity_usd`. ### Upstream clone (operator) - `pnpm test fees cl8y-dex` yesterday UTC → numeric dailyFees, revenue = fees, SSR 0. - `pnpm test fees cl8y-dex <start>` → 200 path, no 404. - `pnpm test dexs cl8y-dex` same days → numeric dailyVolume. - Repeat “last few days” (yesterday, −1, −2, −3) — none throw. ### Live Coolify - Yesterday and `start` curl as in `scripts/defillama/README.md`. - A mid-lookback day that previously 404s (e.g. 2026-08-13) either stays 404 (if before start) or 200 after backfill — consistent with adapter `start`. --- ## Test plan (attack, hack, abuse) | ID | Vector | Expectation | |----|--------|-------------| | A1 | Publish CG `liquidity_in_usd` as Llama TVL/volume | Still forbidden; this ticket does not touch TVL adapter. | | A2 | Double-count hybrid fills into `dailyVolume` | Unchanged L10; volume tests still exclude `limit_order_fills`. | | A3 | Wash gem pairs to inflate Llama fees/volume | Gem-pair exclude unchanged on pair-linked sources. | | A4 | One unpriced CL8Y commission nulls wrap/window headline | **Must not** after this ticket (partial SUM). | | A5 | Map unpriced headline to `$0` so Llama stops throwing | Forbidden — silent undercount. `null` or skip, not zero. | | A6 | Count `spread_amount` / community-tax extra-debit as residual `dailyFees` | Residual only from already-ingested treasury sources; no new source invented to “balance” income statement. | | A7 | Reclassify residual as `dailySupplySideRevenue` | Forbidden. SSR stays 0. | | A8 | Swallow HTTP 404 as `$0` from May 2026 | Forbidden. Move `start` or backfill real rows. | | A9 | Symbol spoof `CL8Y` on a native/gem to mint economic `fee_usd` | EFee-1 contract pin; gems stay unpriced. | | A10 | Compromised indexer inflates `daily_fees_usd` | GET remains public read, rollup-only; TVL stays on-chain. No write methods. | | A11 | `timestamp` injection / unaligned / future | **400**; bind i64; no raw SQL. | | A12 | `from`/`to` range dump | Still ignored / 400 without single-day timestamp. | | A13 | Unbounded lookback refresh DoS | Refresh cap stays bounded (8d or documented start-backfill job, not GET). | | A14 | SSRF / user-supplied indexer URL in adapter | Host pin `indexer.dex.cl8y.com` only. | | A15 | CORS browser spoof of daily API | CORS allowlist unchanged; Llama is server-side. | | A16 | Double-count labeled METRIC + headline in `createBalances` | In-repo + GitHub fees adapters add labels **or** residual, not total+labels. | | A17 | Rewrite historical non-null `fee_usd` when the mark jumps | C568-1 / EFee-5 NULL-only backfill. | | A18 | Price vFDUSD / gems via CEX or Venus into Llama fees | X4 / EFee-3; still unpriced. | | A19 | Widen GET to live `SUM(protocol_fee_events)` | Forbidden (A11–A12 class). | | A20 | Adapter version 2 hourly split of the UTC-day rollup | Forbidden. Stay version 1. | --- ## Verification criteria Issue is **done** when all of the following are true: 1. Coolify `GET /api/v1/defillama/daily?timestamp=<yesterday_utc>` returns 200 with **numeric** `daily_fees_usd` whenever priced fee events exist; maintainer-style “last few days” do not return headline `null` solely because one source is unpriced. 2. Adapter `start` matches the first 200 UTC day; GitHub + `scripts/defillama/gems.js` agree. 3. Comment on [dimension-adapters#8987](https://github.com/DefiLlama/dimension-adapters/pull/8987) with `pnpm test fees cl8y-dex` and `pnpm test dexs cl8y-dex` logs (yesterday + start). Maintainer can re-run without throw. 4. `make verify-issue-<this-iid>` passes (`make setup-indexer-postgres`). Related `verify-issue-631` and `verify-issue-683` pass. 5. Docs/skill/invariants describe partial-SUM headline vs per-source fail-closed; #8987 is not documented as waiting on a missing Coolify route. 6. In-repo fees copy does not double-count; GitHub PR remains METRIC + residual; SSR is 0. **Close is blocked** if Llama fees tests still throw on recent days, if unpriced activity is published as `$0`, if SSR becomes non-zero, if `from`/`to` is added, or if TVL/volume start using indexer USD / CG `liquidity_in_usd`. --- ## Related - [#631](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/631) DeFiLlama UTC-day listing (closed) - [#683](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/683) economic `fee_usd` stamps (closed; EFee-6) - [#586](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/586) / [#613](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/613) / [#614](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/614) treasury fee ingest - [#562](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/562) gem hide - [#568](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/568) do not rewrite stamped USD - Upstream [dimension-adapters#8987](https://github.com/DefiLlama/dimension-adapters/pull/8987); TVL stays [DefiLlama-Adapters#20676](https://github.com/DefiLlama/DefiLlama-Adapters/pull/20676) (out of scope unless it shares the start-date pin)
PlasticDigits commented 2026-08-27 06:03:02 +00:00 (Migrated from gitlab.com)

marked as related to #631

marked as related to #631
PlasticDigits commented 2026-08-27 06:03:03 +00:00 (Migrated from gitlab.com)

marked as related to #683

marked as related to #683
PlasticDigits commented 2026-08-27 06:33:37 +00:00 (Migrated from gitlab.com)

mentioned in merge request !1187

mentioned in merge request !1187
PlasticDigits commented 2026-08-27 07:36:55 +00:00 (Migrated from gitlab.com)

mentioned in commit 8801b0fcf2

mentioned in commit 8801b0fcf28afb08e8aa0d802b85cb83263f1c20
PlasticDigits commented 2026-08-27 07:38:08 +00:00 (Migrated from gitlab.com)

mentioned in commit 6ae8b7270b

mentioned in commit 6ae8b7270b55e370142e6fe75e3d510bdcc5f54d
PlasticDigits commented 2026-08-27 07:43:02 +00:00 (Migrated from gitlab.com)

mentioned in issue #688

mentioned in issue #688
PlasticDigits commented 2026-08-27 07:43:04 +00:00 (Migrated from gitlab.com)

marked as related to #688

marked as related to #688
PlasticDigits commented 2026-08-27 07:43:22 +00:00 (Migrated from gitlab.com)

mentioned in issue #684

mentioned in issue #684
PlasticDigits commented 2026-08-27 07:43:23 +00:00 (Migrated from gitlab.com)

Merged as !1187 onto main (8801b0fc). In-repo AC for indexer partial-SUM headline, start 2026-08-17 / 1786924800, METRIC+residual, and make verify-issue-687 passed. Makefile conflict with #686 leftover verify resolved (kept both targets).

This issue stays open. Close gate is operator, not GitLab merge:

  1. Patch GitHub dimension-adapters#8987 (start: "2026-08-17", METRIC+residual; no throw on "0"; do not map JSON null → $0; do not swallow 404 as zero).
  2. Paste pnpm test fees cl8y-dex and pnpm test dexs cl8y-dex (yesterday + start) on #8987.
  3. Coolify indexer deploy, then yesterday + start curl per scripts/defillama/README.md.

Coolify + GitHub ops also listed on leftover #688. Do not reopen #631 / #683.

CI was failed (ci_quota_exceeded); not waited; not a product signal.

Merged as !1187 onto `main` (`8801b0fc`). In-repo AC for indexer partial-SUM headline, `start` `2026-08-17` / `1786924800`, METRIC+residual, and `make verify-issue-687` passed. Makefile conflict with #686 leftover verify resolved (kept both targets). **This issue stays open.** Close gate is operator, not GitLab merge: 1. Patch GitHub dimension-adapters#8987 (`start: "2026-08-17"`, METRIC+residual; no throw on `"0"`; do not map JSON `null` → `$0`; do not swallow 404 as zero). 2. Paste `pnpm test fees cl8y-dex` and `pnpm test dexs cl8y-dex` (yesterday + start) on #8987. 3. Coolify indexer deploy, then yesterday + start curl per `scripts/defillama/README.md`. Coolify + GitHub ops also listed on leftover #688. Do not reopen #631 / #683. CI was `failed` (`ci_quota_exceeded`); not waited; not a product signal.
PlasticDigits commented 2026-08-27 08:03:46 +00:00 (Migrated from gitlab.com)

Ops 2026-08-27 — leftover #688 / GitHub #8987

Indexer GET on Coolify already returns priced-SUM headlines (daily_fees_usd is a number or "0", not null). Prior-start day stays 404.

Patched upstream https://github.com/DefiLlama/dimension-adapters/pull/8987 6e87a48: start: "2026-08-17". METRIC groups + residual unchanged. Still throws on JSON null. Does not map HTTP 404 to $0.

pnpm test on that branch (also pasted on #8987):

pnpm test fees cl8y-dex
Start Date:	Wed, 26 Aug 2026 00:00:00 GMT
Daily fees: 90.00
Daily revenue: 90.00
Daily protocol revenue: 90.00
Daily supply side revenue: 0.00
Token Swap Fees 52 | Deposit/Withdraw Fees 31 | Mint/Redeem Fees 7.97

pnpm test dexs cl8y-dex
Daily volume: 1.51 k
Backfill start time: 17/8/2026

pnpm test fees cl8y-dex 2026-08-18
Daily fees: 0.00

pnpm test dexs cl8y-dex 2026-08-18
Daily volume: 717.00

Live: timestamp=1786924800 → 200, fees "0", volume ~716.51. timestamp=1786838400 → 404.

Local make verify-issue-687 green (includes 631 + 683).

Do not reopen #631 / #683. Close this issue after Llama merges #8987 (crawler test gate is green on our fork). Parent leftover: #688 (Coolify GT backfill still open).

## Ops 2026-08-27 — leftover #688 / GitHub #8987 Indexer GET on Coolify already returns priced-SUM headlines (`daily_fees_usd` is a number or `"0"`, not `null`). Prior-start day stays **404**. Patched upstream https://github.com/DefiLlama/dimension-adapters/pull/8987 `6e87a48`: `start: "2026-08-17"`. METRIC groups + residual unchanged. Still throws on JSON `null`. Does not map HTTP 404 to `$0`. `pnpm test` on that branch (also pasted on #8987): ``` pnpm test fees cl8y-dex Start Date: Wed, 26 Aug 2026 00:00:00 GMT Daily fees: 90.00 Daily revenue: 90.00 Daily protocol revenue: 90.00 Daily supply side revenue: 0.00 Token Swap Fees 52 | Deposit/Withdraw Fees 31 | Mint/Redeem Fees 7.97 pnpm test dexs cl8y-dex Daily volume: 1.51 k Backfill start time: 17/8/2026 pnpm test fees cl8y-dex 2026-08-18 Daily fees: 0.00 pnpm test dexs cl8y-dex 2026-08-18 Daily volume: 717.00 ``` Live: `timestamp=1786924800` → 200, fees `"0"`, volume ~716.51. `timestamp=1786838400` → 404. Local `make verify-issue-687` green (includes 631 + 683). Do **not** reopen #631 / #683. Close this issue after Llama merges #8987 (crawler test gate is green on our fork). Parent leftover: #688 (Coolify GT backfill still open).

/agent implement

/agent implement

/agent implement

/agent implement

/agent implement

/agent implement

/agent implement

/agent implement

Verified 2026-09-24. make verify-issue-687 passed 7/7 in an isolated Postgres database, including #631 and #683. Live GET: yesterday UTC and adapter start 1786924800 returned 200 (numeric fees / idle "0"); 1786838400 returned 404 as expected. GitHub dimension-adapters#8987 merged 2026-08-27. Newer ops tracker #688 is closed; no #687 verification remains. Related: #631, #683, #688. Documentation, invariant crosslinks, third-party playbook, and verifier assertion updated in local worktree branch verify/issue-687.

Verified 2026-09-24. `make verify-issue-687` passed 7/7 in an isolated Postgres database, including #631 and #683. Live GET: yesterday UTC and adapter start 1786924800 returned 200 (numeric fees / idle "0"); 1786838400 returned 404 as expected. GitHub dimension-adapters#8987 merged 2026-08-27. Newer ops tracker #688 is closed; no #687 verification remains. Related: #631, #683, #688. Documentation, invariant crosslinks, third-party playbook, and verifier assertion updated in local worktree branch verify/issue-687.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
code/cl8y-dex-terraclassic#687
No description provided.