code/cl8y-dex-terraclassic
4
0
Fork 0
Code Issues 35 Pull requests 6 Projects Releases Packages Wiki Activity

fix(indexer): persist every multihop AMM hop in protocol_fee_events #1269

New issue
Closed
opened 2026-09-16 05:41:48 +00:00 by PlasticDigits · 3 comments
PlasticDigits commented 2026-09-16 05:41:48 +00:00
Owner
Copy link

Summary

/protocol AMM commissions undercount multihop swaps. parse_swaps assigns swap_index per pair (restarts at 0 on the next pair — #287). process_swap then stores swap_amm fees with that index as ordinal into protocol_fee_events, whose uniqueness is UNIQUE (tx_hash, source, ordinal) and has no pair. Hop 2+ in the same router tx collide; ON CONFLICT DO NOTHING keeps the first hop and silently drops the rest.

swap_events already stores every hop via (tx_hash, pair_id, swap_index). Volume is right; treasury fee census is not.

Operator reconstruction of a recent trailing 7-day AMM window: on-chain commissions ≈ $176 vs dashboard ≈ $110. Dropping later hops that share (tx_hash, swap_amm, ordinal=0) closely reproduces the dashboard total.

Related (not this ticket): #287 (swap_events uniqueness — shipped), #586 (fee ledger that introduced the 3-column unique key), #316 / #331 (fill ↔ swap_index), #1209 / #1210 / #1211 (new sources — must inherit the widened key, not copy the colliding one).

Given / When / Then

Given a single router tx with two (or more) factory-listed hops on distinct pairs, each wasm action=swap carrying commission_amount > 0, and parse_swaps assigning swap_index = 0 to both hops
When the indexer runs process_swap → ingest_protocol_fee(FeeSource::SwapAmm, i64::from(swap.swap_index), …) against UNIQUE (tx_hash, source, ordinal)
Then both hop commissions persist as protocol_fee_events (source=swap_amm, event_count 2 for that tx)
And refresh_protocol_fees / GET /api/v1/protocol/fees?window=7d / overview total_fees_7d_usd include the sum of both stamped fee_usd values
And a poller replay of the same tx inserts zero extra fee rows

Given two genuine swaps on the same pair in one tx (swap_index 0 then 1 — #287)
When both have commission_amount > 0
Then both swap_amm rows persist (ordinals 0 and 1 must not collapse either)

Expected vs actual

Expected Actual
Unique key Distinguishes hops the way swap_events does: pair + per-pair swap_index (or a deterministic per-tx fee ordinal that never restarts per pair) UNIQUE (tx_hash, source, ordinal) — no pair
Ordinal for swap_amm Either global-in-tx or scoped by pair swap.swap_index from per_pair_swap_index (pair B restarts at 0)
2-hop tx, both commissions > 0 Two swap_amm rows One row; second ON CONFLICT DO NOTHING
7d AMM USD Sum of hop commissions (reconstructed ≈ $176 in the measured window) Dashboard ≈ $110; collision replay matches
Replay / reorg Still idempotent Idempotent and lossy

Current codebase

Layer Behavior today
Parser ordinal parse_swaps: per_pair_swap_index: HashMap<pair, i32> — each _contract_address starts at 0. Unit test parse_swaps_assigns_per_pair_swap_index: pairA→0, pairA→1, pairB→0.
Swap persist (#287) insert_swap / trade_exists unique (tx_hash, pair_id, swap_index). Multihop volume rows are complete.
Fee ingest process_swap after successful insert: ingest_protocol_fee(…, SwapAmm, i64::from(swap.swap_index), ask_asset_id, commission, …).
Fee schema (#586) indexer/migrations/20260821120000_protocol_fees.sql: UNIQUE (tx_hash, source, ordinal). No pair_id column. Comment: replay must not double-count.
Insert insert_fee_event: ON CONFLICT (tx_hash, source, ordinal) DO NOTHING. FeeEventDraft has no pair field.
Early-return trade_exists → Ok(()) before fee ingest. Historical hops already in swap_events will not grow missing fee rows on replay.
Other sources Wrap / UST1 window: per-tx sequential ordinal. book_take / limit_place: order_id as ordinal (usually unique in-tx). Collision is swap_amm × per-pair index.
Rollup ~5 min refresh_protocol_fees → global_stats_24h.total_fees_* + protocol_fee_stats_by_source (window 24h/7d/30d, incl. swap_amm). GET is O(1) / 60s cache — no live SUM.
UI ProtocolFeeStats headlines overview 7d; source table from GET /api/v1/protocol/fees. Dropped hops never enter the SUM.
L7 Hybrid still counts pool commission_amount + fill commission_amount once. Do not “fix” undercount by also adding book_commission_amount.

Router (smartcontracts/contracts/router) chains pair execute; each hop emits wasm action=swap + commission_amount (pool leg) under one txhash with distinct _contract_address.

Why the new implementation is needed

  1. #287 made swap persistence pair-aware. #586 reused the per-pair index as a per-tx fee ordinal. Multihop is the default Swap path (#101); every extra hop with swap_index == 0 is a dropped treasury row.
  2. /protocol and DeFiLlama daily fees are the public census. A ~38% 7d AMM gap (measured window) is not a display rounding bug.
  3. Replay cannot heal it: trade_exists skips ingest_protocol_fee. Need a uniqueness fix and a one-shot backfill from swap_events.commission_amount.
  4. Open source tickets (#1209 pair_creation, #1210 SKU invoices) copy UNIQUE (tx_hash, source, ordinal). Leaving the colliding key in place spreads the bug.

Constraints / guardrails

  • Do not ON CONFLICT DO UPDATE — replay/spoof must not overwrite a stored amount.
  • Do not count spread_amount, burn tax, gas, hook_fee_amount / AfterSwap, LP, book escrow, or book_commission_amount (L7 / #196). Treasury pool commission only for swap_amm.
  • #285: accept wasm only from reserved _contract_address that is a discovered factory pair. Unreserved contract_address / spoof hops are not fee rows.
  • Replay: duplicate delivery of the same hop still inserts 0 rows.
  • Wrap / UST1 / book / place: uniqueness must stay idempotent after the schema change. PostgreSQL UNIQUE treats NULL as distinct — a nullable pair_id without a partial unique / sentinel will break wrap replay dedup. Use one of: (a) pair_id NOT NULL with sentinel 0 for non-pair sources, (b) two partial uniques (pair_id IS NOT NULL vs IS NULL), (c) per-tx swap_amm ordinal (no pair column) that does not restart per pair.
  • GET stays O(1). Do not SUM protocol_fee_events on /overview or /protocol/fees. Refresh rollup after backfill.
  • Stamp fee_usd at ingest / backfill from the same catalog as volume (P522-Q / hub / economic marks #683). Do not rewrite historical non-null fee_usd from the live hub (#568). Unpriced → store raw, fee_usd NULL; activity + all unpriced → API null, not $0.
  • amount_raw > 0 still required. Zero-commission hops: no fee row.
  • Additive JSON. Do not break FeeSource::ALL length / CHECK until #1209 lands its own source.
  • Out of scope: changing on-chain bps; #1209/#1210/#1211 feature work; #1263 top-pairs volume table; #687 Llama adapter nulls; frontend chrome.

Relevant files

Path Why
indexer/src/indexer/parser.rs parse_swaps per-pair swap_index; process_swap passes it as fee ordinal; trade_exists early-return
indexer/src/indexer/protocol_fees.rs FeeEventDraft, FeeSource::SwapAmm
indexer/src/db/queries/protocol_fees.rs insert_fee_event ON CONFLICT (tx_hash, source, ordinal)
indexer/migrations/20260821120000_protocol_fees.sql Original unique key (do not edit in place — new migration)
indexer/migrations/20260605000000_swap_events_per_tx_pair_swap_index.sql #287 precedent
indexer/src/indexer/volume_aggregator.rs + db/queries/volume.rs refresh_protocol_fee_stats after backfill
indexer/src/api/protocol_fees.rs + api/overview.rs Still rollup-only
indexer/tests/indexer_protocol_fees.rs Ingest + rollup; add hop-collision case
indexer/src/indexer/parser.rs parse_swaps_assigns_per_pair_swap_index Documents pairB→0; extend with fee-ingest assertion
docs/indexer-invariants.md Protocol fees unique-key sentence
docs/runbooks/overview-global-stats-brin.md + indexer-reorg-replay-dedup.md Replay / backfill
skills/AGENTS_FRONTEND_PROTOCOL_STATS.md PFee uniqueness
scripts/qa/verify-issue-586.sh Keep green; add verify-issue-<this>
frontend-dapp/src/components/protocol/ProtocolFeeStats.tsx No copy change required unless labels lie

Recommended direction

Preferred (mirror #287, keep fee ↔ swap alignment):

  1. New migration: add pair_id INT NULL REFERENCES pairs(id) (or NOT NULL DEFAULT 0 sentinel). Replace UNIQUE (tx_hash, source, ordinal) with a key that includes pair for pair-scoped sources. Document NULL/sentinel semantics for wrap/window.
  2. FeeEventDraft + insert_fee_event bind pair_id. process_swap passes pair.id and existing swap.swap_index.
  3. Backfill: for each swap_events row with commission_amount > 0 and no matching swap_amm fee, insert one event (same USD helper as ingest). Do not rely on indexer replay (trade_exists will skip). Then one refresh_protocol_fees.
  4. Tests: two pairs, one tx, both swap_index == 0 → two fee rows; same-pair 0 then 1 → two rows; second insert of the same key → 0 rows; wrap two ordinals in one tx still dedup.

Simpler alternative: keep three-column unique; assign a per-tx swap_amm ordinal (global walk counter in parse_swaps / process_block_txs), independent of swap_index. Wrap already does this. Still needs backfill. Do not mix this with per-pair swap_index on the same unique key.

Do not use DO UPDATE.

Acceptance criteria

  • AC1. Router tx, two distinct pairs, both commission_amount > 0, both swap_index == 0 → COUNT(*) FILTER (source=swap_amm) for that tx_hash is 2.
  • AC2. Same-pair two swaps (swap_index 0, 1) with commissions → two swap_amm rows.
  • AC3. Duplicate delivery of one hop → still one row (ON CONFLICT DO NOTHING).
  • AC4. Wrap / unwrap / ust1_mint / ust1_redeem / book_take / limit_place replay uniqueness unchanged (no new double-count, no dropped legitimate rows).
  • AC5. After backfill + rollup, 7d swap_amm USD equals SUM(fee_usd) of surviving priced hop commissions (within existing clamp / unpriced rules). Collision-only reconstruction no longer matches the dashboard.
  • AC6. GET /overview and GET /protocol/fees still do not scan protocol_fee_events.
  • AC7. L7: hybrid tx does not double-count book + pool.
  • AC8. Docs/skills/invariants name the new unique key. make verify-issue-586 stays green. New make verify-issue-<this> exists.
  • AC9. #285 spoof hops still produce no fee row.

Test plan (functional paths)

# Path Expect
T1 Parse 3 wasm swaps: pairA, pairA, pairB swap_index 0, 1, 0 (existing). Fee ingest stores 3 swap_amm rows if all commissions > 0
T2 insert_fee_event twice with same (tx, swap_amm, pair, ordinal) Second rows_affected == 0
T3 Two drafts, same tx/source/ordinal, different pair Both persist
T4 process_block_txs fixture: 2-hop router wasm with commissions protocol_fee_events count 2; swap_events count 2
T5 Replay T4 Counts unchanged
T6 Rollup 7d after T4 protocol_fee_stats_by_source swap_amm event_count + USD include both hops
T7 commission_amount = 0 hop No fee row
T8 Wrap two fees, same tx, ordinal 0 then 1 Both persist; replay deduped
T9 Backfill idempotent on already-complete txs No duplicate swap_amm
T10 hybrid_counts_amm_and_book_once Still green

Vitest/RTL not required (indexer census). Postgres tests: indexer_protocol_fees.rs + parser unit + optional process_block fixture.

Test plan (attack, hack, and abuse)

# Vector Expect
A1 Replay / reorg re-delivery of a multihop tx No second swap_amm set; amounts unchanged
A2 ON CONFLICT DO UPDATE Forbidden; would let a later payload replace treasury USD
A3 Unreserved contract_address (no underscore) spoof swap + commission (#285) No pair discover → no fee row
A4 Hostile / huge swap_index wasm attr Fail closed or clamp with existing parse; no unique-key integer overflow panic; no overwrite of another hop
A5 Crafted second hop that intends to collide (same ordinal, omit pair) After fix, cannot drop the first hop’s amount
A6 Nullable pair_id unique (PG NULLS DISTINCT) Must not allow two wrap rows with identical (tx, source, ordinal)
A7 Backfill double-run Unique key stops a second insert
A8 Count spread / hook / book_commission as swap_amm to “close the gap” Forbidden (L7 / PFee)
A9 Live SUM(protocol_fee_events) on GET to hide missing ingest Forbidden (DoS / V5)
A10 Gem / vFDUSD identity in backfill USD Same omit rules as #683 / economic marks
A11 Pair-creation / SKU parsers copying the old 3-col unique (#1209) Do not land that copy after this ships

Verification criteria

  • New make verify-issue-<this>: parser unit (pairB→0 still, fees do not collapse), indexer_protocol_fees hop insert + replay, unique-constraint grep (pair or per-tx ordinal — not the old three-column-only comment as the sole truth), docs/skills.
  • make verify-issue-586, verify-issue-613, verify-issue-614, verify-issue-683 stay green.
  • Manual: after indexer deploy + backfill + one aggregator tick, 7d AMM on /protocol matches a hop-complete SUM from swap_events.commission_amount (priced), not the collision-truncated sum.
  • Coolify/indexer restart: poller replay does not inflate fee_event_count.

Out of scope

  • #1209 pair_creation / #1210 SKU / #1211 cohort (stack after uniqueness).
  • #1263 top-5 volume/TVL table.
  • #687 DeFiLlama adapter null throw.
  • Changing pair commission bps or router hop count.
  • Frontend layout of ProtocolFeeStats.
  • Reopening #287 or #586 (those tickets shipped their original ACs).

First-pass model recommendation

Recommendation: grok-high

Rationale: This is a persisted uniqueness + backfill change across schema (protocol_fee_events), ingest (parser.rs / FeeEventDraft / insert_fee_event), aggregator refresh, invariants/runbook, and focused Postgres tests — more than three production files and not a local helper edit. Wrong NULL unique semantics would double-count wrap/window on replay; wrong backfill would fork /protocol and DeFiLlama. Comparable to control-plane work that crosses persisted state (not a single-file docs/test pass). Verify with the new verify-issue-* plus indexer_protocol_fees / parser ordinal tests; no live host choice in this ticket.

## Summary `/protocol` AMM commissions undercount multihop swaps. `parse_swaps` assigns `swap_index` **per pair** (restarts at 0 on the next pair — #287). `process_swap` then stores `swap_amm` fees with that index as `ordinal` into `protocol_fee_events`, whose uniqueness is `UNIQUE (tx_hash, source, ordinal)` and has **no pair**. Hop 2+ in the same router tx collide; `ON CONFLICT DO NOTHING` keeps the first hop and silently drops the rest. `swap_events` already stores every hop via `(tx_hash, pair_id, swap_index)`. Volume is right; treasury fee census is not. Operator reconstruction of a recent trailing 7-day AMM window: on-chain commissions ≈ **$176** vs dashboard ≈ **$110**. Dropping later hops that share `(tx_hash, swap_amm, ordinal=0)` closely reproduces the dashboard total. Related (not this ticket): #287 (swap_events uniqueness — shipped), #586 (fee ledger that introduced the 3-column unique key), #316 / #331 (fill ↔ swap_index), #1209 / #1210 / #1211 (new sources — must inherit the **widened** key, not copy the colliding one). ### Given / When / Then Given a single router tx with two (or more) factory-listed hops on **distinct** pairs, each wasm `action=swap` carrying `commission_amount > 0`, and `parse_swaps` assigning `swap_index = 0` to both hops When the indexer runs `process_swap` → `ingest_protocol_fee(FeeSource::SwapAmm, i64::from(swap.swap_index), …)` against `UNIQUE (tx_hash, source, ordinal)` Then **both** hop commissions persist as `protocol_fee_events` (`source=swap_amm`, `event_count` 2 for that tx) And `refresh_protocol_fees` / `GET /api/v1/protocol/fees?window=7d` / overview `total_fees_7d_usd` include the sum of both stamped `fee_usd` values And a poller replay of the same tx inserts **zero** extra fee rows Given two genuine swaps on the **same** pair in one tx (`swap_index` 0 then 1 — #287) When both have `commission_amount > 0` Then both `swap_amm` rows persist (ordinals 0 and 1 must not collapse either) ### Expected vs actual | | Expected | Actual | | --- | --- | --- | | Unique key | Distinguishes hops the way `swap_events` does: pair + per-pair `swap_index` (or a deterministic per-tx fee ordinal that never restarts per pair) | `UNIQUE (tx_hash, source, ordinal)` — no pair | | Ordinal for `swap_amm` | Either global-in-tx or scoped by pair | `swap.swap_index` from `per_pair_swap_index` (pair B restarts at 0) | | 2-hop tx, both commissions > 0 | Two `swap_amm` rows | One row; second `ON CONFLICT DO NOTHING` | | 7d AMM USD | Sum of hop commissions (reconstructed ≈ $176 in the measured window) | Dashboard ≈ $110; collision replay matches | | Replay / reorg | Still idempotent | Idempotent **and** lossy | ## Current codebase | Layer | Behavior today | | --- | --- | | Parser ordinal | `parse_swaps`: `per_pair_swap_index: HashMap<pair, i32>` — each `_contract_address` starts at 0. Unit test `parse_swaps_assigns_per_pair_swap_index`: pairA→0, pairA→1, **pairB→0**. | | Swap persist (#287) | `insert_swap` / `trade_exists` unique `(tx_hash, pair_id, swap_index)`. Multihop volume rows are complete. | | Fee ingest | `process_swap` after successful insert: `ingest_protocol_fee(…, SwapAmm, i64::from(swap.swap_index), ask_asset_id, commission, …)`. | | Fee schema (#586) | `indexer/migrations/20260821120000_protocol_fees.sql`: `UNIQUE (tx_hash, source, ordinal)`. **No `pair_id` column.** Comment: replay must not double-count. | | Insert | `insert_fee_event`: `ON CONFLICT (tx_hash, source, ordinal) DO NOTHING`. `FeeEventDraft` has no pair field. | | Early-return | `trade_exists` → `Ok(())` **before** fee ingest. Historical hops already in `swap_events` will **not** grow missing fee rows on replay. | | Other sources | Wrap / UST1 window: per-tx sequential `ordinal`. `book_take` / `limit_place`: `order_id` as ordinal (usually unique in-tx). Collision is `swap_amm` × per-pair index. | | Rollup | ~5 min `refresh_protocol_fees` → `global_stats_24h.total_fees_*` + `protocol_fee_stats_by_source` (`window` 24h/7d/30d, incl. `swap_amm`). GET is O(1) / 60s cache — no live `SUM`. | | UI | `ProtocolFeeStats` headlines overview 7d; source table from `GET /api/v1/protocol/fees`. Dropped hops never enter the SUM. | | L7 | Hybrid still counts pool `commission_amount` + fill `commission_amount` once. Do not “fix” undercount by also adding `book_commission_amount`. | Router (`smartcontracts/contracts/router`) chains pair `execute`; each hop emits wasm `action=swap` + `commission_amount` (pool leg) under one `txhash` with distinct `_contract_address`. ## Why the new implementation is needed 1. `#287` made swap persistence pair-aware. `#586` reused the **per-pair** index as a **per-tx** fee ordinal. Multihop is the default Swap path (#101); every extra hop with `swap_index == 0` is a dropped treasury row. 2. `/protocol` and DeFiLlama daily fees are the public census. A ~38% 7d AMM gap (measured window) is not a display rounding bug. 3. Replay cannot heal it: `trade_exists` skips `ingest_protocol_fee`. Need a uniqueness fix **and** a one-shot backfill from `swap_events.commission_amount`. 4. Open source tickets (#1209 pair_creation, #1210 SKU invoices) copy `UNIQUE (tx_hash, source, ordinal)`. Leaving the colliding key in place spreads the bug. ## Constraints / guardrails - **Do not** `ON CONFLICT DO UPDATE` — replay/spoof must not overwrite a stored amount. - **Do not** count `spread_amount`, burn tax, gas, `hook_fee_amount` / AfterSwap, LP, book escrow, or `book_commission_amount` (L7 / #196). Treasury pool commission only for `swap_amm`. - **#285:** accept wasm only from reserved `_contract_address` that is a discovered factory pair. Unreserved `contract_address` / spoof hops are not fee rows. - **Replay:** duplicate delivery of the **same** hop still inserts 0 rows. - **Wrap / UST1 / book / place:** uniqueness must stay idempotent after the schema change. PostgreSQL `UNIQUE` treats `NULL` as distinct — a nullable `pair_id` **without** a partial unique / sentinel will **break** wrap replay dedup. Use one of: (a) `pair_id NOT NULL` with sentinel `0` for non-pair sources, (b) two partial uniques (`pair_id IS NOT NULL` vs `IS NULL`), (c) per-tx `swap_amm` ordinal (no pair column) that does not restart per pair. - **GET stays O(1).** Do not `SUM protocol_fee_events` on `/overview` or `/protocol/fees`. Refresh rollup after backfill. - **Stamp `fee_usd` at ingest / backfill** from the same catalog as volume (P522-Q / hub / economic marks #683). Do not rewrite historical non-null `fee_usd` from the live hub (#568). Unpriced → store raw, `fee_usd` NULL; activity + all unpriced → API `null`, not `$0`. - **`amount_raw > 0`** still required. Zero-commission hops: no fee row. - Additive JSON. Do not break `FeeSource::ALL` length / CHECK until #1209 lands its own source. - Out of scope: changing on-chain bps; #1209/#1210/#1211 feature work; #1263 top-pairs volume table; #687 Llama adapter nulls; frontend chrome. ## Relevant files | Path | Why | | --- | --- | | `indexer/src/indexer/parser.rs` | `parse_swaps` per-pair `swap_index`; `process_swap` passes it as fee `ordinal`; `trade_exists` early-return | | `indexer/src/indexer/protocol_fees.rs` | `FeeEventDraft`, `FeeSource::SwapAmm` | | `indexer/src/db/queries/protocol_fees.rs` | `insert_fee_event` `ON CONFLICT (tx_hash, source, ordinal)` | | `indexer/migrations/20260821120000_protocol_fees.sql` | Original unique key (do not edit in place — new migration) | | `indexer/migrations/20260605000000_swap_events_per_tx_pair_swap_index.sql` | #287 precedent | | `indexer/src/indexer/volume_aggregator.rs` + `db/queries/volume.rs` | `refresh_protocol_fee_stats` after backfill | | `indexer/src/api/protocol_fees.rs` + `api/overview.rs` | Still rollup-only | | `indexer/tests/indexer_protocol_fees.rs` | Ingest + rollup; add hop-collision case | | `indexer/src/indexer/parser.rs` `parse_swaps_assigns_per_pair_swap_index` | Documents pairB→0; extend with fee-ingest assertion | | `docs/indexer-invariants.md` | Protocol fees unique-key sentence | | `docs/runbooks/overview-global-stats-brin.md` + `indexer-reorg-replay-dedup.md` | Replay / backfill | | `skills/AGENTS_FRONTEND_PROTOCOL_STATS.md` | PFee uniqueness | | `scripts/qa/verify-issue-586.sh` | Keep green; add `verify-issue-<this>` | | `frontend-dapp/src/components/protocol/ProtocolFeeStats.tsx` | No copy change required unless labels lie | ## Recommended direction **Preferred (mirror #287, keep fee ↔ swap alignment):** 1. New migration: add `pair_id INT NULL REFERENCES pairs(id)` (or `NOT NULL DEFAULT 0` sentinel). Replace `UNIQUE (tx_hash, source, ordinal)` with a key that includes pair for pair-scoped sources. Document NULL/sentinel semantics for wrap/window. 2. `FeeEventDraft` + `insert_fee_event` bind `pair_id`. `process_swap` passes `pair.id` and existing `swap.swap_index`. 3. **Backfill:** for each `swap_events` row with `commission_amount > 0` and no matching `swap_amm` fee, insert one event (same USD helper as ingest). Do not rely on indexer replay (`trade_exists` will skip). Then one `refresh_protocol_fees`. 4. Tests: two pairs, one tx, both `swap_index == 0` → two fee rows; same-pair 0 then 1 → two rows; second insert of the same key → 0 rows; wrap two ordinals in one tx still dedup. **Simpler alternative:** keep three-column unique; assign a **per-tx** `swap_amm` ordinal (global walk counter in `parse_swaps` / `process_block_txs`), independent of `swap_index`. Wrap already does this. Still needs backfill. Do **not** mix this with per-pair `swap_index` on the same unique key. Do not use `DO UPDATE`. ## Acceptance criteria - AC1. Router tx, two distinct pairs, both `commission_amount > 0`, both `swap_index == 0` → `COUNT(*) FILTER (source=swap_amm)` for that `tx_hash` is **2**. - AC2. Same-pair two swaps (`swap_index` 0, 1) with commissions → two `swap_amm` rows. - AC3. Duplicate delivery of one hop → still one row (`ON CONFLICT DO NOTHING`). - AC4. Wrap / unwrap / ust1_mint / ust1_redeem / book_take / limit_place replay uniqueness unchanged (no new double-count, no dropped legitimate rows). - AC5. After backfill + rollup, 7d `swap_amm` USD equals `SUM(fee_usd)` of surviving priced hop commissions (within existing clamp / unpriced rules). Collision-only reconstruction no longer matches the dashboard. - AC6. `GET /overview` and `GET /protocol/fees` still do not scan `protocol_fee_events`. - AC7. L7: hybrid tx does not double-count book + pool. - AC8. Docs/skills/invariants name the new unique key. `make verify-issue-586` stays green. New `make verify-issue-<this>` exists. - AC9. #285 spoof hops still produce no fee row. ## Test plan (functional paths) | # | Path | Expect | | --- | --- | --- | | T1 | Parse 3 wasm swaps: pairA, pairA, pairB | `swap_index` 0, 1, 0 (existing). Fee ingest stores **3** `swap_amm` rows if all commissions > 0 | | T2 | `insert_fee_event` twice with same `(tx, swap_amm, pair, ordinal)` | Second `rows_affected == 0` | | T3 | Two drafts, same tx/source/ordinal, **different** pair | Both persist | | T4 | `process_block_txs` fixture: 2-hop router wasm with commissions | `protocol_fee_events` count 2; `swap_events` count 2 | | T5 | Replay T4 | Counts unchanged | | T6 | Rollup 7d after T4 | `protocol_fee_stats_by_source` `swap_amm` `event_count` + USD include both hops | | T7 | `commission_amount = 0` hop | No fee row | | T8 | Wrap two fees, same tx, ordinal 0 then 1 | Both persist; replay deduped | | T9 | Backfill idempotent on already-complete txs | No duplicate `swap_amm` | | T10 | `hybrid_counts_amm_and_book_once` | Still green | Vitest/RTL not required (indexer census). Postgres tests: `indexer_protocol_fees.rs` + parser unit + optional process_block fixture. ## Test plan (attack, hack, and abuse) | # | Vector | Expect | | --- | --- | --- | | A1 | Replay / reorg re-delivery of a multihop tx | No second `swap_amm` set; amounts unchanged | | A2 | `ON CONFLICT DO UPDATE` | Forbidden; would let a later payload replace treasury USD | | A3 | Unreserved `contract_address` (no underscore) spoof swap + commission (#285) | No pair discover → no fee row | | A4 | Hostile / huge `swap_index` wasm attr | Fail closed or clamp with existing parse; no unique-key integer overflow panic; no overwrite of another hop | | A5 | Crafted second hop that **intends** to collide (same ordinal, omit pair) | After fix, cannot drop the first hop’s amount | | A6 | Nullable `pair_id` unique (PG NULLS DISTINCT) | Must not allow two wrap rows with identical `(tx, source, ordinal)` | | A7 | Backfill double-run | Unique key stops a second insert | | A8 | Count spread / hook / book_commission as `swap_amm` to “close the gap” | Forbidden (L7 / PFee) | | A9 | Live `SUM(protocol_fee_events)` on GET to hide missing ingest | Forbidden (DoS / V5) | | A10 | Gem / vFDUSD identity in backfill USD | Same omit rules as #683 / economic marks | | A11 | Pair-creation / SKU parsers copying the old 3-col unique (#1209) | Do not land that copy after this ships | ## Verification criteria - New `make verify-issue-<this>`: parser unit (pairB→0 still, **fees** do not collapse), `indexer_protocol_fees` hop insert + replay, unique-constraint grep (pair or per-tx ordinal — not the old three-column-only comment as the sole truth), docs/skills. - `make verify-issue-586`, `verify-issue-613`, `verify-issue-614`, `verify-issue-683` stay green. - Manual: after indexer deploy + backfill + one aggregator tick, 7d AMM on `/protocol` matches a hop-complete `SUM` from `swap_events.commission_amount` (priced), not the collision-truncated sum. - Coolify/indexer restart: poller replay does not inflate `fee_event_count`. ## Out of scope - #1209 pair_creation / #1210 SKU / #1211 cohort (stack after uniqueness). - #1263 top-5 volume/TVL table. - #687 DeFiLlama adapter `null` throw. - Changing pair commission bps or router hop count. - Frontend layout of `ProtocolFeeStats`. - Reopening #287 or #586 (those tickets shipped their original ACs). ## First-pass model recommendation Recommendation: grok-high Rationale: This is a persisted uniqueness + backfill change across schema (`protocol_fee_events`), ingest (`parser.rs` / `FeeEventDraft` / `insert_fee_event`), aggregator refresh, invariants/runbook, and focused Postgres tests — more than three production files and not a local helper edit. Wrong NULL unique semantics would double-count wrap/window on replay; wrong backfill would fork `/protocol` and DeFiLlama. Comparable to control-plane work that crosses persisted state (not a single-file docs/test pass). Verify with the new `verify-issue-*` plus `indexer_protocol_fees` / parser ordinal tests; no live host choice in this ticket.
PlasticDigits commented 2026-09-16 20:39:21 +00:00
Author
Owner
Copy link

/agent implement

/agent implement
PlasticDigits commented 2026-09-16 20:39:23 +00:00
Author
Owner
Copy link

cl8y-agent-control: queued implement job f87a932f-0734-4a30-bb0f-24e6b4cde642 (not executed; no Hetzner VM).

cl8y-agent-control: queued `implement` job `f87a932f-0734-4a30-bb0f-24e6b4cde642` (not executed; no Hetzner VM).
PlasticDigits commented 2026-09-21 04:03:03 +00:00
Author
Owner
Copy link

Design record PR #1275 (ADR 0005) merged to main (16dfa302). Ingest uniqueness from #1269 / PR #1274 was already on main. sqlx leftover (shared 20260916120000 prefix) was resolved in PR #1282: USDT stays 20260916120000, hops file is 20260916120001.

Design record PR #1275 (ADR 0005) merged to main (`16dfa302`). Ingest uniqueness from #1269 / PR #1274 was already on main. sqlx leftover (shared `20260916120000` prefix) was resolved in PR #1282: USDT stays `20260916120000`, hops file is `20260916120001`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/tokio-util-0.x-lockfile
renovate/thiserror-2.x-lockfile
renovate/serde_json-1.x-lockfile
renovate/serde-monorepo
renovate/libc-0.x-lockfile
renovate/chrono-0.x-lockfile
cac-design-issue-1206
cac-design-issue-1317
cac-design-issue-1316
cac-design-issue-1311
cac-design-issue-1300
cac-design-issue-1306
cac-design-issue-1305
cac-design-issue-1302
cac-design-issue-1277
No results found.
Labels
Clear labels   
agent:fix_bugfix

   
agent:fix_conflicts

   
agent:fix_security

   
agent:gap_analysis

   
agent:implement

   
agent:implement
Queue implement gate

  
agent:implement
Queue implement gate

  
agent:open_issues

   
agent:ready

   
agent:research

   
agent:security_audit

   
agent:verify

   
architecture
Architecture

  
backend
API and services

  
blocker:hybrid
Blocks hybrid launch

  
blocker:launch
Blocks launch

  
blocker:limit-orders
Blocks limits launch

  
blocker:v2
Blocks v2

  
block:log_only

   
block:security

   
bug

   
ci

   
contracts
Smart contracts

  
correctness

   
deploy

   
dev

   
devops

   
docs

   
documentation

   
duplicate

   
e2e

   
enhancement

   
epic
Epic

  
feature

   
frontend

   
functional-completion
Functional

  
gas

   
good first issue

   
governance

   
help wanted

   
high-risk

   
hooks

   
hybrid
Hybrid swaps

  
indexer
Indexer

  
infra
CI/CD

  
infrastructure

   
integrators

   
invalid

   
launch-blocker

   
limit-orders
Limit orders

  
localnet

   
localterra

   
low priority

   
missing-implementation
Implementation

  
needs-design
Design decision required

  
ops

   
performance
  
priority
high

  
priority
medium

  
product

   
qa

   
QA

   
question

   
ready
Definition of Ready

  
ready
Definition of Ready

  
research

   
scripts

   
security
Security

  
security-hardening
Security hardening

  
smartcontracts

   
tech-debt

   
testing
Testing

  
ux

   
UX

   
v2
Pool-only swaps

  
verification

   
wontfix
No labels
agent:fix_bugfix
agent:fix_conflicts
agent:fix_security
agent:gap_analysis
agent:implement
agent:implement
agent:implement
agent:open_issues
agent:ready
agent:research
agent:security_audit
agent:verify
architecture
backend
blocker:hybrid
blocker:launch
blocker:limit-orders
blocker:v2
block:log_only
block:security
bug
ci
contracts
correctness
deploy
dev
devops
docs
documentation
duplicate
e2e
enhancement
epic
feature
frontend
functional-completion
gas
good first issue
governance
help wanted
high-risk
hooks
hybrid
indexer
infra
infrastructure
integrators
invalid
launch-blocker
limit-orders
localnet
localterra
low priority
missing-implementation
needs-design
ops
performance
priority
high
priority
medium
product
qa
QA
question
ready
ready
research
scripts
security
security-hardening
smartcontracts
tech-debt
testing
ux
UX
v2
verification
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
code/cl8y-dex-terraclassic#1269
No description provided.