code/cl8y-dex-terraclassic

Fork 0

feat(indexer): redacted UTC-day evidence JSON tagged swap/wrap/limit/LP #1205

New issue

Closed

opened 2026-09-04 08:07:48 +00:00 by PlasticDigits · 4 comments

PlasticDigits commented

2026-09-04 08:07:48 +00:00

Owner

Summary

Add one read-only indexer HTTP export that returns a UTC calendar-day JSON of already-indexed on-chain activity, each row tagged surface ∈ {swap, wrap, limit, lp}, with user bech32 redacted. Research and incident-triage agents today must fan out across pair/trader tapes, /gt/events, and fee rollups, then strip wallets themselves. That is not an export.

Bundle (do not split):

GET /api/v1/evidence/daily — required day=YYYY-MM-DD (UTC [00:00Z, +1d)), optional surface= allowlist, cursor pagination, bounded page size.
Surface tags — swap from swap_events; lp from liquidity_events; limit from placements + cancellations + fills; wrap from protocol_fee_events source ∈ {wrap, unwrap}.
Redaction — never emit sender, receiver, maker, owner, or provider. Emit a non-reversible actor_hash when the source row has an actor. Pair/token contracts stay.
OpenAPI + tests — utoipa path on ApiDoc, handler tests for every surface and the abuse matrix below.

This is a new business endpoint on the existing ingest. No frontend. No wrap-principal ingest. No CSV. No API keys.

Related (do not merge into this ticket):

#1204 — docs/OpenAPI pack for existing swaps/pools/fees/burns/windows routes. Explicitly “no new business endpoints.”
#1202 — dApp product events (campaign → tx_succeeded). Off-chain, unredacted by design, not an on-chain tape.
#694 / #646 / #684 — GeckoTerminal /gt/events (swap+join/exit, unredacted maker, block window, 5000-row 400).
#631 — DeFiLlama aggregates on UTC day, not per-event JSON.
#216 L10 — headline volume is the parent swap_events row; do not also count limit_order_fills as swap.
#586 / #614 — wrap-mapper fees (notify_deposit / unwrap) into protocol_fee_events only.
#432 — CSV formula prefix (this ticket is JSON-only).
#557 — amounts as plain integer digit strings.
Operator SQL in docs/runbooks/suspicious-activity-queries.md stays the unredacted incident path.

Current codebase

There is no unified event stream, no surface enum, and no address redaction on JSON. Events live in separate tables and pair/trader routes. Daily HTTP exists only as aggregates.

Tapes are pair- or trader-scoped and unredacted

Surface wanted	Storage	Live HTTP	Actor field
swap	`swap_events`	`GET /api/v1/pairs/{addr}/trades`, `GET /api/v1/traders/{addr}/trades`	`sender` (+ optional `receiver`)
lp	`liquidity_events` (`event_type` `add` \| `remove`)	`GET /api/v1/pairs/{addr}/liquidity-events`	`provider`
limit	`limit_order_placements`, `limit_order_cancellations`, `limit_order_fills`	pair + trader `limit-placements` / `limit-cancellations` / `limit-fills`	`owner` / `maker`
wrap	`protocol_fee_events.source` = `wrap` \| `unwrap`	fee rollups only (`/api/v1/protocol/fees`, DeFiLlama daily)	none (no actor column)

Router: indexer/src/api/mod.rs build_router. Pair handlers: indexer/src/api/pairs.rs. Trader handlers: indexer/src/api/traders.rs. Rows: indexer/src/db/queries/swap_events.rs (SwapEventRow.sender), liquidity.rs (provider), limit_order_fills.rs (maker).

GET /api/v1/traders/{addr}/trades?format=csv is a wallet-scoped history dump with the full sender column (#163). It is not a protocol-wide daily export and it is not redacted.

`/gt/events` is a listing adapter, not evidence

indexer/src/api/gt.rs: eventType swap | join | exit, full maker, inclusive fromBlock/toBlock (max 2000 blocks), combined rows ≤ MAX_GT_EVENT_ROWS 5000 else 400 (GT_EVENT_ROW_CAP_MSG). No limit book, no wrap. Gems omitted (L639-2). Reserves are persisted post-event columns, never live pair_reserves (#684).

Wrap is a fee source, not a wrap tape

indexer/src/indexer/protocol_fees.rs: FeeSource::{Wrap, Unwrap, …}. Retail wrap wasm is pinned-mapper action=notify_deposit; unwrap is action=unwrap; amount key fee (not principal). Insert: indexer/src/db/queries/protocol_fees.rs protocol_fee_events (block_height, block_timestamp, tx_hash, source, ordinal, asset_id, amount_raw, decimals, fee_usd) — no sender. Spoof notify_deposit off the pin is ignored. UST1 mint/redeem is a different family (ust1_mint / ust1_redeem) and is not wrap.

Daily HTTP is aggregates

GET /api/v1/defillama/daily?timestamp= (#631) and GET /api/v1/protocol/{volume,fees,liquidity}/daily are UTC-day sums. Volume rule: parent swap_events only — never fills, wrap, or UST1 window. They are not event dumps.

Redaction exists only in logs

indexer/src/lcd/mod.rs redacts LCD paths at WARN. API JSON returns full bech32. No sha256 actor alias helper.

Auth and governors

Public GET. Global tower_governor (default 60 RPS; prod refuses 0). LCD-heavy router is a second 10 RPS list (limit-book, route/solve, blacklist-check, CG/CMC orderbook). Keys are socket peer IP (PeerIpKeyExtractor) — no trusted X-Forwarded-For. SQL list limit is clamped (SEC-F05 / #431). Internal errors: internal_err() → "Internal server error".

Docs / discovery today

docs/runbooks/suspicious-activity-queries.md tells operators to curl unredacted leaderboard + trader trades or raw SQL. That is the opposite of a shareable daily blob. #1204 will document existing paths; it will not add this route.

Why the new implementation is needed

Evidence consumers need one day’s protocol activity as JSON they can store and share without a wallet list. Today that requires:

Enumerating pairs, paging /trades + /liquidity-events + three limit routes per pair.
Separately reading wrap fee rollups that are not an event tape.
Manually mapping GT join/exit vs indexer add/remove vs limit fills that must not be double-counted as swaps (L10).
Stripping sender / maker / owner / provider after the fact.

Without a dedicated export, agents either keep PII, miss wrap/limit, or mix listing adapters (/gt/events) with protocol truth. A UTC-day, surface-tagged, redacted document is the missing contract.

Constraints / guardrails

Read-only. GET only. No indexer DB writes. Not on lcd_heavy_router (Postgres only). No LCD on the request path.
No new ingest. Do not add wrap_events. Do not persist wrap principal. Do not re-parse historical wasm. Export what is already indexed.
No API keys / bearer / ?redact=0. The public API stays unauthenticated. Redaction is the access control for this route. Do not add a secret salt (new key material). Do not add an unredacted twin.
One UTC calendar day. Same clock as #631 ([day 00:00:00Z, next 00:00:00Z)). No multi-day from/to. No trailing 24h. day is YYYY-MM-DD parsed in Rust; bind timestamptz parameters — never concatenate SQL.
Surface allowlist. swap | wrap | limit | lp only (lowercase after trim). Repeatable surface= or comma list. Unknown / empty token after split → 400. Default: all four.
L10. surface=swap is swap_events only. Limit fills are surface=limit + kind=fill. Do not emit a swap row and a fill row that an agent would sum as two volumes. Optional swap_event_id on fill rows is correlation, not a second swap.
Wrap honesty. Wrap rows are treasury fee events (amount_raw = indexed fee). Label kind wrap | unwrap from source. Omit actor_hash when the table has no actor. Do not fold ust1_mint / ust1_redeem / swap_amm / book_take / limit_place into wrap.
LP kinds. kind add | remove from liquidity_events.event_type. Do not rename to GT join/exit on this route.
Redaction. SHA-256 of the canonical actor bech32 (trim only; do not case-fold terra1). Wire actor_hash = first 32 hex chars (16 bytes). Drop actor fields entirely (not null). Pair address, token contracts, native denoms (uusd / uluna) stay. tx_hash stays so the blob is chain-linkable; document that LCD lookup deanonymizes a row. That tradeoff is accepted for evidence; the goal is “no wallet list in the JSON,” not cryptographic unlinkability.
Pagination, not GT 400-over-cap. Busy days must be exportable. Default limit=500, clamp 1–1000. Opaque cursor / next_cursor on (block_height, tx_hash, surface, kind, ordinal). has_more when another page exists. Do not 400 when the day has >5000 events.
Amounts. Plain integer digit strings (bd_plain_string / #557). No scientific notation. Optional decimals when known. fee_usd on wrap may be JSON number/string consistent with fee APIs, or omitted if unpriced — do not emit null that a client treats as $0 without a field.
Errors. Bad day / future day (strictly after today’s UTC date) / bad surface / bad cursor / bad limit → 400 with a short message. DB failure → 500 "Internal server error". Empty day → 200 { "events": [], "has_more": false }. Today’s incomplete UTC day → 200 with "complete": false.
Do not mutate /gt/events, pair/trader tapes, DeFiLlama, or CSV. Those stay unredacted.
Gems. Include indexed gem/test-pair activity (this is evidence, not a listing adapter). Document that the dump is not L639-safe.
No CSV on this route (format=csv → 400). Formula injection is out of scope because the body is JSON.
IPv4 peer governor still applies. Do not add a SmartIp/XFF extractor.
Timeouts. Stay under the 30s TimeoutLayer. One page = bounded SQL (four optional UNION ALL branches, each LIMIT n+1). No unbounded SELECT * for the day.

Relevant files

Path	Role
`indexer/src/api/mod.rs`	`build_router`, `ApiDoc` paths/tags/schemas, governors
`indexer/src/api/gt.rs`	Listing events — do not reuse unredacted `maker` or 400-over-cap
`indexer/src/api/pairs.rs`	Trade / LP / limit JSON shapes, `bd_plain_string`
`indexer/src/api/traders.rs`	Wallet-scoped history (unredacted; not this route)
`indexer/src/api/defillama.rs`	UTC-day query validation pattern
`indexer/src/api/protocol_fees.rs` / `protocol_fee_series.rs`	Fee aggregates — wrap counts, not tape
`indexer/src/indexer/protocol_fees.rs`	`FeeSource`, wrap pin, spoof rejection
`indexer/src/db/queries/swap_events.rs`	Swap tape
`indexer/src/db/queries/liquidity.rs`	LP tape
`indexer/src/db/queries/limit_order_fills.rs` / `limit_order_lifecycle.rs`	Limit fills + place/cancel
`indexer/src/db/queries/protocol_fees.rs`	Wrap/unwrap fee rows
`indexer/src/api/errors.rs`	`internal_err`
`indexer/tests/security.rs`	Rate limit, 400/500, OpenAPI smoke
`indexer/tests/api_gt.rs` / `gt_event_reserves.rs`	GT caps — regression that this route does not change them
`indexer/tests/api_limit_lower_bound.rs` / `limit_clamp_guardrail.rs`	`limit` clamp idiom
`docs/indexer-invariants.md`	New invariant row
`docs/runbooks/suspicious-activity-queries.md`	Pointer: redacted daily vs unredacted incident SQL

New (expected): indexer/src/api/evidence.rs (handler + redaction helper + query), indexer/tests/api_evidence_daily.rs.

Recommended direction

One handler, four parameterized SELECTs, merge in Rust.

GET /api/v1/evidence/daily?day=2026-09-03&surface=swap&surface=lp&limit=500&cursor=

Response shape (illustrative — keep additive JSON, #[serde(skip_serializing_if)] for absent optional fields):

{
  "day": "2026-09-03",
  "timezone": "UTC",
  "complete": true,
  "surfaces": ["swap", "lp"],
  "events": [
    {
      "surface": "swap",
      "kind": "swap",
      "block_height": 123,
      "block_timestamp": "2026-09-03T01:02:03Z",
      "tx_hash": "ABCD…",
      "pair_address": "terra1…",
      "actor_hash": "a1b2c3d4e5f67890a1b2c3d4e5f67890",
      "offer_amount": "1000000",
      "return_amount": "950000",
      "offer_decimals": 6,
      "ask_decimals": 6
    }
  ],
  "next_cursor": null,
  "has_more": false
}

Implementation notes:

Parse day with chrono::NaiveDate + and_hms_opt(0,0,0) + Utc. Reject 2026-13-40, 2026-09-03T00:00:00Z, unix timestamps, and day=today.
complete = day < Utc::now().date_naive().
Redaction helper unit-tested: same address → same hash; different addresses → different; output charset [0-9a-f]{32}; input with spaces rejected the same way as other terra1 pins (do not hash garbage — omit actor_hash if the stored actor is empty).
limit clamp: reuse the SEC-F05 idiom (.clamp(1, 1000)), not unwrap_or(n).min(n).
Cursor: versioned prefix + base64url of a small struct; tamper / truncated / foreign → 400 "invalid cursor".
Register on the global router (with /api/v1/pairs/.../trades), not lcd_heavy_router.
Add utoipa::path + ApiDoc path/schema + tag Evidence.
Short curl in docs/indexer-invariants.md (new row) and a 10-line subsection — do not expand #1204’s five-surface pack here.
Hybrid swap optional fields (pool_return_amount / book_return_amount) may be included on surface=swap as plain strings when indexed; they do not create extra rows.

Wrap row example: surface=wrap, kind=unwrap, amount_raw fee, token denom or CW20, no actor_hash, no pair_address unless you join a pair (do not invent a pair).

Limit row example: surface=limit, kind place | cancel | fill, order_id, pair_address, actor_hash from owner/maker, fill may include swap_tx_hash / swap_event_id when linkage exists (#316) without duplicating the parent swap’s amounts as a second swap.

Acceptance criteria

AC1. GET /api/v1/evidence/daily?day=YYYY-MM-DD returns 200 JSON with events[] sorted by (block_height, tx_hash, surface, kind, ordinal) ascending.
AC2. Every event has surface in {swap, wrap, limit, lp} and a kind from that surface’s set (swap; wrap/unwrap; place/cancel/fill; add/remove).
AC3. Response body (including nested objects) contains no terra1 user actor matching seeded senders/makers/owners/providers. Pair/token contracts may appear. Keys sender, receiver, maker, owner, provider are absent.
AC4. Same seeded actor → identical actor_hash across swap and limit rows on that day. Wrap fee rows omit actor_hash.
AC5. surface=swap row count equals swap_events in the UTC window. Limit fills are not extra surface=swap rows (L10).
AC6. surface=wrap includes only protocol_fee_events with source wrap or unwrap in the window. ust1_mint / swap_amm / limit_place are absent from wrap.
AC7. surface=lp kind matches liquidity_events.event_type. surface=limit covers place + cancel + fill tables (cancels that omit still-open placements follow existing lifecycle listing rules: export indexed rows, not LCD resting book).
AC8. surface=limit (only) omits swap/lp/wrap rows. Comma list surface=swap,wrap is the union of those two.
AC9. Missing day, malformed day, future day, unknown surface, format=csv, limit=0 (clamped to 1, not 500), limit=10000 (clamped to 1000), invalid cursor → 400 except limit clamp which is 200 with clamped page (match existing list routes: negative/zero → 1, oversized → max). Pick one behavior and test it: prefer existing clamp (200) for limit; 400 for the rest.
AC10. Empty indexed day → 200 events: []. Today UTC → "complete": false. A past day with rows → "complete": true.
AC11. Page 1 has_more=true + next_cursor when more than limit events exist; page 2 with that cursor returns the remainder without overlap or omission (stable sort).
AC12. ApiDoc includes the path; GET /api-docs/openapi.json lists /api/v1/evidence/daily. GET /gt/events caps and bodies unchanged.
AC13. Amounts on swap/lp/limit are plain digit strings. Wrap amount_raw likewise.
AC14. Handler uses sqlx bound parameters only. internal_err on DB failure (no sqlx text).
AC15. Route is not on the LCD-heavy governor list.

Test plan (functional paths)

Use the existing indexer test Postgres harness (same pattern as indexer/tests/api_pairs.rs / indexer/tests/indexer_protocol_fees.rs). New file indexer/tests/api_evidence_daily.rs.

Seed all four surfaces on one UTC day (swap, LP add+remove, limit place+fill+cancel, wrap + unwrap fee rows) plus a swap on the previous UTC day and the next UTC day. day= returns only the middle day.
Default surfaces — all four present; counts match tables.
Filter surface=swap — only swaps; previous-day swap absent.
Filter surface=wrap — wrap + unwrap; ust1_mint seeded sibling absent.
Filter surface=limit — place, cancel, fill; fill does not appear as swap.
Filter surface=lp — add and remove; GT-style names join/exit never appear.
Multi-filter surface=swap&surface=lp (and comma form if implemented — if only one form, document and test that form only).
Redaction — seeded terra1… senders/makers/owners/providers do not appear as substrings in serde_json::to_string. actor_hash length 32 hex. Same sender hashed equally on two swaps.
Wrap has no actor_hash.
Pagination — seed limit+2 swaps; limit=limit → has_more; second page last two; union of pages equals full set; intersection empty.
Empty day — 200 empty array.
Incomplete today — freeze time or use Utc::now().date_naive(); complete=false.
OpenAPI — spec contains the path (extend openapi_spec_available or a dedicated assert).
GT regression — existing /gt/events test still passes with unredacted maker (do not “fix” GT as part of this).
Hybrid swap — one swap_events row with pool+book legs; one surface=swap event (L10); optional leg fields if exposed.
Plain amounts — 18-decimal offer does not serialize as 1e+19.

Test plan (attack, hack, and abuse)

SQL injection in day — day=2026-09-03'%20OR%201=1--, day=2026-09-03;DROP TABLE swap_events. 400, tables intact.
SQL injection in surface — surface=swap' UNION SELECT sender FROM swap_events. 400. No actor leak.
SQL injection in cursor — raw SQL / ../ / huge binary. 400 invalid cursor.
redact=0, unredacted=1, raw=1 — ignored; body still redacted (no extra mode).
Actor in query string — sender=terra1… / trader= must not switch this route into trader history. Ignore unknown params (or 400 — pick ignore to match most list routes) and never echo the address.
Future day — day=2099-01-01 → 400.
Overflow limit — limit=-1, 0, 999999, limit=1e999. Clamp or 400; never negative SQL LIMIT (500).
Response size — page cap holds; a 10k-event day is exportable via cursors without a single multi-MB unbounded JSON (assert page events.len() <= 1000).
Rate limit — burst the new path under test governor like security.rs; 429 with Retry-After. Not LCD-heavy (a test that LCD-heavy 10 RPS does not uniquely wrap this route is enough if dual governors are hard to split in-process — at minimum, route is registered on api_router not lcd_heavy_router).
Error sanitization — force a DB error (closed pool) → body "Internal server error", no sqlx.
Bech32 in wrap token — wrap token may be a CW20 contract (protocol asset, allowed). User actors still absent.
Hash stability / enumeration — hashing empty string or terra1 garbage: omit actor_hash, do not panic.
Method abuse — POST/PUT/DELETE → 405.
CSV / formula — format=csv → 400. JSON strings starting with = are amounts/hashes only; no spreadsheet download.
Listing gem inclusion — seed a gem pair swap; evidence includes it; /gt/events still omits (existing test). Confirms this is not a GT clone.
Do not scan pair_reserves — unit/integration: handler SQL (string assert or query spy) mentions only the four event tables (+ pairs/assets for addresses). Fail if pair_reserves appears.

Verification criteria

cargo test --manifest-path indexer/Cargo.toml --test api_evidence_daily -- --nocapture (and existing security.rs / api_gt.rs) green.
Manual: curl GET /api/v1/evidence/daily?day=<seeded> against the test server; jq shows surface tags; rg -o 'terra1[a-z0-9]+' on the body matches only pair/token contracts from fixtures, not the seeded trader.
OpenAPI: curl /api-docs/openapi.json | jq '.paths["/api/v1/evidence/daily"]' non-null.
Invariants doc updated with the table row (UTC day, redaction, four surfaces, page cap, L10, wrap-is-fee).
No change to /gt/events request/response contract.

Out of scope

Frontend, share buttons, dApp product events (#1202).
OpenAPI/curl pack for the five existing listing/analytics surfaces (#1204).
Wrap principal tape, new mapper ingest, UST1 window as a fifth surface.
Hooks, community-token catalog events, CG/CMC trades, resting-book snapshots.
Unredacted export, API tokens, HMAC salts, operator-only routes.
CSV, NDJSON, object storage, signed URLs, scheduled dump jobs.
Multi-day ranges, block-height windows (use /gt/events for block windows).
Changing suspicious-activity SQL runbook recipes (a one-line pointer is enough).
Deploy, image, or host selection.

## Summary Add one **read-only indexer HTTP export** that returns a **UTC calendar-day JSON** of already-indexed on-chain activity, each row tagged `surface` ∈ {`swap`, `wrap`, `limit`, `lp`}, with **user bech32 redacted**. Research and incident-triage agents today must fan out across pair/trader tapes, `/gt/events`, and fee rollups, then strip wallets themselves. That is not an export. Bundle (do not split): 1. **`GET /api/v1/evidence/daily`** — required `day=YYYY-MM-DD` (UTC `[00:00Z, +1d)`), optional `surface=` allowlist, cursor pagination, bounded page size. 2. **Surface tags** — `swap` from `swap_events`; `lp` from `liquidity_events`; `limit` from placements + cancellations + fills; `wrap` from `protocol_fee_events` `source ∈ {wrap, unwrap}`. 3. **Redaction** — never emit `sender`, `receiver`, `maker`, `owner`, or `provider`. Emit a non-reversible `actor_hash` when the source row has an actor. Pair/token **contracts** stay. 4. **OpenAPI + tests** — `utoipa` path on `ApiDoc`, handler tests for every surface and the abuse matrix below. This is a **new business endpoint** on the existing ingest. No frontend. No wrap-principal ingest. No CSV. No API keys. Related (do **not** merge into this ticket): - [#1204](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1204) — docs/OpenAPI pack for **existing** swaps/pools/fees/burns/windows routes. Explicitly “no new business endpoints.” - [#1202](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1202) — dApp **product** events (campaign → `tx_succeeded`). Off-chain, unredacted by design, not an on-chain tape. - [#694](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/694) / [#646](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/646) / [#684](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/684) — GeckoTerminal `/gt/events` (swap+join/exit, unredacted `maker`, block window, 5000-row **400**). - [#631](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/631) — DeFiLlama **aggregates** on UTC day, not per-event JSON. - [#216](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/216) **L10** — headline volume is the parent `swap_events` row; do not also count `limit_order_fills` as `swap`. - [#586](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/586) / [#614](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/614) — wrap-mapper fees (`notify_deposit` / `unwrap`) into `protocol_fee_events` only. - [#432](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/432) — CSV formula prefix (this ticket is JSON-only). - [#557](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/557) — amounts as plain integer digit strings. - Operator SQL in [`docs/runbooks/suspicious-activity-queries.md`](docs/runbooks/suspicious-activity-queries.md) stays the **unredacted** incident path. --- ## Current codebase There is **no** unified event stream, **no** `surface` enum, and **no** address redaction on JSON. Events live in separate tables and pair/trader routes. Daily HTTP exists only as **aggregates**. ### Tapes are pair- or trader-scoped and unredacted | Surface wanted | Storage | Live HTTP | Actor field | |----------------|---------|-----------|-------------| | **swap** | `swap_events` | `GET /api/v1/pairs/{addr}/trades`, `GET /api/v1/traders/{addr}/trades` | `sender` (+ optional `receiver`) | | **lp** | `liquidity_events` (`event_type` `add` \| `remove`) | `GET /api/v1/pairs/{addr}/liquidity-events` | `provider` | | **limit** | `limit_order_placements`, `limit_order_cancellations`, `limit_order_fills` | pair + trader `limit-placements` / `limit-cancellations` / `limit-fills` | `owner` / `maker` | | **wrap** | `protocol_fee_events.source` = `wrap` \| `unwrap` | fee **rollups** only (`/api/v1/protocol/fees`, DeFiLlama daily) | **none** (no actor column) | Router: [`indexer/src/api/mod.rs`](indexer/src/api/mod.rs) `build_router`. Pair handlers: [`indexer/src/api/pairs.rs`](indexer/src/api/pairs.rs). Trader handlers: [`indexer/src/api/traders.rs`](indexer/src/api/traders.rs). Rows: [`indexer/src/db/queries/swap_events.rs`](indexer/src/db/queries/swap_events.rs) (`SwapEventRow.sender`), [`liquidity.rs`](indexer/src/db/queries/liquidity.rs) (`provider`), [`limit_order_fills.rs`](indexer/src/db/queries/limit_order_fills.rs) (`maker`). `GET /api/v1/traders/{addr}/trades?format=csv` is a **wallet-scoped** history dump with the full `sender` column ([#163](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/163)). It is not a protocol-wide daily export and it is not redacted. ### `/gt/events` is a listing adapter, not evidence [`indexer/src/api/gt.rs`](indexer/src/api/gt.rs): `eventType` `swap` \| `join` \| `exit`, **full `maker`**, inclusive `fromBlock`/`toBlock` (max **2000** blocks), combined rows ≤ **`MAX_GT_EVENT_ROWS` 5000** else **400** (`GT_EVENT_ROW_CAP_MSG`). No limit book, no wrap. Gems omitted (**L639-2**). Reserves are persisted post-event columns, never live `pair_reserves` (#684). ### Wrap is a fee source, not a wrap tape [`indexer/src/indexer/protocol_fees.rs`](indexer/src/indexer/protocol_fees.rs): `FeeSource::{Wrap, Unwrap, …}`. Retail wrap wasm is pinned-mapper `action=notify_deposit`; unwrap is `action=unwrap`; amount key **`fee`** (not principal). Insert: [`indexer/src/db/queries/protocol_fees.rs`](indexer/src/db/queries/protocol_fees.rs) `protocol_fee_events (block_height, block_timestamp, tx_hash, source, ordinal, asset_id, amount_raw, decimals, fee_usd)` — **no sender**. Spoof `notify_deposit` off the pin is ignored. UST1 mint/redeem is a **different** family (`ust1_mint` / `ust1_redeem`) and is not wrap. ### Daily HTTP is aggregates `GET /api/v1/defillama/daily?timestamp=` (#631) and `GET /api/v1/protocol/{volume,fees,liquidity}/daily` are UTC-day **sums**. Volume rule: parent `swap_events` only — never fills, wrap, or UST1 window. They are not event dumps. ### Redaction exists only in logs [`indexer/src/lcd/mod.rs`](indexer/src/lcd/mod.rs) redacts LCD paths at WARN. API JSON returns full bech32. No `sha256` actor alias helper. ### Auth and governors Public `GET`. Global `tower_governor` (default **60 RPS**; prod refuses `0`). LCD-heavy router is a **second** 10 RPS list (limit-book, route/solve, blacklist-check, CG/CMC orderbook). Keys are **socket peer IP** (`PeerIpKeyExtractor`) — no trusted `X-Forwarded-For`. SQL list `limit` is clamped (SEC-F05 / #431). Internal errors: `internal_err()` → `"Internal server error"`. ### Docs / discovery today [`docs/runbooks/suspicious-activity-queries.md`](docs/runbooks/suspicious-activity-queries.md) tells operators to curl **unredacted** leaderboard + trader trades or raw SQL. That is the opposite of a shareable daily blob. [#1204](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1204) will document existing paths; it will not add this route. --- ## Why the new implementation is needed Evidence consumers need **one day’s** protocol activity as JSON they can store and share **without a wallet list**. Today that requires: 1. Enumerating pairs, paging `/trades` + `/liquidity-events` + three limit routes per pair. 2. Separately reading wrap **fee** rollups that are not an event tape. 3. Manually mapping GT `join`/`exit` vs indexer `add`/`remove` vs limit fills that must not be double-counted as swaps (**L10**). 4. Stripping `sender` / `maker` / `owner` / `provider` after the fact. Without a dedicated export, agents either keep PII, miss wrap/limit, or mix listing adapters (`/gt/events`) with protocol truth. A UTC-day, surface-tagged, redacted document is the missing contract. --- ## Constraints / guardrails 1. **Read-only.** `GET` only. No indexer DB writes. Not on `lcd_heavy_router` (Postgres only). No LCD on the request path. 2. **No new ingest.** Do not add `wrap_events`. Do not persist wrap **principal**. Do not re-parse historical wasm. Export what is already indexed. 3. **No API keys / bearer / `?redact=0`.** The public API stays unauthenticated. Redaction **is** the access control for this route. Do not add a secret salt (new key material). Do not add an unredacted twin. 4. **One UTC calendar day.** Same clock as #631 (`[day 00:00:00Z, next 00:00:00Z)`). No multi-day `from`/`to`. No trailing `24h`. `day` is `YYYY-MM-DD` parsed in Rust; bind `timestamptz` parameters — never concatenate SQL. 5. **Surface allowlist.** `swap` \| `wrap` \| `limit` \| `lp` only (lowercase after trim). Repeatable `surface=` or comma list. Unknown / empty token after split → **400**. Default: all four. 6. **L10.** `surface=swap` is `swap_events` only. Limit fills are `surface=limit` + `kind=fill`. Do not emit a swap row and a fill row that an agent would sum as two volumes. Optional `swap_event_id` on fill rows is correlation, not a second swap. 7. **Wrap honesty.** Wrap rows are **treasury fee** events (`amount_raw` = indexed fee). Label `kind` `wrap` \| `unwrap` from `source`. Omit `actor_hash` when the table has no actor. Do **not** fold `ust1_mint` / `ust1_redeem` / `swap_amm` / `book_take` / `limit_place` into wrap. 8. **LP kinds.** `kind` `add` \| `remove` from `liquidity_events.event_type`. Do not rename to GT `join`/`exit` on this route. 9. **Redaction.** SHA-256 of the **canonical** actor bech32 (trim only; do not case-fold terra1). Wire `actor_hash` = first **32 hex chars** (16 bytes). Drop actor fields entirely (not `null`). Pair address, token contracts, native denoms (`uusd` / `uluna`) stay. `tx_hash` stays so the blob is chain-linkable; document that LCD lookup deanonymizes a row. That tradeoff is accepted for evidence; the goal is “no wallet list in the JSON,” not cryptographic unlinkability. 10. **Pagination, not GT 400-over-cap.** Busy days must be exportable. Default `limit=500`, clamp **1–1000**. Opaque `cursor` / `next_cursor` on `(block_height, tx_hash, surface, kind, ordinal)`. `has_more` when another page exists. Do **not** 400 when the day has >5000 events. 11. **Amounts.** Plain integer digit strings (`bd_plain_string` / #557). No scientific notation. Optional `decimals` when known. `fee_usd` on wrap may be JSON number/string consistent with fee APIs, or omitted if unpriced — do not emit `null` that a client treats as `$0` without a field. 12. **Errors.** Bad `day` / future `day` (strictly after today’s UTC date) / bad `surface` / bad `cursor` / bad `limit` → **400** with a short message. DB failure → **500** `"Internal server error"`. Empty day → **200** `{ "events": [], "has_more": false }`. Today’s incomplete UTC day → **200** with `"complete": false`. 13. **Do not mutate** `/gt/events`, pair/trader tapes, DeFiLlama, or CSV. Those stay unredacted. 14. **Gems.** Include indexed gem/test-pair activity (this is evidence, not a listing adapter). Document that the dump is **not** L639-safe. 15. **No CSV** on this route (`format=csv` → **400**). Formula injection is out of scope because the body is JSON. 16. **IPv4 peer governor** still applies. Do not add a SmartIp/XFF extractor. 17. **Timeouts.** Stay under the 30s `TimeoutLayer`. One page = bounded SQL (four optional `UNION ALL` branches, each `LIMIT n+1`). No unbounded `SELECT *` for the day. --- ## Relevant files | Path | Role | |------|------| | [`indexer/src/api/mod.rs`](indexer/src/api/mod.rs) | `build_router`, `ApiDoc` paths/tags/schemas, governors | | [`indexer/src/api/gt.rs`](indexer/src/api/gt.rs) | Listing events — **do not reuse** unredacted `maker` or 400-over-cap | | [`indexer/src/api/pairs.rs`](indexer/src/api/pairs.rs) | Trade / LP / limit JSON shapes, `bd_plain_string` | | [`indexer/src/api/traders.rs`](indexer/src/api/traders.rs) | Wallet-scoped history (unredacted; not this route) | | [`indexer/src/api/defillama.rs`](indexer/src/api/defillama.rs) | UTC-day query validation pattern | | [`indexer/src/api/protocol_fees.rs`](indexer/src/api/protocol_fees.rs) / [`protocol_fee_series.rs`](indexer/src/api/protocol_fee_series.rs) | Fee aggregates — wrap **counts**, not tape | | [`indexer/src/indexer/protocol_fees.rs`](indexer/src/indexer/protocol_fees.rs) | `FeeSource`, wrap pin, spoof rejection | | [`indexer/src/db/queries/swap_events.rs`](indexer/src/db/queries/swap_events.rs) | Swap tape | | [`indexer/src/db/queries/liquidity.rs`](indexer/src/db/queries/liquidity.rs) | LP tape | | [`indexer/src/db/queries/limit_order_fills.rs`](indexer/src/db/queries/limit_order_fills.rs) / `limit_order_lifecycle.rs` | Limit fills + place/cancel | | [`indexer/src/db/queries/protocol_fees.rs`](indexer/src/db/queries/protocol_fees.rs) | Wrap/unwrap fee rows | | [`indexer/src/api/errors.rs`](indexer/src/api/errors.rs) | `internal_err` | | [`indexer/tests/security.rs`](indexer/tests/security.rs) | Rate limit, 400/500, OpenAPI smoke | | [`indexer/tests/api_gt.rs`](indexer/tests/api_gt.rs) / [`gt_event_reserves.rs`](indexer/tests/gt_event_reserves.rs) | GT caps — regression that this route does not change them | | [`indexer/tests/api_limit_lower_bound.rs`](indexer/tests/api_limit_lower_bound.rs) / [`limit_clamp_guardrail.rs`](indexer/tests/limit_clamp_guardrail.rs) | `limit` clamp idiom | | [`docs/indexer-invariants.md`](docs/indexer-invariants.md) | New invariant row | | [`docs/runbooks/suspicious-activity-queries.md`](docs/runbooks/suspicious-activity-queries.md) | Pointer: redacted daily vs unredacted incident SQL | New (expected): `indexer/src/api/evidence.rs` (handler + redaction helper + query), `indexer/tests/api_evidence_daily.rs`. --- ## Recommended direction **One handler, four parameterized SELECTs, merge in Rust.** ``` GET /api/v1/evidence/daily?day=2026-09-03&surface=swap&surface=lp&limit=500&cursor= ``` Response shape (illustrative — keep additive JSON, `#[serde(skip_serializing_if)]` for absent optional fields): ```json { "day": "2026-09-03", "timezone": "UTC", "complete": true, "surfaces": ["swap", "lp"], "events": [ { "surface": "swap", "kind": "swap", "block_height": 123, "block_timestamp": "2026-09-03T01:02:03Z", "tx_hash": "ABCD…", "pair_address": "terra1…", "actor_hash": "a1b2c3d4e5f67890a1b2c3d4e5f67890", "offer_amount": "1000000", "return_amount": "950000", "offer_decimals": 6, "ask_decimals": 6 } ], "next_cursor": null, "has_more": false } ``` Implementation notes: - Parse `day` with `chrono::NaiveDate` + `and_hms_opt(0,0,0)` + `Utc`. Reject `2026-13-40`, `2026-09-03T00:00:00Z`, unix timestamps, and `day=today`. - `complete` = `day < Utc::now().date_naive()`. - Redaction helper unit-tested: same address → same hash; different addresses → different; output charset `[0-9a-f]{32}`; input with spaces rejected the same way as other terra1 pins (do not hash garbage — omit `actor_hash` if the stored actor is empty). - `limit` clamp: reuse the SEC-F05 idiom (`.clamp(1, 1000)`), not `unwrap_or(n).min(n)`. - Cursor: versioned prefix + base64url of a small struct; tamper / truncated / foreign → **400** `"invalid cursor"`. - Register on the **global** router (with `/api/v1/pairs/.../trades`), not `lcd_heavy_router`. - Add `utoipa::path` + `ApiDoc` path/schema + tag **Evidence**. - Short curl in [`docs/indexer-invariants.md`](docs/indexer-invariants.md) (new row) and a 10-line subsection — do **not** expand [#1204](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1204)’s five-surface pack here. - Hybrid swap optional fields (`pool_return_amount` / `book_return_amount`) may be included on `surface=swap` as plain strings when indexed; they do not create extra rows. Wrap row example: `surface=wrap`, `kind=unwrap`, `amount_raw` fee, `token` denom or CW20, no `actor_hash`, no `pair_address` unless you join a pair (do not invent a pair). Limit row example: `surface=limit`, `kind` `place` \| `cancel` \| `fill`, `order_id`, `pair_address`, `actor_hash` from owner/maker, fill may include `swap_tx_hash` / `swap_event_id` when linkage exists (#316) **without** duplicating the parent swap’s amounts as a second swap. --- ## Acceptance criteria - **AC1.** `GET /api/v1/evidence/daily?day=YYYY-MM-DD` returns **200** JSON with `events[]` sorted by `(block_height, tx_hash, surface, kind, ordinal)` ascending. - **AC2.** Every event has `surface` in `{swap, wrap, limit, lp}` and a `kind` from that surface’s set (`swap`; `wrap`/`unwrap`; `place`/`cancel`/`fill`; `add`/`remove`). - **AC3.** Response body (including nested objects) contains **no** `terra1` user actor matching seeded senders/makers/owners/providers. Pair/token contracts may appear. Keys `sender`, `receiver`, `maker`, `owner`, `provider` are absent. - **AC4.** Same seeded actor → identical `actor_hash` across swap and limit rows on that day. Wrap fee rows omit `actor_hash`. - **AC5.** `surface=swap` row count equals `swap_events` in the UTC window. Limit fills are **not** extra `surface=swap` rows (**L10**). - **AC6.** `surface=wrap` includes only `protocol_fee_events` with `source` `wrap` or `unwrap` in the window. `ust1_mint` / `swap_amm` / `limit_place` are absent from wrap. - **AC7.** `surface=lp` `kind` matches `liquidity_events.event_type`. `surface=limit` covers place + cancel + fill tables (cancels that omit still-open placements follow existing lifecycle listing rules: export **indexed rows**, not LCD resting book). - **AC8.** `surface=limit` (only) omits swap/lp/wrap rows. Comma list `surface=swap,wrap` is the union of those two. - **AC9.** Missing `day`, malformed `day`, future `day`, unknown `surface`, `format=csv`, `limit=0` (clamped to 1, not 500), `limit=10000` (clamped to 1000), invalid `cursor` → **400** except `limit` clamp which is **200** with clamped page (match existing list routes: negative/zero → 1, oversized → max). Pick **one** behavior and test it: **prefer existing clamp (200) for `limit`; 400 for the rest.** - **AC10.** Empty indexed day → **200** `events: []`. Today UTC → `"complete": false`. A past day with rows → `"complete": true`. - **AC11.** Page 1 `has_more=true` + `next_cursor` when more than `limit` events exist; page 2 with that cursor returns the remainder without overlap or omission (stable sort). - **AC12.** `ApiDoc` includes the path; `GET /api-docs/openapi.json` lists `/api/v1/evidence/daily`. `GET /gt/events` caps and bodies unchanged. - **AC13.** Amounts on swap/lp/limit are plain digit strings. Wrap `amount_raw` likewise. - **AC14.** Handler uses sqlx bound parameters only. `internal_err` on DB failure (no sqlx text). - **AC15.** Route is not on the LCD-heavy governor list. --- ## Test plan (functional paths) Use the existing indexer test Postgres harness (same pattern as [`indexer/tests/api_pairs.rs`](indexer/tests/api_pairs.rs) / [`indexer/tests/indexer_protocol_fees.rs`](indexer/tests/indexer_protocol_fees.rs)). New file `indexer/tests/api_evidence_daily.rs`. 1. **Seed all four surfaces** on one UTC day (swap, LP add+remove, limit place+fill+cancel, wrap + unwrap fee rows) plus a swap on the previous UTC day and the next UTC day. `day=` returns only the middle day. 2. **Default surfaces** — all four present; counts match tables. 3. **Filter `surface=swap`** — only swaps; previous-day swap absent. 4. **Filter `surface=wrap`** — wrap + unwrap; `ust1_mint` seeded sibling absent. 5. **Filter `surface=limit`** — place, cancel, fill; fill does not appear as `swap`. 6. **Filter `surface=lp`** — add and remove; GT-style names `join`/`exit` never appear. 7. **Multi-filter** `surface=swap&surface=lp` (and comma form if implemented — if only one form, document and test that form only). 8. **Redaction** — seeded `terra1…` senders/makers/owners/providers do not appear as substrings in `serde_json::to_string`. `actor_hash` length 32 hex. Same sender hashed equally on two swaps. 9. **Wrap has no actor_hash.** 10. **Pagination** — seed `limit+2` swaps; `limit=limit` → `has_more`; second page last two; union of pages equals full set; intersection empty. 11. **Empty day** — 200 empty array. 12. **Incomplete today** — freeze time or use `Utc::now().date_naive()`; `complete=false`. 13. **OpenAPI** — spec contains the path (extend `openapi_spec_available` or a dedicated assert). 14. **GT regression** — existing `/gt/events` test still passes with unredacted `maker` (do not “fix” GT as part of this). 15. **Hybrid swap** — one `swap_events` row with pool+book legs; **one** `surface=swap` event (L10); optional leg fields if exposed. 16. **Plain amounts** — 18-decimal offer does not serialize as `1e+19`. --- ## Test plan (attack, hack, and abuse) 1. **SQL injection in `day`** — `day=2026-09-03'%20OR%201=1--`, `day=2026-09-03;DROP TABLE swap_events`. **400**, tables intact. 2. **SQL injection in `surface`** — `surface=swap' UNION SELECT sender FROM swap_events`. **400**. No actor leak. 3. **SQL injection in `cursor`** — raw SQL / `../` / huge binary. **400** `invalid cursor`. 4. **`redact=0`, `unredacted=1`, `raw=1`** — ignored; body still redacted (no extra mode). 5. **Actor in query string** — `sender=terra1…` / `trader=` must not switch this route into trader history. Ignore unknown params (or 400 — pick ignore to match most list routes) and never echo the address. 6. **Future day** — `day=2099-01-01` → **400**. 7. **Overflow `limit`** — `limit=-1`, `0`, `999999`, `limit=1e999`. Clamp or 400; never negative SQL `LIMIT` (**500**). 8. **Response size** — page cap holds; a 10k-event day is exportable via cursors without a single multi-MB unbounded JSON (assert page `events.len() <= 1000`). 9. **Rate limit** — burst the new path under test governor like `security.rs`; **429** with `Retry-After`. Not LCD-heavy (a test that LCD-heavy 10 RPS does not uniquely wrap this route is enough if dual governors are hard to split in-process — at minimum, route is registered on `api_router` not `lcd_heavy_router`). 10. **Error sanitization** — force a DB error (closed pool) → body `"Internal server error"`, no sqlx. 11. **Bech32 in wrap token** — wrap `token` may be a CW20 contract (protocol asset, allowed). User actors still absent. 12. **Hash stability / enumeration** — hashing empty string or `terra1` garbage: omit `actor_hash`, do not panic. 13. **Method abuse** — `POST`/`PUT`/`DELETE` → **405**. 14. **CSV / formula** — `format=csv` → **400**. JSON strings starting with `=` are amounts/hashes only; no spreadsheet download. 15. **Listing gem inclusion** — seed a gem pair swap; evidence **includes** it; `/gt/events` still omits (existing test). Confirms this is not a GT clone. 16. **Do not scan `pair_reserves`** — unit/integration: handler SQL (string assert or query spy) mentions only the four event tables (+ `pairs`/`assets` for addresses). Fail if `pair_reserves` appears. --- ## Verification criteria - `cargo test --manifest-path indexer/Cargo.toml --test api_evidence_daily -- --nocapture` (and existing `security.rs` / `api_gt.rs`) green. - Manual: curl `GET /api/v1/evidence/daily?day=<seeded>` against the test server; `jq` shows `surface` tags; `rg -o 'terra1[a-z0-9]+'` on the body matches only pair/token contracts from fixtures, not the seeded trader. - OpenAPI: `curl /api-docs/openapi.json | jq '.paths["/api/v1/evidence/daily"]'` non-null. - Invariants doc updated with the table row (UTC day, redaction, four surfaces, page cap, L10, wrap-is-fee). - No change to `/gt/events` request/response contract. --- ## Out of scope - Frontend, share buttons, dApp product events (#1202). - OpenAPI/curl pack for the five existing listing/analytics surfaces (#1204). - Wrap **principal** tape, new mapper ingest, UST1 window as a fifth surface. - Hooks, community-token catalog events, CG/CMC trades, resting-book snapshots. - Unredacted export, API tokens, HMAC salts, operator-only routes. - CSV, NDJSON, object storage, signed URLs, scheduled dump jobs. - Multi-day ranges, block-height windows (use `/gt/events` for block windows). - Changing suspicious-activity SQL runbook recipes (a one-line pointer is enough). - Deploy, image, or host selection.

PlasticDigits referenced this issue

2026-09-04 08:13:49 +00:00

feat(indexer): anonymous UTC-day activity digest (counts + fee-path mix) #1206

PlasticDigits referenced this issue from code/cl8y-research

2026-09-04 08:26:20 +00:00

feat(collect): competitor fee/liquidity watch into raw notes ingest #14