feat(indexer): anonymous UTC-day activity digest (counts + fee-path mix) #1206
Labels
No labels
agent:fix_bugfix
agent:fix_conflicts
agent:fix_security
agent:gap_analysis
agent:implement
agent:implement
agent:implement
agent:open_issues
agent:ready
agent:research
agent:security_audit
agent:verify
architecture
backend
blocker:hybrid
blocker:launch
blocker:limit-orders
blocker:v2
block:log_only
block:security
bug
ci
contracts
correctness
deploy
dev
devops
docs
documentation
duplicate
e2e
enhancement
epic
feature
frontend
functional-completion
gas
good first issue
governance
help wanted
high-risk
hooks
hybrid
indexer
infra
infrastructure
integrators
invalid
launch-blocker
limit-orders
localnet
localterra
low priority
missing-implementation
needs-design
ops
performance
priority
high
priority
medium
product
qa
QA
question
ready
ready
research
scripts
security
security-hardening
smartcontracts
tech-debt
testing
ux
UX
v2
verification
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
code/cl8y-dex-terraclassic#1206
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Add one read-only indexer HTTP digest that returns a UTC calendar-day JSON of protocol activity counts plus treasury fee-path mix, with no wallets, no tx hashes, and no joins from events to people. Research agents today must stitch trailing
/protocol/fees, Llama UTC-day USD, overview trader counts, and unredacted pair/trader tapes. That is not an anonymous digest.Bundle (do not split):
GET /api/v1/evidence/digest— requiredday=YYYY-MM-DD(UTC[00:00Z, +1d)). One document. No pagination (aggregates only).COUNT(*)by surface ∈ {swap,wrap,limit,lp} and kind (swap;wrap/unwrap;place/cancel/fill;add/remove).FeeSourcevalues (swap_amm,book_take,limit_place,wrap,unwrap,ust1_mint,ust1_redeem):event_count, pricedamount_usd,share_pct. Protocol-grain token mix (no wallets).sender,receiver,maker,owner,provider,actor_hash,tx_hash,swap_event_id, trader addresses, or unique-trader counts. No SQL joins from fee/activity rows to people tables.utoipapath onApiDoc, handler tests for every count/mix path and the abuse matrix below.This is a new business endpoint on existing ingest. No frontend. No new tables. No CSV. No API keys. No event tape.
Related (do not merge into this ticket):
tx_hash(LCD can deanonymize a row) andactor_hash. Different product: evidence tape, not a digest. Do not implement this digest inside#1205.tx_succeeded). Off-chain, unredacted by design, joinstxhashto fees. Forbidden here.24h/7d/30dGET /api/v1/protocol/feessource+token mix (shipped). Not a UTC calendar day.GET /api/v1/protocol/fees/dailygrain series (hourly/daily/monthly) offees_usd+ totalevent_count. Noby_source. Not one-day evidence.GET /api/v1/defillama/daily?timestamp=listing adapter: gem-exclude, 404 if day not rolled, unixtimestamp, USD + Llamafees.*only. No LP/limit activity counts. Not shareable as an evidence digest.swap_eventsonly; do not addlimit_order_fillsintosurface=swap.docs/runbooks/suspicious-activity-queries.mdstays the unredacted incident path.Current codebase
There is no anonymous one-day document that combines activity counts with fee-path mix. Nearby HTTP is either trailing, listing-adapter, chart series, people-scoped, or (proposed) event-level.
Fee mix exists, but not as a UTC-day evidence digest
GET /api/v1/protocol/fees?window=by_source+by_tokenwithevent_count/amount_usd/share_pct(indexer/src/api/protocol_fees.rs)24h|7d|30donly (parse_fee_window). Reads rollup tables, not a calendar day.GET /api/v1/protocol/fees/dailyfees_usd+ totalevent_count(indexer/src/api/protocol_fee_series.rs)by_source.from/to/window→ 400. Does not COUNT swaps/LP/limits.GET /api/v1/defillama/daily?timestamp=volume_usd,trade_count,fees.{swap_amm,…}(indexer/src/api/defillama.rs)lp/ limit place+cancel activity counts.GET /api/v1/overviewunique_traders_24h(indexer/src/api/overview.rs)FeeSourceis already canonical (indexer/src/indexer/protocol_fees.rs):swap_amm,book_take,limit_place,wrap,unwrap,ust1_mint,ust1_redeem. Wrap/unwrap rows are treasury fees (pinned mappernotify_deposit/unwrap, amount keyfee). UST1 window is a different family. Do not invent new source strings.Activity counts are scattered and often people-scoped
swap_events/trades, trader/trades(unredactedsender). Overviewtotal_trades_*is trailing, not UTC day. Llamatrade_countis gem-filtered and 404-gated.liquidity_events(add|remove)/liquidity-events(provider). No protocol-wide daily count.limit_order_placements/_cancellations/_fillsowner/maker). Fill count ≠ swap count (L10).protocol_fee_events.source∈ {wrap,unwrap}Router:
indexer/src/api/mod.rsbuild_router. Pair/trader handlers stay unredacted. There is noevidencemodule onmain.Joins to people are the default on existing tapes
Trader routes take a bech32 path param. Pair trades emit
sender. GT/gt/eventsemits fullmaker. Overview materializesunique_traders_24h(distinct swap senders).#1205(open) still keepstx_hashso LCD lookup deanonymizes a row, plusactor_hashthat is stable across surfaces.A digest that is safe to share must not SELECT those columns, must not
COUNT(DISTINCT sender), and must not JOINprotocol_fee_eventstoswap_events/tradersontx_hash.Auth and governors
Public
GET. Globaltower_governor(default 60 RPS; prod refuses0). Keys are socket peer IP (PeerIpKeyExtractor) — no trustedX-Forwarded-For. LCD-heavy list is a second 10 RPS router (limit-book, route/solve, blacklist-check, CG/CMC orderbook). This digest is PostgresCOUNT/GROUP BYonly — not LCD-heavy. Internal errors:internal_err()→"Internal server error".Why the new implementation is needed
Evidence consumers need one day’s protocol picture they can store and share without a wallet list and without a tx list that joins to people.
Today that requires:
/protocol/fees(wrong clock vs UTC day, #576).sender/maker/owner/providerafter the fact — which still leavestx_hashjoinable via LCD.Without a dedicated digest, agents either keep PII, mix trailing windows with calendar days, or treat listing adapters as protocol truth. Counts + fee-path mix in one anonymous JSON is the missing cover sheet.
Constraints / guardrails
GETonly. No indexer DB writes. Not onlcd_heavy_router. No LCD on the request path.wrap_events. Do not persist wrap principal. ExportCOUNT/GROUP BYover what is already indexed.?redact=0/?join=1. Public API stays unauthenticated. Anonymity is the access control. No unredacted twin. No secret salt.[day 00:00:00Z, next 00:00:00Z)). No multi-dayfrom/to. No trailing24h.dayisYYYY-MM-DDparsed in Rust; bindtimestamptzparameters — never concatenate SQL. Reuseutc_day_start/ NaiveDate parsing already used by DeFiLlama / volume daily — do not invent a third clock.terra1…actors);sender,receiver,maker,owner,provider,trader,actor_hash;tx_hash,swap_tx_hash,swap_event_id,txhash;unique_traders/distinct_senders/ anyCOUNT(DISTINCT sender|maker|owner|provider).Pair/token contracts may appear only on fee
by_tokenrows (protocol assets), never as “who did this.” Native denomsuusd/ulunastay. Factory address must not be required in the digest (Llama embeds it in methodology; this route should omit a factory pin unless it is a static methodology string with no user addresses)./tradestape would re-identify. Do not emitpairs[], leaderboards, or top-N wallets.protocol_fee_eventstoswap_events,traders, or LCD. Fee-path mix isGROUP BY source(and token mixGROUP BY asset_id) onprotocol_fee_eventsin the UTC window. Activity counts are independentCOUNT(*)on each event table. Correlation across people is out of scope.counts.swap=COUNT(*)fromswap_eventsin the window. Limit fills incrementcounts.limit.kind=fillonly. Do not add fill count into swap count. Hybrid pool+book is still one swap row.protocol_fee_events.sourcewrap/unwrap), labeled as treasury-fee events, not wrap principal. Do not foldust1_mint/ust1_redeeminto wrap. Those arefees.by_sourcepaths and optionalcountskeys under a window family, notsurface=wrap.FeeSource::ALLrows every day (idle source →event_count=0,amount_usd="0"). Unknown DB source strings: omit from JSON (do not interpolate into JS). Same PFee/L7 rules: notspread_amount, burn tax, gas, hook fees, community-tax extra-debit."0"; activity + unpriced → JSONnull(not"0", not omit in a way that a client treats as $0).share_pctnull when the day’s priced fee SUM is 0. Headlinefees.total_usduses the same idle/unpriced rule as overview fees (priced SUM; one unpriced source must not silently zero a priced wrap — follow EFee-6 / #687 for the headline, fail-closed per source).bd_plain_string/ #557). No scientific notation.nullper source, not fake $1).day/ futureday(strictly after today’s UTC date) /format=csv/ unknown query that would change grain (window,timestamp,from,to,trader,sender) → 400 with a short message. DB failure → 500"Internal server error". Empty indexed day → 200 with zero counts and zero-idle fee rows (not 404). Today’s incomplete UTC day → 200 with"complete": false. Do not copy Llama’s 404-if-not-rolled./protocol/fees,/protocol/fees/daily,/defillama/daily,/overview,/gt/events, pair/trader tapes, or #1205’s proposed/evidence/daily.format=csv→ 400).TimeoutLayer. Bounded SQL: a handful ofCOUNT(*)/GROUP BY source/GROUP BY asset_idwith time bounds. NoSELECT *of the day’s events. No unboundedDISTINCTover senders.by_tokenwould be large, reuse the existing/protocol/fees“other” bucket (is_other) — do not return thousands of dust tokens. Still no wallets.Relevant files
indexer/src/api/mod.rsbuild_router,ApiDocpaths/tags/schemas, governorsindexer/src/api/protocol_fees.rsby_source/by_token) — clock is wrong; copy field semantics, not the windowindexer/src/api/protocol_fee_series.rsday=mixindexer/src/api/defillama.rsindexer/src/api/overview.rsunique_traders_24h— do not copy this fieldindexer/src/indexer/protocol_fees.rsFeeSource, wrap pin, UST1 window, PFee/L7indexer/src/db/queries/protocol_fees.rsGROUP BY sourceidiomindexer/src/db/queries/swap_events.rsindexer/src/db/queries/liquidity.rsindexer/src/db/queries/limit_order_fills.rs/limit_order_lifecycle.rsindexer/src/api/errors.rsinternal_errindexer/src/indexer/defillama.rsutc_day_start/ timestamp parse — reuse day bounds, not Llama responseindexer/tests/security.rsdocs/indexer-invariants.mddocs/runbooks/suspicious-activity-queries.mdNew (expected):
indexer/src/api/evidence_digest.rs(handler + queries),indexer/tests/api_evidence_digest.rs.If #1205 lands first and adds
indexer/src/api/evidence.rs, put the digest handler in that module as a second path — do not fold digest fields onto paginated events.Recommended direction
One handler, bounded aggregate SQL, assemble in Rust.
Response shape (illustrative — additive JSON,
#[serde(skip_serializing_if)]for absent optionals):Implementation notes:
daywithchrono::NaiveDate+and_hms_opt(0,0,0)+Utc. Reject2026-13-40,2026-09-03T00:00:00Z, unix timestamps, andday=today.complete=day < Utc::now().date_naive().COUNTswap;COUNTLPGROUP BY event_type;COUNTplace/cancel/fill;COUNTwrap/unwrap fees; feeGROUP BY source; optional token mix). Bind$1/$2as window start/end.api_router(with/api/v1/protocol/fees), notlcd_heavy_router.utoipa::path+ApiDocpath/schema + tag Evidence.docs/indexer-invariants.md. Do not expand #1204’s five-surface pack here.counts.wrapis fee-event counts. Document that next tofees.by_sourcewrap/unwrap so agents do not double-count wrap as both volume and fee path.counts.window{ "ust1_mint", "ust1_redeem" }as fee-event counts if it keeps wrap vs window honest; if omitted, those exist only underfees.by_source. Pick one and test it — prefer explicitcounts.windowso mint/redeem are not mistaken for wrap.Acceptance criteria
GET /api/v1/evidence/digest?day=YYYY-MM-DDreturns 200 JSON withday,timezone=UTC,complete,counts,fees.by_source(length 7,FeeSource::ALLorder stable).counts.swap.swapequalsCOUNT(*)ofswap_eventsin the UTC window. Limit fills do not increase swap count (L10). Hybrid pool+book is one swap.counts.lpadd/removematchliquidity_events.event_type. GT namesjoin/exitnever appear.counts.limitcovers place + cancel + fill tables. Fill count may differ from swap count.counts.wrapincludes onlyprotocol_fee_eventswrap/unwrap. Seededust1_mintdoes not increment wrap.fees.by_sourcefor that day matchesGROUP BY sourceonprotocol_fee_eventsin the same window (event_count). Idle sources present withevent_count=0andamount_usd="0"./protocol/fees?window=24his not required to equal this document (different clock). A test that seeds all events at00:30Zvs previous day23:30Zproves calendar bounds.tx_hash/actor_hash.day, malformedday, futureday,format=csv,window=24h,timestamp=,from=,sender=→ 400 (orsenderignored and still not used — prefer 400 for identity-shaped params so a client cannot think it filtered by wallet)."complete": false. A past day →"complete": true.ApiDocincludes the path;GET /api-docs/openapi.jsonlists/api/v1/evidence/digest./protocol/fees,/protocol/fees/daily,/defillama/dailycontracts unchanged.internal_erron DB failure (no sqlx text).traders,sender,maker,owner,provider, orCOUNT(DISTINCT. Mentions only event/fee aggregate tables (+assetsfor token symbols).amount_usdJSONnull.Test plan (functional paths)
Use the existing indexer test Postgres harness (same pattern as
indexer/tests/api_pairs.rs/ protocol fee tests). New fileindexer/tests/api_evidence_digest.rs.day=counts/mix include only the middle day.counts.swap.swap=1,counts.limit.fill=1.join/exit.ust1_mint+ust1_redeem: wrap counts exclude mint/redeem; fee-path rows keep all four distinct.book_takethat day → row still present, zeros.fee_usdnull → sourceamount_usdnull, idle sibling"0". Headline follows EFee-6 (priced SUM; do not zero wrap because AMM is unpriced).complete=false.day=matches calendar; does not need to match/protocol/fees?window=24h.otherbucket if that is the implemented cap; no trader addresses inby_token.counts.swap; Llama daily (if asserted in the same harness) still omits gems. Confirms this is not a Llama clone./protocol/fees,/protocol/fees/daily,/defillama/dailytests still pass without response-shape changes.serde_json::to_stringof the digest does not contain seeded actor bech32,tx_hashhex from fixtures, or keyssender/maker/actor_hash.Test plan (attack, hack, and abuse)
day—day=2026-09-03'%20OR%201=1--,day=2026-09-03;DROP TABLE swap_events. 400, tables intact.sender=terra1…,trader=,maker=,address=. 400 (preferred) or ignored; response must not filter or echo the address; counts remain protocol-wide.tx=,tx_hash=,include_txs=1,events=1,unredacted=1,redact=0,join=traders. Ignored or 400; body still has no txs/wallets.COUNT(DISTINCT sender)smuggling — if a future patch adds it, AC14 SQL assert fails. Test thatunique_tradersis absent even when overview has a non-zerounique_traders_24hin the same DB.day=2099-01-01→ 400.timestamp=— unix 00:00 on this route → 400 (wrong API), not a silent alias.window=24hon this route → 400 (wrong clock).POST/PUT/DELETE→ 405.format=csv→ 400."Internal server error", no sqlx.security.rs; 429 withRetry-After. Not LCD-heavy.eventsarray).by_token. Seeded user actors still absent. Do not put wrap-mapper or user wallets inby_token.symbol.pair_reservesortraders. Query spy / SQL string assert.Verification criteria
cargo test --manifest-path indexer/Cargo.toml --test api_evidence_digest -- --nocapture(and existingsecurity.rs/ protocol fee / defillama tests) green.GET /api/v1/evidence/digest?day=<seeded>against the test server;jqshowscounts+ sevenfees.by_source;rg -o 'terra1[a-z0-9]+'on the body matches only token contracts from fee mix fixtures (or matches nothing if only native denoms), never seeded traders.curl /api-docs/openapi.json | jq '.paths["/api/v1/evidence/digest"]'non-null.tx_hashallowed there, forbidden here) vs Llama vs trailing/protocol/fees./gt/events,/protocol/fees,/defillama/daily, or trader route contracts.Out of scope
actor_hash/tx_hash).tx_succeededjoin)./agent implement
cl8y-agent-control: queued
implementjobc7b93778-8a41-405a-8e52-5b53f767308a(not executed; no Hetzner VM)./agent implement
cl8y-agent-control: queued
design_authorjobd90b2425-e5b2-4675-9877-aa95148693f4(not executed; no Hetzner VM).cl8y-agent-control: queued
design_authorjob61f15a45-8b48-4f7d-bab1-bcea2967ceb8(not executed; no Hetzner VM).cl8y-agent-control: needs_human inbox card POST failed. Job stays parked.
Verified 2026-09-24 against main
54c4868e: #1206 is not closable. Current router exposes only GET /api/v1/evidence/daily (#1205); the digest handler/route, API integration tests, OpenAPI path, and make verify-issue-1206 are absent. No newer issue through #1335 tracks or replaces this scope, and #1205 is a separate tx-linkable event tape.Remaining:
Related: #1204, #1205, #586, #689, #631, #687, #216, #614, #557, #576, #1209, #1210, #1211, #1202.
The worktree adds proposed ADR 0014, the invariant row, architecture/runbook links, and the third-party agent playbook. Those docs do not claim the route is shipped.