fix(indexer): NUMERIC overflow on rolling trader volume refresh #1277
Labels
No labels
agent:fix_bugfix
agent:fix_conflicts
agent:fix_security
agent:gap_analysis
agent:implement
agent:implement
agent:implement
agent:open_issues
agent:ready
agent:research
agent:security_audit
agent:verify
architecture
backend
blocker:hybrid
blocker:launch
blocker:limit-orders
blocker:v2
block:log_only
block:security
bug
ci
contracts
correctness
deploy
dev
devops
docs
documentation
duplicate
e2e
enhancement
epic
feature
frontend
functional-completion
gas
good first issue
governance
help wanted
high-risk
hooks
hybrid
indexer
infra
infrastructure
integrators
invalid
launch-blocker
limit-orders
localnet
localterra
low priority
missing-implementation
needs-design
ops
performance
priority
high
priority
medium
product
qa
QA
question
ready
ready
research
scripts
security
security-hardening
smartcontracts
tech-debt
testing
ux
UX
v2
verification
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
code/cl8y-dex-terraclassic#1277
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
volume_aggregatorlogs Failed to refresh rolling trader volumes and leavestraders.volume_24h/volume_7d/volume_30dstale whenever a sender’s trailing-windowSUM(swap_events.offer_amount)cannot fitNUMERIC(38, 18).This is the same leftover type as pre-#548 global/pair raw volume: PostgreSQL
NUMERIC(38, 18)only stores|x| < 10^20. Raw 18-decimal CW20 notionals (~100 human USTR / CL8Y) already hit that bound. Pair and global rollups were widened toNUMERIC(38, 0)in20260817120000_*. Trader rolling columns were renamed fromvolume_*_usdin20260310000002_*and never widened. Positions/P&L went toNUMERIC(78, 18)in #676; this path is still the old(38, 18)raw-sum columns.Not a +1 on:
LEAST(…, POWER(10,38)-1), which does not fit(38, 18).NUMERIC(38, 0)for those raw columns. Trader rolling left behind.trader_positions/ P&L storage, not rolling volume refresh./healthgit SHA. Overflow is out of scope there.Current codebase
Refresh path that errors
refresh_all_volume_windowscallstraders::refresh_rolling_volumes. Onsqlx::Errorthe 5-minute loop logsFailed to refresh rolling trader volumes(startup = false→ error, GitLab #577 D5). Token / pair / global refresh in the same function can still succeed; trader windows stay at the last good stamp.LEASTto10^38-1is the integer cap used after #548 forNUMERIC(38, 0)destinations (token_volume_stats.volume,pair_volume_24h.volume_quote,global_stats_24h.total_volume). Assigning that value intoNUMERIC(38, 18)still overflows: integer width is only 20 digits.offer_amountis alreadyNUMERIC(38, 0)onswap_events. The overflow is the assignment intotraders.volume_*, not theSUMitself.Column types
swap_events.offer_amountNUMERIC(38, 0)global_stats_24h.total_volume,pair_volume_24h.volume_quote20260817120000_*(#548)NUMERIC(38, 0)traders.volume_24h/7d/30dvolume_*_usdNUMERIC(38, 18)traders.total_volumetotal_volume_usdNUMERIC(38, 18)traders.total_volume_usd20260818140000_*(#553)NUMERIC(38, 18)USD — keep; cap is10^20 - 10^-18upsert_traderadds rawoffer_amountintototal_volumewithout aLEASTcap. Lifetime raw volume can overflow on ingest even when the aggregator is not running. Bundle that column in the same widen.Existing test
raw_18_decimal_volume_does_not_overflow_global_statsinserts10^21raw USTR and asserts global/pair refresh. It does not callrefresh_rolling_volumes. Decay tests inindexer_volume_window_decay.rsuse 6-decimal seed sizes that never hit10^20.What already works
UPDATE; do not touch lifetime columns there.total_volume_usd) uses the(38, 18)/10^20cap on purpose. Do not widen USD.SUM(se.offer_amount)is computed on GET, not stored intraders.volume_*.Why the new implementation is needed
(38, 18)hole on the aggregator loop.Constraints / guardrails
NUMERIC(38, 0), matchingoffer_amount/pair_volume_24h.volume_quote/global_stats_24h.total_volume. Do not switch rolling volume toNUMERIC(78, 18)(#676 is inventory/P&L, not integer offer sums).LEAST(SUM(…), POWER(10::numeric, 38) - 1)on rolling sums after the destination is(38, 0). Do not “fix” overflow by clamping rolling raw volume to10^20 - 10^-18(that is the USD cap; it truncates ~100 human 18-dec tokens).total_volume/total_volume_usd/total_tradesin the idle-traderUPDATE(#577 D2 / #553).volume_usd_for_swap, P522-Q) unchanged.block_timestamp, no 24h wall-clock sleeps.SUM(swap_events)on trader GET for unscoped rolling windows (keep the rollup).upsert_tradermust not numeric-overflow on a single 18-dec swap oncetotal_volumeis(38, 0); still cap additive lifetime raw at10^38-1if a running sum could exceed the type.Relevant files
indexer/src/indexer/volume_aggregator.rsindexer/src/db/queries/traders.rsindexer/migrations/(widentraders.volume_24h,volume_7d,volume_30d,total_volume)indexer/migrations/20260817120000_backfill_swap_volume_usd_catalog.sqlindexer/tests/indexer_volume_window_decay.rsindexer/tests/volume_usd_catalog.rs(10^21raw USTR)indexer/src/api/traders.rs(bdplain decimal; no1e+19)docs/indexer-invariants.md(trailing window decay row),skills/AGENTS_INDEXER_VOLUME_WINDOW_DECAY.mdRecommended direction
ALTER TABLE traders ALTER COLUMN volume_24h TYPE NUMERIC(38, 0)(same forvolume_7d,volume_30d,total_volume). Comment that these are rawoffer_amountsums, not USD.LEAST(…, POWER(10,38)-1)on rollingSUMs. Add the same cap onupsert_tradertotal_volume = traders.total_volume + $2.offer_amount = 10^21inside 24h;refresh_rolling_volumesreturnsOk; storedvolume_24hequals the raw sum (or10^38-1only if the fixture exceeds that). Repeat for 7d/30d CASE branches. Asserttotal_volume_usdunchanged / still(38, 18)-legal.make verify-issue-<iid>plus an invariants/skill sentence that rolling raw volume is(38, 0)and must not fail the aggregator.Acceptance criteria
10^21raw offer:refresh_rolling_volumessucceeds;volume_24hstores that integer (plain decimal JSON, no scientific notation).upsert_traderwith10^21raw does not fail;total_volumeholds the running raw sum (LEASTat10^38-1).total_volume_usdremainsNUMERIC(38, 18)with the existing10^20USD cap. Unpriced stays NULL.make verify-issue-<iid>.refresh_all_volume_windowsstill run when trader refresh would previously have been the only failure (after the fix, trader refresh must not be the failure).Given / When / Then
Given
traders.volume_24h(and 7d/30d) are populated fromSUM(swap_events.offer_amount)and a sender has ≥10^20raw offer in the trailing windowWhen
volume_aggregatorrunsrefresh_rolling_volumes(startup or 5-minute loop)Then the statement commits without numeric overflow, rolling columns equal the capped raw integer, #577 zero-out still applies to idle senders, and USD lifetime columns are not rewritten
Given a new 18-decimal swap of
10^21raw offerWhen
upsert_traderadds that amount tototal_volumeThen ingest succeeds and
total_volumeis the raw running sum (capped at10^38-1), not a(38, 18)overflowTest plan (functional paths)
offer_amount = 10^21refresh_rolling_volumesOk;volume_24h = 10^21volume_24h = 0;volume_7d = 10^21total_volumestill includes the swap10^21upsert_trader10^21then refreshsort=volume_24hrefresh_all_volume_windows(..., false)Failed to refresh rolling trader volumesfor I1 fixturetotal_volume_usdstill(38, 18)legal; I1 18-dec raw does not smash USDTest plan (attack, hack, and abuse)
offer_amountat10^38LEASTto10^38-1; no panic; no wrap to negativeswap_eventsCHECK/type; do notCOALESCENULL into USD10^20cap “to reuse #553 SQL”total_volume_usdto “heal” overflowwindowstring concatblock_timestampinflating the windowUtc::now(); documentSUM(offer_amount)unboundedVerification criteria
make verify-issue-<iid>:cd indexer && cargo test --test indexer_volume_window_decay --test volume_usd_catalog --test api_traders -- --test-threads=1plus the new overflow test aftermake setup-indexer-postgresif needed.\d tradersshowsvolume_24h/volume_7d/volume_30d/total_volumeasnumeric(38,0);total_volume_usdstillnumeric(38,18).refresh_rolling_volumesreturnsOk(()); aggregator loop does not log the rolling-trader error for that dataset.Out of scope
/healthgit SHA (#1276).#676types.First-pass model recommendation
Recommendation: grok-high
Rationale: This is a schema migration on
tradersraw-volume columns plus aggregator SQL and overflow/decay tests. Composer’s local-edit bar fails on migration and on cross-check with #548 / #577 / #553 type caps. Wrong cap (10^20USD vs10^38-1integer) would silently truncate 18-decimal volume. Risk is data-integrity on leaderboard windows, not a three-file helper. Verify with the new10^21refresh test plus existingindexer_volume_window_decayandapi_traders.cl8y-agent-control: queued
design_authorjob6b144b5a-6583-4d3f-9253-15d40333c589(not executed; no Hetzner VM).Merged as PR #1292. Rolling trader volume columns are NUMERIC(38, 0).
Leftover: Coolify indexer migrate of
20260921120000_traders_rolling_volume_numeric_38_0(before #1263's20260921120001_pair_volume_30d). Tracked on #1300.Follow-up heal PR #1303 merged (migrate + gated poller before D5). First compile on this workstation failed; #1304 landed
&mut **tx.make verify-issue-12776/6 after #1304. R1–R7 remain on main via #1292.Operator leftover: Coolify apply
20260921130000_traders_lifetime_heal_from_swaps(after the widen + pair 30d versions on #1300). See #1305.