code/cl8y-research
2
0
Fork 0
Code Issues 10 Pull requests Projects Releases Packages Wiki Activity

Live week collection: DEX indexer, GitHub, GitLab, tokenlist, and prices #4

New issue
Open
opened 2026-08-31 05:32:57 +00:00 by PlasticDigits · 3 comments
PlasticDigits commented 2026-08-31 05:32:57 +00:00 (Migrated from gitlab.com)
Copy link

Summary

Make week able to collect live primary sources instead of only fixtures/*/sources.json: allowlisted DEX indexer HTTP, GitHub/GitLab activity on DEFAULT_REPOS, tokenlist JSON from cl8y-dex-terraclassic, and prices from the DEX indexer. --live today only swaps the Replicate client.

CI merge pipelines stay on fixtures. Live collect is explicit (--live-sources or equivalent).

Gap: gaps/GAP_1788152435.md §2, §4.3–4.4.

Current codebase

run_week requires fixtures/.../sources.json. fetch_allowlisted_json in src/collect.rs is unused. GITHUB_TOKEN / GITLAB_TOKEN are loaded and unused. Config::default_repos() is unused.

DEX path constant: DEX_OVERVIEW_PATH = "/api/v1/overview" on host indexer.dex.cl8y.com. Live documented fields include total_volume_24h_usd, hub USD (ustc_price_usd, custc_price_usd, ust1_price_usd, ustr_price_usd), liquidity, fees. Additional documented GETs on the same host: /api/v1/hub-prices, /api/v1/tokens (paginate; do not scrape HTML).

Tokenlist (not consumed):

https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/raw/main/tokenlist/tokenlist.json

CANONICAL_ADDRESSES is four hardcoded CL8Y-web #2 strings. Lint strips listed DEX CW20s (USTR, UST1, cLUNC, …) as “unofficial.”

Bridge: keep skip + docs/bridge-indexer-gap.md. Do not invent indexer.bridge.cl8y.com unless documented.

Why this is needed

Without live collect, weekly posts cannot cite real volume, repo shipping, listed tokens, or hub prices. Address lint fights ecosystem recaps. GitHub/GitLab tokens were added for this harvest.

Constraints / guardrails

  1. HTTPS, exact host allowlist, no redirects, timeouts, graceful gaps (no invented numbers).
  2. Time-bound every stat (as_of + citation URL/path).
  3. Repos: hosts github.com / gitlab.com, orgs PlasticDigits / CeramicLiberty / CL8Y only. Do not clone untrusted forks. Do not execute repo content (execute_me remains text).
  4. Do not scrape dex.cl8y.com / bridge.cl8y.com HTML.
  5. Tokenlist: exact raw path allowlist; do not open all of GitLab raw. Identity policy:
    • Official CL8Y = tokenDirectory.ts / current four addresses (sync, don’t drift).
    • DEX-listed = tokenlist addresses allowed in MDX as listed assets, not marketed as official CL8Y deployments. Prefer #token for official CL8Y.
    • Do not use tokenlist decimals as amount truth.
  6. Prices: only documented indexer JSON. Null USD stays a gap, not "0" invention (match indexer contract: unpriced ≠ zero).
  7. SSRF: no user-controlled URLs; no private/link-local; see security issue for DNS pinning.
  8. Secrets never in sources.json artifacts.
  9. Persist into pgvector when DATABASE_URL is set (store issue).

Relevant files

Path Role
src/collect.rs Fixture collect + unused live GET
src/allowlist.rs Host / repo checks
src/invariants.rs DEFAULT_REPOS, INDEXER_HOSTS, CANONICAL_ADDRESSES, DEX_OVERVIEW_PATH
src/config.rs Indexer bases, git tokens
src/numeric.rs Unsourced numbers
src/pipeline.rs Always loads fixture file
docs/bridge-indexer-gap.md Bridge skip
DEX indexer/src/api/overview.rs, hub_prices.rs Live JSON contract
cl8y-dex-terraclassic/tokenlist/tokenlist.json Listed assets

Recommended direction

  1. Split collect_live vs collect_from_fixture. week default fixtures; --live-sources uses live.
  2. DEX: GET {DEX_INDEXER_BASE}/api/v1/overview and GET .../api/v1/hub-prices via existing allowlisted GET (after DNS-pin work or same helper). Record numeric claims with RFC3339 as_of.
  3. GitHub/GitLab: list merged PRs/MRs / compare commits in a lookback window for DEFAULT_REPOS only. Events as text; wrap untrusted.
  4. Tokenlist: fetch allowlisted raw JSON; ingest SourceKind (new TokenList or metadata on records); feed lint allowlist for listed CW20/native denoms.
  5. Mock HTTP in tests (httpmock / wiremock). Never hit production from CI.

Acceptance criteria

  • --live-sources collects overview + hub-prices + tokenlist + allowlisted repo events without sources.json (or merges live over empty fixture).
  • Indexer 5xx/timeout → gap, week continues, no invented volume/price.
  • Unallowlisted remote skipped.
  • Tokenlist addresses (e.g. USTR CW20) are not stripped from otherwise-clean MDX; unofficial random 0x still stripped.
  • Official four CL8Y addresses stay canonical and tested against tokenDirectory constants (fixture copy or documented sync test).
  • Fixture cargo test / CI week unchanged (no live network).
  • Bridge still skipped with documented gap unless a real API is added later.

Test plan — functional paths

  1. Fixture happy path — existing tests still pass.
  2. Mock overview 200 — total_volume_24h_usd and hub prices become NumericClaims with citations.
  3. Mock overview 503 — gap; no 99999 invention.
  4. Mock hub-prices — ustr / ust1 / custc / lunc recorded; null price → gap for that ticker.
  5. Mock tokenlist — 11 tokens ingested; lint allows listed terra1; rejects 0x1111….
  6. Mock GitHub/GitLab — allowlisted org events ingested; github.com/attacker/malware skipped; execute_me not run.
  7. Empty repo week — no fabricated “we shipped X.”
  8. SSRF fixture — http://169.254.169.254/ still rejected.

Test plan — attack, hack, and abuse

  1. SSRF / open redirect — indexer URL with ?url=, punycode, cl8y.com.evil, redirects to metadata IP.
  2. Tokenlist path confusion — gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/raw/main/../../other rejected.
  3. Price spoof — extra JSON fields / huge numbers still require citation; unsourced body numbers fail emit.
  4. Community token as official CL8Y — ALPHA/USTRIX must not be described as official CL8Y deployment in lint/policy tests if we add a phrase check; addresses allowed as listed.
  5. Repo supply chain — README says curl \| sh; ingested as untrusted text.
  6. Token leak — GitHub/GitLab token not in sources.json.
  7. HTML scrape temptation — test that https://dex.cl8y.com is not in indexer fetch allowlist.

Verification criteria

  1. Reviewer runs fixture week without tokens.
  2. Reviewer runs --live-sources against mocks (and optionally staging indexer) and inspects sources.json citations.
  3. A draft mentioning the USTR tokenlist address is not stripped; a random 0x is.
## Summary Make `week` able to **collect live primary sources** instead of only `fixtures/*/sources.json`: allowlisted DEX indexer HTTP, GitHub/GitLab activity on `DEFAULT_REPOS`, **tokenlist** JSON from `cl8y-dex-terraclassic`, and **prices** from the DEX indexer. `--live` today only swaps the Replicate client. CI merge pipelines stay on fixtures. Live collect is explicit (`--live-sources` or equivalent). Gap: [`gaps/GAP_1788152435.md`](https://gitlab.com/PlasticDigits/cl8y-research/-/blob/main/gaps/GAP_1788152435.md) §2, §4.3–4.4. --- ## Current codebase `run_week` **requires** `fixtures/.../sources.json`. `fetch_allowlisted_json` in `src/collect.rs` is unused. `GITHUB_TOKEN` / `GITLAB_TOKEN` are loaded and unused. `Config::default_repos()` is unused. DEX path constant: `DEX_OVERVIEW_PATH = "/api/v1/overview"` on host `indexer.dex.cl8y.com`. Live documented fields include `total_volume_24h_usd`, hub USD (`ustc_price_usd`, `custc_price_usd`, `ust1_price_usd`, `ustr_price_usd`), liquidity, fees. Additional documented GETs on the same host: `/api/v1/hub-prices`, `/api/v1/tokens` (paginate; do not scrape HTML). Tokenlist (not consumed): ``` https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/raw/main/tokenlist/tokenlist.json ``` `CANONICAL_ADDRESSES` is four hardcoded CL8Y-web #2 strings. Lint **strips** listed DEX CW20s (USTR, UST1, cLUNC, …) as “unofficial.” Bridge: keep skip + `docs/bridge-indexer-gap.md`. Do not invent `indexer.bridge.cl8y.com` unless documented. --- ## Why this is needed Without live collect, weekly posts cannot cite real volume, repo shipping, listed tokens, or hub prices. Address lint fights ecosystem recaps. GitHub/GitLab tokens were added for this harvest. --- ## Constraints / guardrails 1. HTTPS, exact host allowlist, **no redirects**, timeouts, graceful gaps (no invented numbers). 2. Time-bound every stat (`as_of` + citation URL/path). 3. Repos: hosts `github.com` / `gitlab.com`, orgs `PlasticDigits` / `CeramicLiberty` / `CL8Y` only. Do not clone untrusted forks. Do not execute repo content (`execute_me` remains text). 4. **Do not scrape** `dex.cl8y.com` / `bridge.cl8y.com` HTML. 5. Tokenlist: exact raw path allowlist; do not open all of GitLab raw. Identity policy: - **Official CL8Y** = `tokenDirectory.ts` / current four addresses (sync, don’t drift). - **DEX-listed** = tokenlist addresses allowed in MDX as listed assets, **not** marketed as official CL8Y deployments. Prefer `#token` for official CL8Y. - Do not use tokenlist `decimals` as amount truth. 6. Prices: only documented indexer JSON. Null USD stays a gap, not `"0"` invention (match indexer contract: unpriced ≠ zero). 7. SSRF: no user-controlled URLs; no private/link-local; see security issue for DNS pinning. 8. Secrets never in `sources.json` artifacts. 9. Persist into pgvector when `DATABASE_URL` is set (store issue). --- ## Relevant files | Path | Role | | --- | --- | | `src/collect.rs` | Fixture collect + unused live GET | | `src/allowlist.rs` | Host / repo checks | | `src/invariants.rs` | `DEFAULT_REPOS`, `INDEXER_HOSTS`, `CANONICAL_ADDRESSES`, `DEX_OVERVIEW_PATH` | | `src/config.rs` | Indexer bases, git tokens | | `src/numeric.rs` | Unsourced numbers | | `src/pipeline.rs` | Always loads fixture file | | `docs/bridge-indexer-gap.md` | Bridge skip | | DEX `indexer/src/api/overview.rs`, `hub_prices.rs` | Live JSON contract | | `cl8y-dex-terraclassic/tokenlist/tokenlist.json` | Listed assets | --- ## Recommended direction 1. Split `collect_live` vs `collect_from_fixture`. `week` default fixtures; `--live-sources` uses live. 2. DEX: `GET {DEX_INDEXER_BASE}/api/v1/overview` and `GET .../api/v1/hub-prices` via existing allowlisted GET (after DNS-pin work or same helper). Record numeric claims with RFC3339 `as_of`. 3. GitHub/GitLab: list merged PRs/MRs / compare commits in a lookback window for `DEFAULT_REPOS` only. Events as text; wrap untrusted. 4. Tokenlist: fetch allowlisted raw JSON; ingest `SourceKind` (new `TokenList` or metadata on records); feed lint allowlist for listed CW20/native denoms. 5. Mock HTTP in tests (httpmock / wiremock). Never hit production from CI. --- ## Acceptance criteria - [ ] `--live-sources` collects overview + hub-prices + tokenlist + allowlisted repo events without `sources.json` (or merges live over empty fixture). - [ ] Indexer 5xx/timeout → gap, week continues, no invented volume/price. - [ ] Unallowlisted remote skipped. - [ ] Tokenlist addresses (e.g. USTR CW20) are not stripped from otherwise-clean MDX; unofficial random `0x` still stripped. - [ ] Official four CL8Y addresses stay canonical and tested against `tokenDirectory` constants (fixture copy or documented sync test). - [ ] Fixture `cargo test` / CI week **unchanged** (no live network). - [ ] Bridge still skipped with documented gap unless a real API is added later. --- ## Test plan — functional paths 1. **Fixture happy path** — existing tests still pass. 2. **Mock overview 200** — `total_volume_24h_usd` and hub prices become `NumericClaim`s with citations. 3. **Mock overview 503** — gap; no `99999` invention. 4. **Mock hub-prices** — `ustr` / `ust1` / `custc` / `lunc` recorded; null price → gap for that ticker. 5. **Mock tokenlist** — 11 tokens ingested; lint allows listed terra1; rejects `0x1111…`. 6. **Mock GitHub/GitLab** — allowlisted org events ingested; `github.com/attacker/malware` skipped; `execute_me` not run. 7. **Empty repo week** — no fabricated “we shipped X.” 8. **SSRF fixture** — `http://169.254.169.254/` still rejected. --- ## Test plan — attack, hack, and abuse 1. **SSRF / open redirect** — indexer URL with `?url=`, punycode, `cl8y.com.evil`, redirects to metadata IP. 2. **Tokenlist path confusion** — `gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/raw/main/../../other` rejected. 3. **Price spoof** — extra JSON fields / huge numbers still require citation; unsourced body numbers fail emit. 4. **Community token as official CL8Y** — ALPHA/USTRIX must not be described as official CL8Y deployment in lint/policy tests if we add a phrase check; addresses allowed as listed. 5. **Repo supply chain** — README says `curl \| sh`; ingested as untrusted text. 6. **Token leak** — GitHub/GitLab token not in `sources.json`. 7. **HTML scrape temptation** — test that `https://dex.cl8y.com` is not in indexer fetch allowlist. --- ## Verification criteria 1. Reviewer runs fixture week without tokens. 2. Reviewer runs `--live-sources` against mocks (and optionally staging indexer) and inspects `sources.json` citations. 3. A draft mentioning the USTR tokenlist address is not stripped; a random `0x` is.
PlasticDigits commented 2026-08-31 05:35:01 +00:00 (Migrated from gitlab.com)
Copy link

marked as related to #7

marked as related to #7
PlasticDigits commented 2026-08-31 05:35:02 +00:00 (Migrated from gitlab.com)
Copy link

marked as related to #10

marked as related to #10
PlasticDigits commented 2026-08-31 05:35:07 +00:00 (Migrated from gitlab.com)
Copy link

marked as related to #8

marked as related to #8
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
code/cl8y-research#4
No description provided.