bug: CoinGecko oracle polls 403 without a descriptive User-Agent #579

Closed
opened 2026-08-19 12:11:16 +00:00 by PlasticDigits · 15 comments
PlasticDigits commented 2026-08-19 12:11:16 +00:00 (Migrated from gitlab.com)

Summary

Indexer CoinGecko polls for USTC, LUNC, and vFDUSD/FDUSD return HTTP 403 with Please add a descriptive User-Agent to your request. The oracle HTTP client is built with timeout only — no User-Agent. CoinGecko is a soft-fail fallback (alternate ticks); USTC/LUNC still average KuCoin+MEXC, but vFDUSD’s CEX ticker has no KuCoin pair, so a CG miss leaves MEXC-only (logs: from 1/2 / from 1/1 sources).

Observed (columbus-5):

WARN Oracle: lunc coingecko failed: Parse error: CoinGecko HTTP 403 Forbidden: {"status":{"error_code":403,"error_message":"Please add a descriptive User-Agent to your request. For higher rate limits
WARN Oracle: vfdusd coingecko failed: Parse error: CoinGecko HTTP 403 Forbidden: …
INFO Oracle: LUNC/USD avg $0.00004772 from 2/3 sources
INFO Oracle: vFDUSD/USD avg $0.99800000 from 1/2 sources

Related: #515 / #550 (ticker oracle), #520 (extra CEX sources — do not substitute for a User-Agent). CEX FDUSD labeled as vFDUSD/USD is a separate identity issue.

Current codebase

Layer Behavior today
Client indexer/src/indexer/oracle.rs run_oracle_loop: Client::builder().timeout(Duration::from_secs(10)).build(). No default_headers, no user-agent. reqwest’s default UA is empty or reqwest/x.y — CoinGecko free API now rejects that with 403 (not 429).
Fetch fetch_coingecko_url: 429 or JSON error_code: 429 → OracleError::RateLimited (debug log). Any other non-success → OracleError::Parse("CoinGecko HTTP {status}: {body…}") → warn. 403 therefore looks like a parse error and is not retried as rate-limit.
Schedule CoinGecko on alternate ticks (tick_count % 2 == 0) to spare quota. KuCoin+MEXC every tick.
vFDUSD sources KuCoin skipped (kucoin_symbol() = None). Alternate tick: MEXC + CG = 2 slots; CG 403 → 1/2. Non-CG tick: MEXC only = 1/1.
USTC/LUNC Still 2/3 when CG fails (KuCoin+MEXC). Average quality drops; operators see warn spam.
Tests fetch_coingecko_maps_429_to_rate_limited covers 429. coingecko_body_is_rate_limited is 429-only. No 403 / User-Agent test. Wiremock helpers do not assert request headers.
Other HTTP Indexer does not set a project User-Agent elsewhere for CEX polls (KuCoin/MEXC currently succeed without it).

Why this is needed

  1. Documented fallback is dead. Skills say “don’t treat CoinGecko 429 as a hard outage; KuCoin/MEXC usually suffice.” 403 is a client misconfiguration, not quota. Every CG tick is a guaranteed miss until UA is set.
  2. vFDUSD/FDUSD is single-sourced. No KuCoin. CG 403 means the ticker’s robustness claim (2 aggregators) is false in production; MEXC outage would retain last price with no second live source on CG ticks either.
  3. Warn noise hides real outages. Permanent 403 looks like an intermittent parse failure. Operators cannot tell “fix the User-Agent” from “CoinGecko is down.”
  4. Do not work around with UA spoofing / rotating browsers. CoinGecko asks for a descriptive UA so they can contact operators. A stable indexer identity is the intended fix; a fake Chrome UA is abuse.

Constraints / guardrails

  1. Scope. Set a stable, descriptive User-Agent on the oracle reqwest client (all KuCoin/MEXC/CG requests on that client is OK). Map CoinGecko 403 whose body asks for User-Agent to a distinct, actionable log (not “Parse error”). Optional: treat 403-UA as a one-line error once, then debug, to avoid spam — do not hide a 403 that is not UA-related (key/ban).
  2. UA content. Identify the software, e.g. cl8y-dex-indexer/<version> (+https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic). Do not impersonate browsers, wallets, or other products. Do not put emails, tokens, hostnames of the operator, or wallet addresses in the UA. Version may come from CARGO_PKG_VERSION.
  3. Do not rotate UAs, randomize, or retry 403 with a different UA to evade limits. Do not scrape CoinGecko HTML. Do not raise poll frequency because 403 stopped.
  4. 429 vs 403. Keep 429 → RateLimited (debug). 403 UA-missing is not rate-limited success. After the fix, 403 should be rare; tests must still distinguish 429.
  5. X1–X6 / P550. Do not change ticker allowlists, $1 hardcodes, or volume_usd catalog. Soft-fail: CG down still must not blank KuCoin/MEXC averages.
  6. Secrets. Do not log full CoinGecko bodies if they could contain keys; truncating to ~120 chars is already in fetch_coingecko_url — keep truncation. Do not put API keys in UA or query string unless a documented paid plan is added in a separate issue (this issue is UA-only; no CoinGecko API key required for the free simple/price endpoint).
  7. Do not “fix” 403 by deleting the CoinGecko source. USTC/LUNC still want a third aggregator when UA is valid.
  8. Wiremock / CI. Tests must assert the UA header on the mock; do not hit live CoinGecko in unit tests.

Relevant files

File Role
indexer/src/indexer/oracle.rs Client builder, fetch_coingecko_url, error mapping, tests
indexer/tests/api_oracle.rs Only if API behavior changes (should not)
docs/runbooks/indexer-external-oracle.md Note CG requires descriptive UA; 403 vs 429
skills/AGENTS_INDEXER_EXTERNAL_ORACLE.md Don’t ship oracle client without UA; 403 UA ≠ 429
new scripts/qa/verify-issue-<iid>.sh Grep UA + tests
  1. Client::builder().user_agent(ORACLE_USER_AGENT).timeout(...) with a const / CARGO_PKG_VERSION string as above.
  2. In fetch_coingecko_url, if status is 403 and body mentions User-Agent (or error_code 403 with that message), return a dedicated OracleError variant (e.g. MissingUserAgent) or keep Parse but stable wording CoinGecko HTTP 403: User-Agent required without dumping the whole JSON every tick.
  3. Tests: wiremock 200 still parses; request user-agent matches the const; 403+UA message ≠ RateLimited; 429 still RateLimited.
  4. Docs one-liner in the external-oracle runbook Sources section.
  5. make verify-issue-<iid>.

Acceptance criteria

  • AC1. Oracle reqwest client sends a descriptive, non-browser User-Agent including the indexer name and a repo URL or package name.
  • AC2. Unit test fails if the CG mock is called without that header.
  • AC3. CoinGecko 429 still maps to RateLimited; 403 User-Agent missing is not classified as 429.
  • AC4. KuCoin/MEXC/CG poll loop unchanged aside from headers/error mapping. Soft-fail: one source down still averages the rest. No $1 peg. No ticker rename.
  • AC5. Docs/skill mention descriptive UA and 403 vs 429. make verify-issue-<iid> green without live CoinGecko.

Test plan (all paths)

# Path Steps Expected
T1 CG 200 Wiremock simple/price + UA matcher Parses USD; request has UA
T2 CG 200 missing UA Mock requires UA header Client with UA succeeds; a client without UA would 403 (test the production client, not a stripped one, for AC2)
T3 CG 429 Existing test RateLimited
T4 CG 403 UA body 403 + CoinGecko status JSON Not RateLimited; error distinguishable; body truncated
T5 CG 403 other 403 without UA message (e.g. IP ban) Soft-fail warn; does not look like success
T6 CG 5xx / JSON miss 500 / missing .usd Parse/HTTP error; last-known retained at poll layer
T7 Alternate tick fetch_coingecko = false No CG call; KuCoin/MEXC still run
T8 vFDUSD no KuCoin CG 403 + MEXC ok Average from MEXC; does not abort the ticker
T9 USTC/LUNC CG 403 + two CEX ok 2/3 average; USTC/LUNC ids unchanged
T10 Non-finite NaN/inf from parse X6 safe BigDecimal; no panic
T11 KuCoin/MEXC Still work with the new default UA No regression (wiremock or existing tests)
T12 Docs verify script Runbook/skill mention User-Agent

Test plan (attack, hack, abuse)

# Vector Steps Expected
A1 UA spoof Code review No Chrome/Firefox/Keplr impersonation
A2 UA injection Version/env interpolation UA is compile-time / crate version only; no unsanitized env in UA
A3 Secret in UA GITLAB_TOKEN / RPC URL Must not appear in UA or warn logs
A4 Rotate UA to evade 429 Retry loops No UA rotation; 429 stays RateLimited + backoff via tick skip
A5 SSRF via coin id Ticker enum only coingecko_id() is fixed per OracleTicker; no user string in URL
A6 Log exfiltration Long 403 body Truncate; no cookie/header dump
A7 Amplification Shorten poll interval “to compensate 403” Forbidden; interval stays config
A8 Fake success 403 treated as 1.0 USD Must not insert a price from an error body

Verification criteria

make verify-issue-<iid>
# expected:
# - oracle.rs user_agent + tests for header / 403 vs 429
# - cargo test --lib oracle
# - skill/runbook UA note
make verify-issue-515   # catalog / X1–X6 unchanged

Ship is done when AC1–AC5 pass, T1–T12 and A1–A8 are recorded (or waived with reason), and a local wiremock CG 200 with UA matcher is green (make setup-indexer-postgres not required for --lib oracle).

## Summary Indexer CoinGecko polls for **USTC**, **LUNC**, and **vFDUSD/FDUSD** return **HTTP 403** with `Please add a descriptive User-Agent to your request`. The oracle HTTP client is built with timeout only — no `User-Agent`. CoinGecko is a soft-fail fallback (alternate ticks); USTC/LUNC still average KuCoin+MEXC, but **vFDUSD’s CEX ticker has no KuCoin pair**, so a CG miss leaves **MEXC-only** (logs: `from 1/2` / `from 1/1` sources). Observed (columbus-5): ``` WARN Oracle: lunc coingecko failed: Parse error: CoinGecko HTTP 403 Forbidden: {"status":{"error_code":403,"error_message":"Please add a descriptive User-Agent to your request. For higher rate limits WARN Oracle: vfdusd coingecko failed: Parse error: CoinGecko HTTP 403 Forbidden: … INFO Oracle: LUNC/USD avg $0.00004772 from 2/3 sources INFO Oracle: vFDUSD/USD avg $0.99800000 from 1/2 sources ``` Related: [#515](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/515) / [#550](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/550) (ticker oracle), [#520](https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic/-/issues/520) (extra CEX sources — do not substitute for a User-Agent). CEX FDUSD labeled as vFDUSD/USD is a **separate identity** issue. ## Current codebase | Layer | Behavior today | |-------|----------------| | **Client** | [`indexer/src/indexer/oracle.rs`](indexer/src/indexer/oracle.rs) `run_oracle_loop`: `Client::builder().timeout(Duration::from_secs(10)).build()`. No `default_headers`, no `user-agent`. reqwest’s default UA is empty or `reqwest/x.y` — CoinGecko free API now rejects that with **403** (not 429). | | **Fetch** | `fetch_coingecko_url`: 429 or JSON `error_code: 429` → `OracleError::RateLimited` (debug log). Any other non-success → `OracleError::Parse("CoinGecko HTTP {status}: {body…}")` → **warn**. 403 therefore looks like a parse error and is not retried as rate-limit. | | **Schedule** | CoinGecko on **alternate ticks** (`tick_count % 2 == 0`) to spare quota. KuCoin+MEXC every tick. | | **vFDUSD sources** | KuCoin skipped (`kucoin_symbol() = None`). Alternate tick: MEXC + CG = 2 slots; CG 403 → 1/2. Non-CG tick: MEXC only = 1/1. | | **USTC/LUNC** | Still 2/3 when CG fails (KuCoin+MEXC). Average quality drops; operators see warn spam. | | **Tests** | `fetch_coingecko_maps_429_to_rate_limited` covers 429. `coingecko_body_is_rate_limited` is 429-only. **No** 403 / User-Agent test. Wiremock helpers do not assert request headers. | | **Other HTTP** | Indexer does not set a project User-Agent elsewhere for CEX polls (KuCoin/MEXC currently succeed without it). | ## Why this is needed 1. **Documented fallback is dead.** Skills say “don’t treat CoinGecko 429 as a hard outage; KuCoin/MEXC usually suffice.” 403 is a **client misconfiguration**, not quota. Every CG tick is a guaranteed miss until UA is set. 2. **vFDUSD/FDUSD is single-sourced.** No KuCoin. CG 403 means the ticker’s robustness claim (2 aggregators) is false in production; MEXC outage would retain last price with no second live source on CG ticks either. 3. **Warn noise hides real outages.** Permanent 403 looks like an intermittent parse failure. Operators cannot tell “fix the User-Agent” from “CoinGecko is down.” 4. **Do not work around with UA spoofing / rotating browsers.** CoinGecko asks for a **descriptive** UA so they can contact operators. A stable indexer identity is the intended fix; a fake Chrome UA is abuse. ## Constraints / guardrails 1. **Scope.** Set a **stable, descriptive** `User-Agent` on the oracle `reqwest` client (all KuCoin/MEXC/CG requests on that client is OK). Map CoinGecko **403 whose body asks for User-Agent** to a distinct, actionable log (not “Parse error”). Optional: treat 403-UA as a one-line `error` once, then `debug`, to avoid spam — do not hide a 403 that is **not** UA-related (key/ban). 2. **UA content.** Identify the software, e.g. `cl8y-dex-indexer/<version> (+https://gitlab.com/PlasticDigits/cl8y-dex-terraclassic)`. Do **not** impersonate browsers, wallets, or other products. Do **not** put emails, tokens, hostnames of the operator, or wallet addresses in the UA. Version may come from `CARGO_PKG_VERSION`. 3. **Do not** rotate UAs, randomize, or retry 403 with a different UA to evade limits. Do **not** scrape CoinGecko HTML. Do **not** raise poll frequency because 403 stopped. 4. **429 vs 403.** Keep 429 → `RateLimited` (debug). 403 UA-missing is **not** rate-limited success. After the fix, 403 should be rare; tests must still distinguish 429. 5. **X1–X6 / P550.** Do not change ticker allowlists, `$1` hardcodes, or `volume_usd` catalog. Soft-fail: CG down still must not blank KuCoin/MEXC averages. 6. **Secrets.** Do not log full CoinGecko bodies if they could contain keys; truncating to ~120 chars is already in `fetch_coingecko_url` — keep truncation. Do not put API keys in UA or query string unless a documented paid plan is added in a **separate** issue (this issue is UA-only; no CoinGecko API key required for the free simple/price endpoint). 7. **Do not** “fix” 403 by deleting the CoinGecko source. USTC/LUNC still want a third aggregator when UA is valid. 8. **Wiremock / CI.** Tests must assert the UA header on the mock; do not hit live CoinGecko in unit tests. ## Relevant files | File | Role | |------|------| | [`indexer/src/indexer/oracle.rs`](indexer/src/indexer/oracle.rs) | Client builder, `fetch_coingecko_url`, error mapping, tests | | [`indexer/tests/api_oracle.rs`](indexer/tests/api_oracle.rs) | Only if API behavior changes (should not) | | [`docs/runbooks/indexer-external-oracle.md`](docs/runbooks/indexer-external-oracle.md) | Note CG requires descriptive UA; 403 vs 429 | | [`skills/AGENTS_INDEXER_EXTERNAL_ORACLE.md`](skills/AGENTS_INDEXER_EXTERNAL_ORACLE.md) | Don’t ship oracle client without UA; 403 UA ≠ 429 | | new `scripts/qa/verify-issue-<iid>.sh` | Grep UA + tests | ## Recommended direction 1. `Client::builder().user_agent(ORACLE_USER_AGENT).timeout(...)` with a `const` / `CARGO_PKG_VERSION` string as above. 2. In `fetch_coingecko_url`, if status is 403 and body mentions `User-Agent` (or `error_code` 403 with that message), return a dedicated `OracleError` variant (e.g. `MissingUserAgent`) or keep `Parse` but **stable** wording `CoinGecko HTTP 403: User-Agent required` without dumping the whole JSON every tick. 3. Tests: wiremock 200 still parses; request `user-agent` matches the const; 403+UA message ≠ `RateLimited`; 429 still `RateLimited`. 4. Docs one-liner in the external-oracle runbook Sources section. 5. `make verify-issue-<iid>`. ## Acceptance criteria - [ ] **AC1.** Oracle `reqwest` client sends a descriptive, non-browser `User-Agent` including the indexer name and a repo URL or package name. - [ ] **AC2.** Unit test fails if the CG mock is called without that header. - [ ] **AC3.** CoinGecko 429 still maps to `RateLimited`; 403 User-Agent missing is **not** classified as 429. - [ ] **AC4.** KuCoin/MEXC/CG poll loop unchanged aside from headers/error mapping. Soft-fail: one source down still averages the rest. No `$1` peg. No ticker rename. - [ ] **AC5.** Docs/skill mention descriptive UA and 403 vs 429. `make verify-issue-<iid>` green without live CoinGecko. ## Test plan (all paths) | # | Path | Steps | Expected | |---|------|-------|----------| | T1 | CG 200 | Wiremock simple/price + UA matcher | Parses USD; request has UA | | T2 | CG 200 missing UA | Mock requires UA header | Client with UA succeeds; a client **without** UA would 403 (test the production client, not a stripped one, for AC2) | | T3 | CG 429 | Existing test | `RateLimited` | | T4 | CG 403 UA body | 403 + CoinGecko status JSON | Not `RateLimited`; error distinguishable; body truncated | | T5 | CG 403 other | 403 without UA message (e.g. IP ban) | Soft-fail warn; does not look like success | | T6 | CG 5xx / JSON miss | 500 / missing `.usd` | Parse/HTTP error; last-known retained at poll layer | | T7 | Alternate tick | `fetch_coingecko = false` | No CG call; KuCoin/MEXC still run | | T8 | vFDUSD no KuCoin | CG 403 + MEXC ok | Average from MEXC; does not abort the ticker | | T9 | USTC/LUNC | CG 403 + two CEX ok | 2/3 average; USTC/LUNC ids unchanged | | T10 | Non-finite | NaN/inf from parse | X6 safe BigDecimal; no panic | | T11 | KuCoin/MEXC | Still work with the new default UA | No regression (wiremock or existing tests) | | T12 | Docs | verify script | Runbook/skill mention User-Agent | ## Test plan (attack, hack, abuse) | # | Vector | Steps | Expected | |---|--------|-------|----------| | A1 | UA spoof | Code review | No Chrome/Firefox/Keplr impersonation | | A2 | UA injection | Version/env interpolation | UA is compile-time / crate version only; no unsanitized env in UA | | A3 | Secret in UA | `GITLAB_TOKEN` / RPC URL | Must not appear in UA or warn logs | | A4 | Rotate UA to evade 429 | Retry loops | No UA rotation; 429 stays RateLimited + backoff via tick skip | | A5 | SSRF via coin id | Ticker enum only | `coingecko_id()` is fixed per `OracleTicker`; no user string in URL | | A6 | Log exfiltration | Long 403 body | Truncate; no cookie/header dump | | A7 | Amplification | Shorten poll interval “to compensate 403” | Forbidden; interval stays config | | A8 | Fake success | 403 treated as 1.0 USD | Must not insert a price from an error body | ## Verification criteria ```bash make verify-issue-<iid> # expected: # - oracle.rs user_agent + tests for header / 403 vs 429 # - cargo test --lib oracle # - skill/runbook UA note make verify-issue-515 # catalog / X1–X6 unchanged ``` Ship is done when AC1–AC5 pass, T1–T12 and A1–A8 are recorded (or waived with reason), and a local wiremock CG 200 with UA matcher is green (`make setup-indexer-postgres` not required for `--lib oracle`).
PlasticDigits commented 2026-08-19 12:11:17 +00:00 (Migrated from gitlab.com)

marked as related to #515

marked as related to #515
PlasticDigits commented 2026-08-19 12:11:18 +00:00 (Migrated from gitlab.com)

marked as related to #550

marked as related to #550
PlasticDigits commented 2026-08-19 12:11:19 +00:00 (Migrated from gitlab.com)

marked as related to #520

marked as related to #520
PlasticDigits commented 2026-08-19 12:11:27 +00:00 (Migrated from gitlab.com)

mentioned in issue #580

mentioned in issue #580
PlasticDigits commented 2026-08-19 12:11:30 +00:00 (Migrated from gitlab.com)

marked as related to #580

marked as related to #580
PlasticDigits commented 2026-08-20 01:38:56 +00:00 (Migrated from gitlab.com)

mentioned in merge request !1098

mentioned in merge request !1098
PlasticDigits commented 2026-08-20 02:19:43 +00:00 (Migrated from gitlab.com)

mentioned in merge request !1103

mentioned in merge request !1103
PlasticDigits commented 2026-08-20 02:21:34 +00:00 (Migrated from gitlab.com)

mentioned in merge request !1104

mentioned in merge request !1104
PlasticDigits commented 2026-08-20 03:17:28 +00:00 (Migrated from gitlab.com)

mentioned in commit a30edba3c7

mentioned in commit a30edba3c7a6474482763c75554a9a6f8376d072
PlasticDigits (Migrated from gitlab.com) closed this issue 2026-08-20 03:17:29 +00:00
PlasticDigits commented 2026-08-20 03:54:19 +00:00 (Migrated from gitlab.com)

Merged via !1098.

make verify-issue-579 passed (docs + --lib oracle User-Agent / 403 vs 429). Remaining: production indexer must be redeployed so live CoinGecko polls send the descriptive User-Agent. No code follow-up.

Merged via !1098. `make verify-issue-579` passed (docs + `--lib oracle` User-Agent / 403 vs 429). Remaining: production indexer must be redeployed so live CoinGecko polls send the descriptive User-Agent. No code follow-up.
PlasticDigits commented 2026-08-20 03:54:51 +00:00 (Migrated from gitlab.com)

mentioned in issue #583

mentioned in issue #583
PlasticDigits commented 2026-08-20 03:55:04 +00:00 (Migrated from gitlab.com)

marked as related to #583

marked as related to #583
PlasticDigits commented 2026-08-24 03:15:46 +00:00 (Migrated from gitlab.com)

mentioned in issue #619

mentioned in issue #619
PlasticDigits commented 2026-08-27 01:00:19 +00:00 (Migrated from gitlab.com)

mentioned in issue #682

mentioned in issue #682
PlasticDigits commented 2026-08-27 01:00:29 +00:00 (Migrated from gitlab.com)

mentioned in issue #683

mentioned in issue #683
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
code/cl8y-dex-terraclassic#579
No description provided.