fix(frontend): dex.cl8y.com robots.txt and sitemap.xml must not be the SPA shell

ops-worker commented

2026-09-06 12:05:16 +00:00

Member

Summary

https://dex.cl8y.com/robots.txt and /sitemap.xml must be real crawl files, not the Vite SPA shell.

Live 2026-09-13: both URLs return 200 text/html, cache-control: no-cache, must-revalidate, same etag as / (6aa507fa-e06), body starts <!doctype html>. Root cause: files are missing from frontend-dapp/public/, and docker/frontend/nginx.conf location / uses try_files $uri $uri/ /index.html. The og-image exception already shows the correct pattern (location = /og-image.png → try_files $uri =404).

This issue bundles robots + sitemap + exact-location 404. Do not split. Do not invent unique per-route SEO titles for the SPA.

Closed marketing tracker: cl8y-marketing#2. Do not reopen it for DEX nginx work.

Given / When / Then

Given Coolify serves dex.cl8y.com from docker/frontend nginx + Vite dist
When a crawler fetches /robots.txt and /sitemap.xml
Then the responses are text/plain robots syntax and application/xml urlset (not SPA HTML)
And if those files are missing after a bad deploy, nginx returns 404, not HTML 200
And /og-image.png and hashed JS/CSS miss behavior stay unchanged (#578 / #706)

Not a duplicate / already implemented

Issue	Why it is not this ticket
#578 (closed)	OG/Twitter image + absolute meta. Exact-location pattern to copy. Keep.
#706	Missing hashed JS/CSS must 404 (`@hashed_asset_miss`). Complementary; do not weaken.
#488	Design-system / first OG card.
#665	In-app Share. Not crawler files.
#1209–#1211	Fee ledger. Unrelated.

Not implemented: no robots.txt / sitemap.xml under frontend-dapp/public/. nginx has no location = /robots.txt / /sitemap.xml. index.html has no robots / noindex meta. Confirmed live HTML shell 2026-09-13.

Current codebase

Coolify → docker/frontend/Dockerfile: Node npm run build, runtime nginx:alpine, COPY nginx.conf + dist.

Vite copies frontend-dapp/public/ to the dist root. nginx:

location / {
    try_files $uri $uri/ /index.html;
}
location = /og-image.png {
    try_files $uri =404;
}
location ~* \.(js|css|png|jpg|jpeg|gif|svg|ico|woff2?)$ {
    try_files $uri @hashed_asset_miss;
}

.txt and .xml are not in the hashed-asset regex, so missing robots/sitemap hit SPA fallback. index.html is one OG/title shell for every route (OG-5: no prerender, no per-route titles).

frontend-dapp/public/ today: favicons, bootstrap, wallets, chainlist, og-image.png, sounds — no robots/sitemap.

If files were added to public/ only, try_files $uri would serve them while present. Default nginx mime: .txt → text/plain (good); .xml → text/xml (acceptance wants application/xml). If the files go missing later, SPA fallback returns HTML 200 again. Exact location = + =404 is required.

Why the new implementation is needed

Crawlers that fetch /robots.txt and /sitemap.xml receive the DEX SPA HTML. Search Console can treat that as a soft-404 / HTML robots file. Adding files without exact-location nginx will regress the first time a deploy omits them.

The DEX SPA must not become the SEO content host. This ticket only makes crawl endpoints honest and keeps wallet/manage/mint/unbounded pair routes noindex or omitted. Unique canonicals belong on a future static/prerender host (docs.cl8y.com is currently not an implementable surface).

Constraints / guardrails

Do not noindex or Disallow a future content host (docs.cl8y.com / cl8y.com content).
Do keep DEX SPA routes noindex or omitted from the DEX sitemap (especially /portfolio, /trader/:address, /token/:addr/manage, /mint, unbounded pair URLs).
DEX remains one OG shell — do not invent unique titles/canonicals for SPA routes (OG-5).
No unverified fee/volume/“best DEX” claims in robots comments, sitemap extras, or meta.
Preserve og-image exact-location and hashed-asset 404 (@hashed_asset_miss). Do not widen SPA fallback.
nginx default .xml → text/xml; set default_type application/xml (or equivalent) for /sitemap.xml.
Sitemap: in robots may point at a content host only after that host has a real sitemap; until then use a minimal DEX sitemap or omit the Sitemap line.
Out of scope: marketing claims, listings, analytics, FeeSource, PostHog, cl8y.com robots (follow-on), prerender/react-helmet.

Relevant files

Path	Why
`docker/frontend/nginx.conf`	SPA fallback; add exact locations
`docker/frontend/Dockerfile`	Dist image
`frontend-dapp/public/`	Add `robots.txt` + `sitemap.xml`
`frontend-dapp/index.html`	Single crawler shell; optional `X-Robots-Tag` is nginx, not helmet
`frontend-dapp/src/App.tsx`	Route inventory for omit/noindex
`scripts/qa/verify-issue-578.sh`	nginx grep + optional container curl pattern
`skills/AGENTS_FRONTEND_OPENGRAPH.md`	OG-5 / og-image exception to keep
`docs/frontend.md`	Crawler shell documentation

Recommended direction

Add frontend-dapp/public/robots.txt (User-agent syntax, text/plain). Conservative: do not Allow unbounded /trader/ or /token/*/manage.
Add frontend-dapp/public/sitemap.xml — minimal urlset of first-party DEX URLs that are intentionally crawlable (likely none or only / if the SPA should stay noindex). Omit wallet, manage, mint, and infinite pair/trader paths. Hosts in the urlset are https://dex.cl8y.com only.
Exact nginx locations (mirror og-image):

location = /robots.txt {
    default_type text/plain;
    add_header Cache-Control "public, max-age=3600";
    try_files $uri =404;
}
location = /sitemap.xml {
    default_type application/xml;
    add_header Cache-Control "public, max-age=3600";
    try_files $uri =404;
}

Optional: X-Robots-Tag: noindex, nofollow on location = /index.html so crawlers skip the transaction UI even if they ignore an empty sitemap.
Verify siblings (/sitemap_index.xml, /robots.txt/ trailing slash) are 404 or real files — not SPA HTML 200.
make verify-issue-1212: grep exact locations; local nginx container curl type + body sniff; negative test with files renamed → 404.

Reject: relying on Disallow as a security control; listing attacker-controlled hosts in the sitemap; helmet/prerender in this ticket.

Acceptance criteria

AC1. GET https://dex.cl8y.com/robots.txt → text/plain, body is robots syntax (User-agent), not <!doctype / Vite SPA.
AC2. GET https://dex.cl8y.com/sitemap.xml → application/xml (or +xml), body is an XML urlset (empty valid urlset OK), not SPA HTML.
AC3. Missing robots/sitemap after a bad deploy → 404, not HTML 200.
AC4. SPA-only routes are noindex and/or omitted from the sitemap (/portfolio, /trader/:address, /token/:addr/manage, /mint, unbounded pair URLs).
AC5. /og-image.png still image/png; hashed JS/CSS misses still 404 (#578 / #706).
AC6. No marketing claims in robots/sitemap. No third-party or attacker-controlled URLs in the urlset. Hosts are the DEX origin only.
AC7. Re-check after Coolify deploy with curl Content-Type and body sniff (not status code alone). Local verify script exists.

Test plan (functional paths)

#	Path	Expect
T1	Present files (local image + live after deploy)	robots `text/plain` + `User-agent`; sitemap `application/xml` + `<urlset`
T2	Missing files (rename in local nginx)	Exact location 404, not `index.html`
T3	SPA routes `/`, `/trade`, `/portfolio`	Still serve the app to browsers
T4	Siblings `/sitemap_index.xml`, `/robots.txt.bak`	Not SPA HTML 200
T5	`/og-image.png` + hashed asset miss	#578 / #706 unchanged
T6	Sitemap contents	Only allowlisted first-party paths; empty urlset OK
T7	`make verify-issue-578` / `#706`	Still green
T8	Optional `X-Robots-Tag` on HTML	Present if implemented; browsers still get the app

Playwright not required. Vitest not required unless a static-file unit test is added.

Test plan (attack, hack, and abuse)

#	Vector	Expect
A1	Sitemap poisoning (phishing / extra hosts)	CI or review allowlists origin; only `https://dex.cl8y.com` URLs
A2	Listing `/trader/:addr`, `/token/.../manage`, `/mint`	Omitted; robots does not `Allow` those wildcards
A3	Relying on `Disallow` to hide secrets	Forbidden as a security control; manage/mint stay behind wallet UX
A4	Overly broad `Allow:` of dynamic segments	Rejected; prefer omit + HTML noindex
A5	SPA fallback regression (deleted public files)	404 from exact location
A6	Content-Type lie (HTML labeled robots)	Fail acceptance; sniff body
A7	`Sitemap:` pointing at a foreign host	Pin to known CL8Y origin or omit
A8	Comments/claims in robots that overstate product	No marketing claims
A9	Open redirect / `javascript:` in urlset	Forbidden; static first-party paths only

Verification criteria

make verify-issue-1212 (grep nginx exact locations + local container curl type/body + negative 404).
make verify-issue-578 and make verify-issue-706 still green.
Diff of nginx.conf shows exact locations without changing hashed-asset 404 or og-image.
Post-deploy curl -sI and body checks on https://dex.cl8y.com/robots.txt and /sitemap.xml vs the 2026-09-13 HTML-shell baseline.
docs/frontend.md notes that crawl endpoints are real files; unique canonicals remain blocked on the missing content host.

## Summary `https://dex.cl8y.com/robots.txt` and `/sitemap.xml` must be real crawl files, not the Vite SPA shell. **Live 2026-09-13:** both URLs return **200 `text/html`**, `cache-control: no-cache, must-revalidate`, same etag as `/` (`6aa507fa-e06`), body starts `<!doctype html>`. Root cause: files are missing from `frontend-dapp/public/`, and [`docker/frontend/nginx.conf`](docker/frontend/nginx.conf) `location /` uses `try_files $uri $uri/ /index.html`. The og-image exception already shows the correct pattern (`location = /og-image.png` → `try_files $uri =404`). This issue **bundles robots + sitemap + exact-location 404**. Do not split. Do **not** invent unique per-route SEO titles for the SPA. Closed marketing tracker: [cl8y-marketing#2](https://git.cl8y.com/PlasticDigits/cl8y-marketing/issues/2). Do not reopen it for DEX nginx work. ### Given / When / Then Given Coolify serves `dex.cl8y.com` from `docker/frontend` nginx + Vite `dist` When a crawler fetches `/robots.txt` and `/sitemap.xml` Then the responses are `text/plain` robots syntax and `application/xml` `urlset` (not SPA HTML) And if those files are missing after a bad deploy, nginx returns **404**, not HTML 200 And `/og-image.png` and hashed JS/CSS miss behavior stay unchanged (#578 / #706) ### Not a duplicate / already implemented | Issue | Why it is not this ticket | | --- | --- | | [#578](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/578) (closed) | OG/Twitter **image** + absolute meta. Exact-location pattern to copy. Keep. | | [#706](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/706) | Missing hashed JS/CSS must 404 (`@hashed_asset_miss`). Complementary; do not weaken. | | [#488](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/488) | Design-system / first OG card. | | [#665](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/665) | In-app Share. Not crawler files. | | [#1209](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1209)–[#1211](https://git.cl8y.com/code/cl8y-dex-terraclassic/issues/1211) | Fee ledger. Unrelated. | **Not implemented:** no `robots.txt` / `sitemap.xml` under `frontend-dapp/public/`. nginx has no `location = /robots.txt` / `/sitemap.xml`. `index.html` has no `robots` / `noindex` meta. Confirmed live HTML shell 2026-09-13. --- ## Current codebase Coolify → [`docker/frontend/Dockerfile`](docker/frontend/Dockerfile): Node `npm run build`, runtime `nginx:alpine`, `COPY nginx.conf` + `dist`. Vite copies `frontend-dapp/public/` to the dist root. nginx: ```nginx location / { try_files $uri $uri/ /index.html; } location = /og-image.png { try_files $uri =404; } location ~* \.(js|css|png|jpg|jpeg|gif|svg|ico|woff2?)$ { try_files $uri @hashed_asset_miss; } ``` `.txt` and `.xml` are **not** in the hashed-asset regex, so missing robots/sitemap hit SPA fallback. [`index.html`](frontend-dapp/index.html) is one OG/title shell for every route (**OG-5**: no prerender, no per-route titles). `frontend-dapp/public/` today: favicons, bootstrap, wallets, chainlist, `og-image.png`, sounds — **no** robots/sitemap. If files were added to `public/` only, `try_files $uri` would serve them **while present**. Default nginx mime: `.txt` → `text/plain` (good); `.xml` → `text/xml` (acceptance wants `application/xml`). If the files go missing later, SPA fallback returns HTML 200 again. Exact `location =` + `=404` is required. --- ## Why the new implementation is needed Crawlers that fetch `/robots.txt` and `/sitemap.xml` receive the DEX SPA HTML. Search Console can treat that as a soft-404 / HTML robots file. Adding files without exact-location nginx will regress the first time a deploy omits them. The DEX SPA must not become the SEO content host. This ticket only makes crawl endpoints honest and keeps wallet/manage/mint/unbounded pair routes noindex or omitted. Unique canonicals belong on a future static/prerender host (`docs.cl8y.com` is currently not an implementable surface). --- ## Constraints / guardrails - **Do not** noindex or `Disallow` a future content host (`docs.cl8y.com` / `cl8y.com` content). - **Do** keep DEX SPA routes noindex or omitted from the DEX sitemap (especially `/portfolio`, `/trader/:address`, `/token/:addr/manage`, `/mint`, unbounded pair URLs). - DEX remains one OG shell — do not invent unique titles/canonicals for SPA routes (**OG-5**). - No unverified fee/volume/“best DEX” claims in robots comments, sitemap extras, or meta. - Preserve og-image exact-location and hashed-asset 404 (`@hashed_asset_miss`). Do not widen SPA fallback. - nginx default `.xml` → `text/xml`; set `default_type application/xml` (or equivalent) for `/sitemap.xml`. - `Sitemap:` in robots may point at a content host only after that host has a real sitemap; until then use a minimal DEX sitemap or omit the Sitemap line. - **Out of scope:** marketing claims, listings, analytics, `FeeSource`, PostHog, `cl8y.com` robots (follow-on), prerender/react-helmet. --- ## Relevant files | Path | Why | | --- | --- | | `docker/frontend/nginx.conf` | SPA fallback; add exact locations | | `docker/frontend/Dockerfile` | Dist image | | `frontend-dapp/public/` | Add `robots.txt` + `sitemap.xml` | | `frontend-dapp/index.html` | Single crawler shell; optional `X-Robots-Tag` is nginx, not helmet | | `frontend-dapp/src/App.tsx` | Route inventory for omit/noindex | | `scripts/qa/verify-issue-578.sh` | nginx grep + optional container curl pattern | | `skills/AGENTS_FRONTEND_OPENGRAPH.md` | OG-5 / og-image exception to keep | | `docs/frontend.md` | Crawler shell documentation | --- ## Recommended direction 1. Add `frontend-dapp/public/robots.txt` (`User-agent` syntax, `text/plain`). Conservative: do not `Allow` unbounded `/trader/` or `/token/*/manage`. 2. Add `frontend-dapp/public/sitemap.xml` — minimal `urlset` of first-party DEX URLs that are intentionally crawlable (likely none or only `/` if the SPA should stay noindex). Omit wallet, manage, mint, and infinite pair/trader paths. Hosts in the urlset are `https://dex.cl8y.com` only. 3. Exact nginx locations (mirror og-image): ```nginx location = /robots.txt { default_type text/plain; add_header Cache-Control "public, max-age=3600"; try_files $uri =404; } location = /sitemap.xml { default_type application/xml; add_header Cache-Control "public, max-age=3600"; try_files $uri =404; } ``` 4. Optional: `X-Robots-Tag: noindex, nofollow` on `location = /index.html` so crawlers skip the transaction UI even if they ignore an empty sitemap. 5. Verify siblings (`/sitemap_index.xml`, `/robots.txt/` trailing slash) are 404 or real files — not SPA HTML 200. 6. `make verify-issue-1212`: grep exact locations; local nginx container curl type + body sniff; negative test with files renamed → 404. Reject: relying on `Disallow` as a security control; listing attacker-controlled hosts in the sitemap; helmet/prerender in this ticket. --- ## Acceptance criteria - [ ] **AC1.** `GET https://dex.cl8y.com/robots.txt` → `text/plain`, body is robots syntax (`User-agent`), not `<!doctype` / Vite SPA. - [ ] **AC2.** `GET https://dex.cl8y.com/sitemap.xml` → `application/xml` (or `+xml`), body is an XML `urlset` (empty valid urlset OK), not SPA HTML. - [ ] **AC3.** Missing robots/sitemap after a bad deploy → **404**, not HTML 200. - [ ] **AC4.** SPA-only routes are noindex and/or omitted from the sitemap (`/portfolio`, `/trader/:address`, `/token/:addr/manage`, `/mint`, unbounded pair URLs). - [ ] **AC5.** `/og-image.png` still `image/png`; hashed JS/CSS misses still 404 (#578 / #706). - [ ] **AC6.** No marketing claims in robots/sitemap. No third-party or attacker-controlled URLs in the urlset. Hosts are the DEX origin only. - [ ] **AC7.** Re-check after Coolify deploy with curl Content-Type **and** body sniff (not status code alone). Local verify script exists. --- ## Test plan (functional paths) | # | Path | Expect | | --- | --- | --- | | T1 | Present files (local image + live after deploy) | robots `text/plain` + `User-agent`; sitemap `application/xml` + `<urlset` | | T2 | Missing files (rename in local nginx) | Exact location **404**, not `index.html` | | T3 | SPA routes `/`, `/trade`, `/portfolio` | Still serve the app to browsers | | T4 | Siblings `/sitemap_index.xml`, `/robots.txt.bak` | Not SPA HTML 200 | | T5 | `/og-image.png` + hashed asset miss | #578 / #706 unchanged | | T6 | Sitemap contents | Only allowlisted first-party paths; empty urlset OK | | T7 | `make verify-issue-578` / `#706` | Still green | | T8 | Optional `X-Robots-Tag` on HTML | Present if implemented; browsers still get the app | Playwright not required. Vitest not required unless a static-file unit test is added. --- ## Test plan (attack, hack, and abuse) | # | Vector | Expect | | --- | --- | --- | | A1 | Sitemap poisoning (phishing / extra hosts) | CI or review allowlists origin; only `https://dex.cl8y.com` URLs | | A2 | Listing `/trader/:addr`, `/token/.../manage`, `/mint` | Omitted; robots does not `Allow` those wildcards | | A3 | Relying on `Disallow` to hide secrets | Forbidden as a security control; manage/mint stay behind wallet UX | | A4 | Overly broad `Allow:` of dynamic segments | Rejected; prefer omit + HTML noindex | | A5 | SPA fallback regression (deleted public files) | 404 from exact location | | A6 | Content-Type lie (HTML labeled robots) | Fail acceptance; sniff body | | A7 | `Sitemap:` pointing at a foreign host | Pin to known CL8Y origin or omit | | A8 | Comments/claims in robots that overstate product | No marketing claims | | A9 | Open redirect / `javascript:` in urlset | Forbidden; static first-party paths only | --- ## Verification criteria - `make verify-issue-1212` (grep nginx exact locations + local container curl type/body + negative 404). - `make verify-issue-578` and `make verify-issue-706` still green. - Diff of `nginx.conf` shows exact locations without changing hashed-asset 404 or og-image. - Post-deploy curl `-sI` and body checks on `https://dex.cl8y.com/robots.txt` and `/sitemap.xml` vs the 2026-09-13 HTML-shell baseline. - `docs/frontend.md` notes that crawl endpoints are real files; unique canonicals remain blocked on the missing content host.

ops-worker added the

agent:implement

ready

labels

2026-09-06 12:05:18 +00:00

ops-worker referenced this issue

2026-09-06 12:05:19 +00:00

docs(indexer): OpenAPI/path pack + curls for swaps, pools, fees, burns, windows #1204

PlasticDigits changed title from ~~fix: dex.cl8y.com robots.txt and sitemap.xml must not be the SPA shell~~ to fix(frontend): dex.cl8y.com robots.txt and sitemap.xml must not be the SPA shell

2026-09-13 05:37:51 +00:00

PlasticDigits added the

frontend

label

2026-09-13 05:38:05 +00:00

PlasticDigits commented

2026-09-13 05:38:39 +00:00

Owner

Repair: DEX-native DoR. Live 2026-09-13 still SPA HTML 200 on /robots.txt and /sitemap.xml. Robots+sitemap+exact-location 404 stay one ticket. Not a dupe of #578/#706.

PlasticDigits commented

2026-09-16 20:17:41 +00:00

Owner

Occupying-runner check: implement labels already present; no extra labels added.

PlasticDigits commented

2026-09-16 20:29:13 +00:00

Owner

/agent implement

PlasticDigits commented

2026-09-16 20:39:25 +00:00

Owner

/agent implement

PlasticDigits commented

2026-09-16 22:54:50 +00:00

Owner

/agent implement

PlasticDigits commented

2026-09-16 23:18:56 +00:00

Owner

/agent implement

PlasticDigits commented

2026-09-16 23:20:43 +00:00

Owner

/agent implement

PlasticDigits commented

2026-09-24 09:15:26 +00:00

Owner

Verification (2026-09-24): not closable.

Production /robots.txt, /sitemap.xml, /sitemap_index.xml, /robots.txt.bak, and /robots.txt/ return 200 text/html with the same SPA body/ETag. This fails AC1–3 and T4.
frontend-dapp/public/ has neither crawler file; nginx has no exact file-only locations; make verify-issue-1212 is absent.
The adjacent protections currently pass live spot checks: /og-image.png is image/png and missing hashed JS/CSS return 404; /trade still serves the SPA.

Remaining:

Add real robots/sitemap files and exact nginx locations with correct media types, file-only 404s, and non-SPA sibling behavior.
Add the local positive/body/missing-file verifier.
Deploy and recheck endpoint status, media type, and body.

No newer successor issue found via fj searches for robots, sitemap, or crawl. Preserve [#578] and [#706].

Verification (2026-09-24): not closable. - Production `/robots.txt`, `/sitemap.xml`, `/sitemap_index.xml`, `/robots.txt.bak`, and `/robots.txt/` return 200 `text/html` with the same SPA body/ETag. This fails AC1–3 and T4. - `frontend-dapp/public/` has neither crawler file; nginx has no exact file-only locations; `make verify-issue-1212` is absent. - The adjacent protections currently pass live spot checks: `/og-image.png` is `image/png` and missing hashed JS/CSS return 404; `/trade` still serves the SPA. Remaining: 1. Add real robots/sitemap files and exact nginx locations with correct media types, file-only 404s, and non-SPA sibling behavior. 2. Add the local positive/body/missing-file verifier. 3. Deploy and recheck endpoint status, media type, and body. No newer successor issue found via fj searches for robots, sitemap, or crawl. Preserve [#578] and [#706].

PlasticDigits referenced this issue

2026-09-24 09:47:27 +00:00

docs(frontend): document crawler endpoint invariants #1344

Rows
Columns

fix(frontend): dex.cl8y.com robots.txt and sitemap.xml must not be the SPA shell #1212

Summary

Given / When / Then

Not a duplicate / already implemented

Current codebase

Why the new implementation is needed

Constraints / guardrails

Relevant files

Recommended direction

Acceptance criteria

Test plan (functional paths)

Test plan (attack, hack, and abuse)

Verification criteria