Why does my llms.txt return 200 but still get ignored?

A 200 is only half the answer. Check the Content-Type on that response: if it says text/html, a catch-all rewrite handed the request to your app shell and no agent ever saw markdown. Then check the body — fetch the URL and confirm the first lines are a markdown H1 and a summary line, not an HTML document. Finally, confirm that no robots.txt rule or firewall rule blocks the request in the first place.

What Content-Type should llms.txt be served with?

text/plain with charset=utf-8 is the safe default, and text/markdown is also seen on real files — Stripe serves its large file as text/markdown. What you must never see is text/html, application/octet-stream or text/x-robots. Those three mean a rewrite rule, not a file, answered the request: the first is a single-page app shell, the second is a misconfigured static host, and the third is a rule that mapped /llms.txt onto /robots.txt.

Do AI crawlers follow a redirect to llms.txt?

Some do, some do not, and a chain across hosts adds nothing but failure surface. Serve the file directly on the exact hostname you advertise — if www 301s to the bare domain, publish the file on the bare domain instead of relying on the hop. The one thing a redirect can never fix is a chain that ends at your homepage: that is a missing file with extra steps.

Troubleshooting · September 2026

llms.txt Not Working? Seven Delivery-Level Fixes

The file is in the repository, it passed validation, and no AI engine ever asks for it. Usually the problem is not the format but delivery: what your server actually returns when something requests /llms.txt.

The 30-Second Test

Ask the server, not your editor. Two commands answer the status code, the content type, whether a redirect sits in front of the file, and whether the body is markdown or an HTML shell:

# headers only: status, Content-Type, Location, cache status
curl -sI https://example.com/llms.txt

# fetch the way an AI crawler would, and look at the first lines
curl -sA "GPTBot/1.2 (+https://openai.com/gptbot)" https://example.com/llms.txt | head -20

A healthy response is 200, content-type: text/plain; charset=utf-8 (or text/markdown), no Location header, and a body starting with a markdown H1 and the summary line. Anything else maps onto the table below.

Symptom	What it usually means
200, but `text/html`	Catch-all rewrite served your app shell — fix 1
`text/x-robots` or the wrong body	A rule mapped `/llms.txt` onto `/robots.txt` — fix 1
301 or 302 chain, or a redirect to the homepage	Canonical host mismatch — fixes 2 and 3
404 for that path only	Path case, extension not whitelisted, or wrong subpath — fix 3
403, 429 or an empty reply	WAF, bot rule or a `robots.txt` block — fix 4
200 locally, 404 in production	The file never shipped in the build — fix 5
Old content, or a 404 that comes and goes	Edge cache holding a previous response — fixes 6 and 7

Fix 1: The Response Is HTML, Not Markdown

This failure is invisible in a browser, because the browser renders the app shell anyway. A single-page app or a rewrite-everything rule answers every unknown path with index.html — including /llms.txt. A Common Crawl survey of published llms.txt files found the two signatures: sites that serve their SPA for any unknown path answer text/html, and sites whose rewrite sends /llms.txt to /robots.txt answer text/x-robots.

A static file in public/ (or static/, or your host's root directory) is served as text/plain automatically — add an exception so the SPA fallback never sees that path. If a route handler generates the file, set the header yourself. The v2 spec, modified August 10, 2026, names no required MIME type, but it does require a markdown file at that path, and a response labelled text/html is not one.

Fixes 2 and 3: Redirect Chains and Path Case

Redirects turn a working file into an unreliable one. Some agents follow them, some do not, and a chain across hosts doubles the failure surface. Publish the file on the exact hostname you advertise. A redirect whose destination is your homepage is worse than a 404 — a missing file that looks like a success at the HTTP level.

Path case is the quieter version of the same problem: object storage and Linux hosts are case-sensitive, so /LLMS.TXT and /Llms.txt are 404s even though /llms.txt exists, and NGINX rules that whitelist extensions can 404 a newly added .txt file. Subpath files follow the spec's scope rule — a file describes the URLs under its path, most specific wins — so /docs/llms.txt must be reachable at that path, not only at the root.

Fix 4: Something Is Blocking the Request

Check the two layers in front of your origin. First, robots.txt: the major AI crawlers honour it, so a broad Disallow: / or a pattern such as Disallow: /*.txt blocks the llms.txt fetch itself — the file never gets read. Second, the firewall: bot protection, WAF rules and aggressive rate limiting return 403 or 429 to clients that do not look like browsers. Test with a crawler user agent and compare:

curl -s -o /dev/null -w '%{http_code}\n' -A "GPTBot/1.2" https://example.com/llms.txt
curl -s -o /dev/null -w '%{http_code}\n' -A "ClaudeBot/1.0" https://example.com/llms.txt

A 403 for the crawler while your browser sees 200 puts the block in the CDN or WAF, not your site. Staging has the same trap in another form: basic authentication on the whole hostname fails every fetch, which is harmless on staging and fatal if that rule is ever applied to production.

Fix 5: The File Never Reached Production

Local success is not deployment success. The file may sit in a source folder the build does not copy into the output directory, the deploy may have run from a commit that predates it, or a dynamic route may catch /llms.txt and return something else. The only reliable check is against the production URL after every deploy — and if you deploy manually, confirm the file made it into that build, not the previous one.

Fixes 6 and 7: Caches, Copies and the Verification Pass

An edge cache can keep serving a 404 your origin stopped returning, or a copy from before your last edit. Purge the exact path, and read the cache headers while you are there: an age measured in days on a file you changed an hour ago tells you which copy agents get. Watch for a second hostname on the same site, too — a staging or preview alias that answers /llms.txt is a file agents can find and you will forget.

Then run the pass once, and again after every deploy that touches the file:

Canonical host, curl -sI: 200, text/plain or text/markdown, no Location.
Crawler user agent: same status as your browser, not a 403.
Body: a markdown H1 and summary line, not an HTML document.
Every link inside the file returns 200 — dead links waste the curation even when delivery is perfect.
No robots.txt or firewall rule matches the path.

If the content is the suspect rather than delivery, the six mistakes that get a file skipped cover that side, the crawler reference shows which tokens matter per engine, and the measurement walkthrough answers whether anyone reads the file at all. Run the URL through the llms.txt checker the way an agent would, and build the file itself with the generator if it is still a draft.