Does a site serve a different llms.txt to GPTBot than to a browser?

In our sample, no. We fetched /llms.txt from 20 hosts six times each with the user agents of GPTBot, ClaudeBot, PerplexityBot, Googlebot, CCBot and desktop Chrome on October 20, 2026. Every one of the 18 hosts that answered 200 returned the same status, the same byte count and the same content type to all six agents. Sending the GPTBot string buys you nothing that a plain curl does not, and it also means you can test your own file with a normal request.

Why does my llms.txt return 403 to an AI crawler?

Usually because the path sits behind bot management, not because of a user-agent rule. openai.com/llms.txt answered 403 with a cf-mitigated: challenge header to all six agents, including Googlebot and desktop Chrome, so the block is applied to the path itself. Spoofing the GPTBot user agent does not help, because a WAF verifies the caller by IP range and TLS fingerprint as well as the header. The fix is a rule at the edge — a WAF skip or bypass for the exact path — not a change to the file.

Is a 404 the same as a missing llms.txt?

Effectively yes, and the body tells you which kind of 404 it is. huggingface.co/llms.txt returned 404 with 52,349 bytes of HTML app shell, which means the host routes the request and simply has no file at that path. A 200 that returns text/html is a different failure: the page exists, the file does not, and a server rewrite is sending the site's fallback document to anything that asks for /llms.txt.

Research · October 2026

Does your llms.txt reach AI crawlers?

The standard worry is that a CDN quietly blocks AI agents from your llms.txt while serving the file perfectly to you. So we tested it directly: the same path, six different user agents, 20 hosts, on October 20, 2026. The result was less dramatic than the worry. Eighteen of twenty hosts ignored the user agent completely, and the two that failed failed for everybody — Googlebot and desktop Chrome included.

The test: one path, six agents, 20 hosts

Every request went to https://HOST/llms.txt with redirects followed, a 20-second timeout, and one of six user-agent strings: GPTBot 1.2, ClaudeBot 1.0, PerplexityBot 1.0, Googlebot 2.1, CCBot 2.0, and a desktop Chrome 129 string as the control. Each response was recorded for status code, downloaded bytes and content type. The hosts were documentation and product sites known to publish a file — Stripe, Anthropic, Nuxt, Bun, GitHub, Cloudflare, shadcn, Hono, Prisma, Resend, Vite, Supabase, Mintlify, Vercel, Svelte, llmstxt.org, Checkly, Hugging Face and OpenAI.

18 of 20 hosts ignored the user agent entirely

Not one host returned a different status, a different length or a different content type to GPTBot than to desktop Chrome. The table below is therefore one column of bytes, not six — every row was identical across all six agents.

Host	Bytes (all six agents)	Content-Type
docs.stripe.com	97,564	text/markdown
platform.claude.com	86,059	text/plain
nuxt.com	57,285	text/plain
bun.com	33,834	text/plain
docs.github.com	28,642	text/markdown
www.prisma.io	17,353	text/plain
developers.cloudflare.com	16,787	text/plain
www.checklyhq.com	15,383	text/plain
ui.shadcn.com	12,260	text/plain
resend.com	8,340	text/plain
www.mintlify.com	6,556	text/plain
hono.dev	5,927	text/plain
vercel.com	5,240	text/plain
vite.dev	3,528	text/plain
supabase.com	3,242	text/plain
svelte.dev	1,676	text/plain
llmstxt.org	637	text/plain
huggingface.co	404 · 52,349	text/html
www.openai.com	403 · 9,735–9,928	text/html

Two details in that column are worth copying. First, docs.anthropic.com is a pure alias: the request redirects to platform.claude.com/llms.txt and returns the same 86,059 bytes, so the docs host and the product host publish one file, not two. Second, only two hosts label the file as Markdown — Stripe and GitHub both send text/markdown; charset=utf-8 — while the other sixteen send text/plain. Both are fine; the content-type question only starts to matter when the type says HTML.

The two failures are path-level, not agent-level

openai.com/llms.txt answered 403 to GPTBot, ClaudeBot, PerplexityBot, Googlebot, CCBot and Chrome alike. The headers name the mechanism: cf-mitigated: challenge and server: cloudflare, with a text/html body between 9,735 and 9,928 bytes that changed size on each request. This is bot management applied to the path, and it does not consult the user-agent string you send — Googlebot's own header was challenged too, because an unverified caller from an arbitrary IP is not Googlebot no matter what it claims.

huggingface.co/llms.txt answered 404 with 52,349 bytes of HTML app shell. That is a different signal: the host routes the path and has no file behind it. A 404 with an HTML body is the normal shape of "this site never published the file", and it is indistinguishable, from the agent's side, from a deliberate omission.

robots.txt is where the permissions actually live

We fetched robots.txt from the same 20 hosts. Eighteen returned 200 and two returned 404 — hono.dev and vite.dev publish no robots.txt at all. Not one of the eighteen disallows /llms.txt for any agent, and the two most explicit files show what deliberate permission looks like: nuxt.com names 26 AI agent tokens individually, each with its own Allow: / block, and supabase.com names four plus a Content-Signal: ai-train=yes, search=yes, ai-input=yes line. If a site wanted agents kept off its file, that is where it would say so — which is why a robots.txt audit reads as an allow-list rather than a block-list in this sample.

HEAD lies, and one host proves it

A quick curl -I against docs.stripe.com/llms.txt reports content-length: 0. A GET to the same URL returns 97,564 bytes of Markdown. The response also carries vary: Accept, Accept-Language, so the body is negotiated and a header-only probe cannot tell you what an agent will receive. Check with GET and read the byte count, or skip the shell entirely and run the file through the llms.txt checker, which fetches the same way an agent does.

Run the same test on your own site

Four commands answer the whole question. The first is the control, the second pretends to be GPTBot, the third reports body size and type without a HEAD shortcut, and the fourth shows what the server sees as the final URL after redirects.

# 1. the control: what a browser gets
curl -sSL -o /dev/null -w "%{http_code} %{size_download} %{content_type}\n" \
  https://example.com/llms.txt

# 2. the same path as GPTBot
curl -sSL -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  -o /dev/null -w "%{http_code} %{size_download} %{content_type}\n" \
  https://example.com/llms.txt

# 3. the response headers, including anything the edge adds
curl -sS -D - -o /dev/null https://example.com/llms.txt | grep -iE \
  "^(http|content-type|content-length|cf-mitigated|server|x-robots)"

# 4. does the first line look like a file or an HTML page?
curl -sS https://example.com/llms.txt | head -3

Read the pairs together. If step 1 and step 2 agree and both return text, the file is reachable and no agent-specific work is left to do. If both return 403 with an edge header, you need a rule at the CDN — a WAF skip or bypass for that exact path — not a different file. If both return 200 with text/html, the route is falling through to your site's catch-all and the file needs to be written and served; our delivery troubleshooting guide covers those rewrites. And once the status is clean, keep watching the requests themselves — that is a different measurement, described in measuring llms.txt traffic.

The useful conclusion is smaller than the fear: in this sample, nobody was quietly serving a different llms.txt to AI agents, so the GPTBot user agent is not a magic key and a plain curl is a representative test. What breaks the file is the infrastructure in front of it, and that breaks it for everyone at once.