Research · October 2026

Do AI Companies Publish llms.txt?

Every company on this list sells the premise that machines should be able to read the web. We fetched /llms.txt from 44 of their hostnames on October 29, 2026 to see whether they take their own advice: 19 serve a real plain-text or Markdown file, 13 return 404, five block the request at the edge, and three answer 200 with HTML instead of a file.

45 Requests, Eight Different Outcomes

One request per hostname with a desktop user agent and Accept: text/plain, text/markdown, */*, redirects followed, no retries. A response counted as a file only when the status was 200, the content type was text/plain or text/markdown, and the body opened with an H1.

Outcome Hosts Example
200, real file20platform.claude.com, 90,932 B
of those, a duplicate document1docs.claude.com → platform.claude.com
200, HTML not a file3v0.dev, 1,060,043 B
403 at the edge5openai.com, 9,949 B challenge
40413anthropic.com, 106,777 B page
302 to a login1ai.google.dev → accounts.google.com
308 to itself1voyageai.com
400 / 4012llama.com, meta.ai

Nineteen distinct documents came back, because docs.claude.com redirects to platform.claude.com and serves the identical 90,932 bytes. One host, voyageai.com, answers 308 with a Location pointing at its own URL and a refresh: 0;url= header set to the same address — a loop rather than a redirect.

The Model Vendors Are the Ones Without a File

Sort the list by what the company sells and the pattern is uncomfortable. The labs and assistants that decide what gets crawled are the hosts that fail hardest.

Vendor Host Status Body
OpenAIopenai.com4039,949 B HTML challenge
OpenAIplatform.openai.com4044,083 B app shell
Anthropicanthropic.com404106,777 B error page
Anthropicclaude.com404107,313 B error page
Perplexityperplexity.ai4035,655 B "Just a moment..."
xAIx.ai40392 B block notice
Googleai.google.dev302OAuth login URL
Metameta.ai40135 B JSON
Microsoftcopilot.microsoft.com4034,557 B HTML
DeepSeekdeepseek.com404115,730 B HTML

The two exceptions inside those companies are a documentation host and a research site: platform.claude.com serves the largest file in the set, and deepmind.google serves a 6,928-byte map in four sections. Nothing here proves hostility to the convention — a 403 from bot management and a 404 from a marketing CMS look identical from outside, and one is a stored challenge page rather than a policy. The result for an agent is the same either way: no map at the address the convention specifies. Google's own position on the file is already on the record; for the crawler side, see which AI engines read llms.txt.

Where the File Lives Is the Docs Host, Not the Homepage

The split is almost mechanical. Infrastructure, vector-database, inference and agent companies publish at their root or their docs host and mostly pass both tests: fifteen live files, including Pinecone (31,759 bytes), Weaviate, Qdrant, Modal, Baseten, Replit, Cursor, Lovable, ElevenLabs, Together, Databricks, Mistral, Cohere, OpenRouter and Ollama.

Three shapes are worth copying. Anthropic keeps the developer document at the docs host and lets docs.claude.com redirect there, so both hostnames work. Runway returns a real 8,005-byte file after redirecting runwayml.com to runway.com — a redirect that lands on a file is fine; the failure mode is a redirect that lands on an app shell. LangChain shows the gap in miniature: langchain.com is a 404 behind 62,698 bytes of Webflow HTML, while docs.langchain.com serves a 3,129-byte index whose first instruction is to follow each /_llms/ index recursively until you reach a documentation page.

Three 200s That Are Not Files

The worst outcome in the table is not the 404. v0.dev/llms.txt redirects to v0.app/llms.txt and returns 1,060,043 bytes of text/html; anyscale.com returns 67,705 bytes of HTML; character.ai returns 14,512 bytes of a Cloudflare Waiting Room page. All three say success. An agent that trusts the status code spends its whole context window on markup and learns nothing about the site, and a human auditing with curl -I sees 200 OK and moves on. A 404 is honest; a 200 with HTML is a trap — the delivery failure modes are catalogued in llms.txt not working.

What the 19 Live Files Do Right

Host Bytes H2 URLs .md Optional
platform.claude.com90,9325827821no
pinecone.io31,759132184no
weaviate.io29,789136530no
modal.com28,2133347344no
cursor.com27,65131456453no
qdrant.tech26,91579881yes
baseten.co24,7347966yes
openrouter.ai9,69785714no
cohere.com4,80510610yes
ollama.com1,7864110yes

Across the 19 files: 361,581 bytes, 2,846 URLs, a median of 62 URLs per file, 2,251 Markdown link expressions, and 1,498 of those pointing at a .md twin — two-thirds of all links. Markdown twins are the house style at the top end, not an edge case: 821 of platform.claude.com's 827 URLs end in .md, 453 of Cursor's 456, 344 of Modal's 347, 228 of Replit's 229.

Link syntax is where discipline drops. 595 URLs — one in five — are written as bare addresses with no brackets, so a Markdown parser sees prose rather than a link. Cursor's whole file is bare (456 URLs, zero bracket links), Cohere's is prose with 61 addresses inline, and Weaviate mixes the two in the same document. Anthropic's file is the opposite extreme: 822 bracket links and one bare URL, a console link it marks as requiring login. For why the difference matters at parse time, see link hygiene.

Structure is more consistent than syntax. 17 of 19 files open with the blockquote summary line the spec asks for, and 17 have exactly one H1 — the exceptions are Cursor, which nests four H1s through a bullet hierarchy, and Weaviate, which has 20 lines starting with a single # because shell and code comments inside the document are indistinguishable from headings to a parser. Only six files use the reserved Optional section, and 10 of 19 mention their own llms-full.txt, which is the right split: a small index plus a link to the bundle, as described in llms.txt vs llms-full.txt.

Three Habits Worth Copying

A Two-Minute Check on Your Own File

Six commands separate a file from a 200 that only looks like one. Run them against your own domain before comparing yourself to anything in the tables above. 

# 1. status, content type and size in one line
curl -sS -o /dev/null -w "%{http_code} %{content_type} %{size_download}\n" https://example.com/llms.txt

# 2. does the body actually open with an H1?
curl -s https://example.com/llms.txt | head -c 120

# 3. section count, URL count, bare-URL count
curl -s https://example.com/llms.txt | grep -c '^## '
curl -s https://example.com/llms.txt | grep -oE 'https?://[^ )]+' | wc -l
curl -s https://example.com/llms.txt | grep -oE '\]\(http[^)]+\)' | wc -l

# 4. is Optional last, and does the file mention llms-full.txt?
curl -s https://example.com/llms.txt | grep '^## ' | tail -1
curl -s https://example.com/llms.txt | grep -c 'llms-full'

If step 2 returns HTML, stop — the rest of the numbers are meaningless. If step 3's two counts disagree widely, you have the Cursor or Cohere shape and a parser may be reading your links as text. To build the file from a sitemap rather than by hand, the llms.txt generator produces the H1, summary and sections; the checker reports the structure and link counts it parsed so you can compare them with the numbers above. For the upper bound on size, see how long an llms.txt should be.

The useful lesson in the audit is not that AI companies are hypocrites. It is that the companies who ship the file well are the ones whose product is documentation, and they ship it where the documentation lives, in Markdown, with a directive at the top. That is a cheaper pattern to copy than any of the marketing claims about AI visibility.