Research · October 2026

llms.txt Adoption in 2026: What the Top 300 Websites Actually Serve

Every week someone publishes a number for how many sites have an llms.txt. Most of those numbers come from a directory or a tool vendor's own crawler. We went to the ranking itself instead: the Tranco top 300, fetched one path deep, on October 18, 2026. Twenty-two serve a real file. Forty-one answer 200 with HTML behind the path. Ninety-four return a plain 404, and thirty-four refuse the request outright.

How the sample was built

The list is the Tranco top 300, downloaded the same morning. For each hostname we made one GET https://<domain>/llms.txt with redirect following, a 20-second timeout and a research user agent, then classified the result by status code and by body: a file counts only when the response is 2xx, the declared type is text/plain or text/markdown, and the body contains no HTML document at all.

One caveat matters more than any other: 97 of the 300 hostnames never answer. Tranco ranks DNS and CDN infrastructure alongside websites, so the list includes things like akamai.net, cloudfront.net and gstatic.com that have no site to put a file on. The honest denominator is the 203 hosts that responded.

The outcome table

Response Count Share of the 203 that answered
200 with a real text or Markdown file 22 10.8%
200 with HTML behind the path 41 20.2%
404 or 410 94 46.3%
Refused (403 ×31, 401 ×1, 429 ×2) 34 16.7%
Other odd codes: 202 empty ×2, 417 ×2, 400 ×2, 406 ×1, 418 ×1, JSON error ×1, redirect-only ×2, script shell ×1 11 5.4%
No HTTP response at all 97 —

Grouped differently: 31% of the hosts that answered either serve a file or pretend to. Nearly one in five returns an HTML page under an llms.txt URL, which is the failure mode most likely to mislead a client that checks status codes only. Our checker flags it as a soft 200 for exactly this reason.

The 22 files that are actually there

The one clear pattern in the sample is who publishes. The adopters cluster in payments, developer infrastructure, identity and enterprise software. Media, search, social and consumer hardware are almost entirely absent from the list below — the same split our news publisher study found at larger scale.

Domain Bytes H2 Links
sourceforge.net3,329,99743,606
salesforce.com1,021,70303,772
unity3d.com → unity.com937,665102,427
taboola.com278,284131,825
sentry.io245,108291,113
dropbox.com209,33017829
stripe.com69,92331308
paypal.com40,1037151
hubspot.com35,56429225
github.com28,65812117
gravatar.com21,9271526
cloudflare.com17,1731390
fastly.net → fastly.com16,6181587
roku.com15,683970
amplitude.com14,777554
hcaptcha.com13,8651362
shopify.com12,5201274
digicert.com8,3952154
linktr.ee7,913750
wordpress.com7,355826
intuit.com5,786847
samsung.com4,597531

Totals for the group: 6,342,944 bytes, 15,044 links, a median of 21,927 bytes, five files under 10 KB and six over 100 KB. Fourteen of the 22 are reached through a redirect, usually apex to www; salesforce.com appends a marketing parameter to the target, and unity3d.com now redirects to unity.com, which is what a brand rename does to a root-level file.

What the 41 soft 200s tell you

This is the bucket worth studying, because it is invisible to anyone who trusts the status code. netflix.com/llms.txt redirects to /NotFound?prev=... and returns 200. tiktok.com/llms.txt lands on /404?fromUrl=/llms.txt and returns 200. spotify.com pushes the path to a locale prefix and serves HTML. youtu.be parses llms.txt as a video ID and serves a watch page for ?v=llms.txt. Two hosts, goo.gl and forms.gle, answer 400; theguardian.com answers 406; ozon.ru redirects to itself with a query parameter.

If you have ever run a crawler over a customer's site and reported "llms.txt found", check the body. A file is Markdown: a first line beginning with #, then a blockquote summary. If the response opens with <!doctype html, you found a 404 page wearing a 200.

Gaps inside the files that do exist

Delivery is the quieter half of the problem: those charset and caching details are the ones that break at the edge, and we covered them in content-type and caching. On file size, the three largest entries here go straight past every published budget — see llms-full.txt size and structure for what belongs in the full file instead.

Who publishes, and who does not

Among the 94 plain 404s are google.com, apple.com, amazon.com, microsoft.com, bbc.com, nytimes.com, cnn.com, forbes.com, ebay.com and ibm.com. Among the 34 refusals are wikipedia.org, reddit.com, medium.com, openai.com, chatgpt.com, reuters.com and mit.edu. Both behaviours are consistent with the crawl data we summarised in which AI engines read llms.txt: the file is an opt-in affordance for sites that want to be quoted precisely, not a hygiene requirement.

If your site is in the 404 group

The distance between a 404 and a working file is usually one afternoon, and it is not the writing that takes the time — it is picking the ten to thirty URLs worth listing. Paste your sitemap into the generator, delete what a crawler cannot judge, then prove the delivery with a header read rather than a body read: 

curl -sSI https://example.com/llms.txt | head -20
curl -sS -o /dev/null -D - https://example.com/llms.txt
# expect: 200, content-type text/plain or text/markdown,
# and a body whose first line starts with a single hash

Then take the last step most people skip: confirm an AI crawler can reach the path at all. A 403 to a datacenter IP is the single most common reason a published file is never read, and it looks exactly like success from inside a browser.