Why do llms.txt links return HTML even though the pages load fine in a browser?

Because the URL you listed is an HTML route, not a text route. In the 56 targets we measured, 28 ended in .md and came back as Markdown, while 14 came back as text/html — Stripe's payment pages, Render's own homepage, MongoDB's company page and four external links inside ai-sdk.dev's file. A browser hides the difference because it renders both. A fetcher that wants tokens pays for navigation, consent scripts and hydration markup. The fix is to list the Markdown twin when one exists, not to hope the agent strips the page.

Does a 200 status mean an llms.txt link works for AI agents?

No. Status only proves the route exists. Of the 54 targets that answered 200 in this audit, 14 served HTML and 3 served JSON, so 17 of them needed markup parsing rather than plain reading. A 200 can also be a soft failure: bun.com/docs/index.md returned 404 with a page whose meta robots is noindex and whose canonical is /404, and Stripe's four 200 responses all carried a canonical pointing at a /jp/ variant of the same URL. Check status, content type, canonical and byte size together.

Should llms.txt link to llms-full.txt or to individual pages?

Both, with different jobs. An index links to pages an agent can read one at a time; llms-full.txt is the bulk dump for agents that want the whole corpus. 14 of the 56 targets in this audit were themselves llms files — Cloudflare's four subpath files, Svelte's llms-small, llms-medium and llms-full, plus nuxt.com's 5.17 MB and supabase.com's 6.6 MB full dumps. That is legitimate delegation, but a full dump that large will be truncated by most context windows, so keep the per-page links alongside it.

Research · November 2026

llms.txt Second Hop

Every audit of an llms.txt file stops at the file. Validators check the H1, the blockquote and the section structure; link checkers confirm that each URL answers 200. Nobody follows the link and asks what came back. We did that for 56 URLs taken from 14 live files on November 2, 2026 — and 30% of the second hops returned something other than readable text.

Why the Second Hop Gets Skipped

An llms.txt file is a promise that the URLs inside it are worth fetching. A status-code check tests whether the promise can be met at all, not whether it is cheap to keep. The gap between those two questions is where the interesting failures live: pages that answer 200 with 700 KB of HTML, endpoints that need query parameters to work, index files that point at a Markdown twin that was deleted, and 200 responses whose canonical tag disagrees with the URL you published.

Our earlier pass over 2,768 links in 17 files covered the form of the links — absolute versus root-relative, dead entries, duplicates — in llms.txt link hygiene. This is the payload question: what does an agent actually receive when it follows the first four links in document order?

The Audit: 16 Hostnames, 14 Files, 56 Targets

Method: one request per host for /llms.txt with a desktop user agent, redirects followed, body parsed for Markdown links. Sixteen hostnames were tried; fourteen served a real text or Markdown file. astro.build/llms.txt and docs.astro.build/llms.txt both returned 404. From each of the fourteen live files we took the first four links, then fetched each one and recorded status, content type, byte count, canonical tag and meta robots.

What Comes Back

Content type	Targets	Share	Average bytes
text/markdown	23	41%	14,579
text/plain	16	29%	936,157
text/html	14	25%	646,819
application/json	3	5%	1,808

Status was healthy: 54 of 56 answered 200, one answered 400 and one answered 404. Content type was the dividing line, not availability. The text/plain average is inflated by four bulk dumps that happen to sit in the first four entries — nuxt.com's 5.17 MB and supabase.com's 6.6 MB llms-full.txt among them — which is why the per-type average is not a per-page cost.

The Real Cost Is 48 Times

Strip the bulk dumps out and compare the two kinds of reading target. The 23 Markdown twins averaged 14,579 bytes, roughly 3,600 tokens. The 13 HTML targets that answered 200 averaged 694,393 bytes, roughly 174,000 tokens — 47.6 times the Markdown twin for the same topic. Stripe's payment-methods page alone was 2,938,578 bytes; MongoDB's company page was 998,514.

No context window will hold that, so the practical outcome is truncation, not failure. The agent gets the first chunk of a page whose first chunk is a navigation bar. This is the same trade-off described in Markdown for Agents versus llms.txt: an index can only promise cheap reads if the targets are cheap.

Four Failures Worth Checking For

A listed file that no longer exists. bun.com/docs/index.md returned 404 with a 28,371-byte HTML body whose meta robots is noindex and whose canonical is https://bun.com/404. The file still ships the link.
An API template listed as a page. docs.github.com/llms.txt lists /api/pagelist/versions, /api/pagelist/languages and /api/pagelist/en/free-pro-team@latest — all valid JSON endpoints — then /api/article, which returned 400 because it needs parameters the index never supplies.
The same document twice. vite.dev lists /guide/features.md twice, the second time with a #hot-module-replacement fragment. Both requests returned the identical 39,544 bytes, so half of that file's first four links were one document.
A canonical that disagrees with the URL. All four Stripe targets answered 200 and all four carried a canonical pointing at a /jp/ variant of the same path — what Stripe's edge served the location our requests egressed from, not necessarily what a US visitor sees. If your pages are geo-personalised, the URL in the file is a suggestion.

A Third of the Targets Are Other Indexes

Fourteen of the 56 targets were themselves llms files rather than pages: four Cloudflare subpath files (/cache/llms.txt, /argo-smart-routing/llms.txt and two more), four Svelte variants (llms-small, llms-medium, llms-full and /docs/svelte/llms.txt), the two full dumps at nuxt.com and supabase.com, and mongodb.com/docs/llms.txt. Delegation like this is the correct pattern for large docs sets — see llms.txt versus llms-full.txt — but it means an agent's second hop can be a third hop, and the sizes compound.

The Three-Command Second-Hop Check

Before shipping, resolve the links the way a fetcher would. This prints status, content type and bytes for the first twenty targets:

grep -oE 'https://[^)]+' llms.txt | head -20 | while read u; do
  curl -sSL -o /dev/null --max-time 15 \
    -w "%{http_code} %{content_type} %{size_download} $u\n" "$u"
done

Read the output with four rules:

Anything that is text/html should be a deliberate choice, not an accident.
Anything outside 200–399 is a fail; a 404 is the loudest thing a file can contain.
Duplicate bodies mean duplicate entries — deduplicate by URL path, ignoring fragments and tracking query strings.
If a canonical tag points somewhere else, list the canonical URL instead.

What to Fix This Week

Run the loop above against your own file. For every HTML target, check whether a Markdown twin exists — 28 of the 56 links we measured already ended in .md, which is what a healthy file looks like. Replace dead entries, collapse duplicates, and keep bulk dumps in their own section with the size stated in the description, so an agent can decide before it downloads 6 MB. Then re-run the validator and, if you are rebuilding the file from a sitemap, the generator with your curated page list rather than the raw crawl.

The file is the map. The second hop is the road, and 30% of the roads in this sample are not the surface the map promised.