Is there a hard size limit for llms.txt?

The proposal at llmstxt.org sets no byte limit. Its only size rule is functional: the file itself should stay small enough to fit in a model's context, with the detail living behind the links and being fetched only when needed. Numbers you see quoted elsewhere come from third parties, not the spec. A third-party AI-discovery-file specification recommends staying under 50KB, some platform guidance suggests under 100KB, and most published agency advice puts a healthy curated llms.txt under five thousand tokens. Treat all of them as budgets rather than rules enforced by anything.

How many tokens is my llms.txt file?

Estimate it in one command. Fetch the live file and count bytes with curl -s https://example.com/llms.txt | wc -c. English markdown prose averages roughly four to five characters per token, so 8KB of text is about 1,600 to 2,000 tokens. If you need an exact figure for a specific model, run the file through that model's tokenizer locally. The estimate is precise enough for the decision that matters: whether the file is a two thousand token index or a hundred thousand token dump.

Should I split my llms.txt or use llms-full.txt instead?

Split when the site is the problem and switch formats when the content is the problem. If one file has grown to include a whole documentation tree alongside marketing pages, put the dense part at a subpath such as /docs/llms.txt and link to it from the root file; the spec says a client should use the most specific file that covers the URL it is working in. If instead you genuinely want full page text ingested rather than an index, that is llms-full.txt, and it is a separate file with its own very different size budget.

GEO · September 2026

How Long Should an llms.txt Be?

Every guide quotes a different ceiling — 50KB, 100KB, five thousand tokens — and none of them comes from the specification. Here is the rule the spec actually gives, where the practical ceiling really comes from, and the cut list for a file that grew past it.

The Spec Gives You a Rule, Not a Number

The proposal at llmstxt.org is deliberately quiet about bytes. Its only size statement is functional: the file stays small enough to fit in context, the detail lives behind the links, and is fetched only when a client needs it. That is a design constraint dressed up as a sentence, and it is more useful than a number, because the number would be wrong within a year. The v2 revision kept that framing.

The figures people repeat come from three other places. A third-party AI discovery file specification recommends no more than 50KB. Platform tutorials aimed at marketers say keep it under 100KB and UTF-8 encoded. Agency guidance usually lands nearer five thousand tokens, with the observation that most marketing sites sit comfortably under that. All three are budgets rather than rules, and the useful move is to know which one your consumer enforces — an indexing tool, a context window, or nothing at all.

Where the Real Ceiling Comes From

The binding constraint is whatever parses the file. Two public examples bracket it. Mintlify cites Turborepo's llms.txt at roughly 116,000 tokens as the cautionary tale — a file so large it defeats the purpose of being cheap to read. On the tooling side, developers in the Cursor community report that files above roughly 50,000 to 60,000 tokens get unstable during indexing, while a 4MB file (somewhere between 800,000 and 1.2M tokens at about five characters per token) worked in one test and is well outside anything else quoted.

For a normal site the practical answer is much lower than any of that. An index whose links carry one-line descriptions is a few kilobytes. Once your file is bigger than your homepage, someone has started concatenating content instead of curating it.

File size	Roughly	Verdict
Under 5 KB	800 – 1,500 tokens	Healthy curated index. Typical for a marketing site or blog
5 – 20 KB	1,500 – 5,000 tokens	Fine, but audit the descriptions first — they are the usual growth
20 – 50 KB	5,000 – 12,000 tokens	Over budget. Split the dense area into a subpath file
50 – 100 KB	12,000 – 25,000 tokens	Above every published ceiling. Expect truncation or a skipped read
Over 100 KB	25,000+ tokens	Stop calling it an index. This is llms-full.txt territory

Measure It in Two Commands

Never estimate size from your editor — CDN compression, encoding and duplicated links all change the answer. Fetch what a crawler fetches:

# bytes and word count of the live file
curl -s https://example.com/llms.txt | wc -c
curl -s https://example.com/llms.txt | wc -w

# structure, broken links and missing elements
curl -sI https://example.com/llms.txt | grep -i "content-type"

An index-style file should come back in the low kilobytes. If the byte count is in the tens of thousands, compare it against the table above and then run the same URL through the llms.txt checker to confirm the structure survived whatever generated it.

The Five Things That Bloat an llms.txt

Size problems are nearly always one of five decisions made once and never revisited:

Enumerating every post. List the pillar pages, the category hubs and the feed, not four years of articles. The index points; it does not archive.
Pagination and tag archives. /page/2/ through /page/40/ and every tag landing page add lines that resolve to the same content with no new value.
Descriptions that became paragraphs. One line, eight to twelve words. The annotation rules matter more than usual on a large file, because every extra clause is multiplied by every link.
Near-duplicate pages. Thirty city pages with the city name swapped is fifty kilobytes of noise. List the hub and the top markets.
Concatenated content. If full page text is in the index, it belongs in llms-full.txt instead.

Split Before You Delete

The spec supports subpath files, and the rule is that a client uses the most specific file covering the URL it is working in. That turns size into a routing decision: keep /llms.txt as a short front door for the whole site, and put the dense subtree at /docs/llms.txt, linked from the root file like any other page. The discovery mechanics for those files — paths, link relations and reachability — decide whether they are found.

Two things worth remembering about the split. The root file must link to the subpath file, or nothing connects them. And the subpath file inherits the same budget pressure, so the same rules apply one level down.

What Size Discipline Does Not Buy

A lean file is a cost decision, not a growth lever. SE Ranking's study of roughly 300,000 domains found a 10.13% adoption rate and no correlation between having llms.txt and being cited in AI answers — removing the variable actually made their prediction model slightly more accurate. Ahrefs reached a similar place from the log side in May 2026: across 137,210 domains with a valid file, 97% received no request for it in a month.

Which is the argument for keeping the file small rather than for skipping it. The clients that do fetch it — developer tools, coding agents, a handful of crawlers — pay for every token, and a file that fits in context is the one that gets read. Start from the llms.txt generator if you want a lean first draft, keep it under a few kilobytes, and re-run the byte count whenever you add a section.