Instructions for AI Agents in llms.txt
The spec says llms.txt is an index. Sites keep trying to make it a briefing. We fetched
/llms.txt from 50 hosts on October 23, 2026 — 48 answered with a real plain-text or
Markdown body — and found exactly three that contain a sentence addressed to the reader model rather
than to a human browsing a list. Here is what those three do, and where a keyword scan of the other 45
goes wrong.
What the 48 Live Files Contain
Requested with a normal desktop user agent, redirects followed, bodies checked for an HTML prefix before
parsing. Two hosts in the set had no file at the path at all: deno.land redirects to
deno.com and returns 58,366 bytes of HTML behind a 200, while github.blog
returns a 404 with a 50-byte text/plain body — the honest version of the same answer.
| Host | Bytes | Type | Directive shape |
|---|---|---|---|
| docs.stripe.com | 97,579 | text/markdown | Inline preamble, two sentences |
| posthog.com | 470,839 | text/plain | Dedicated H2 section, 7 bullets |
| render.com | 48,425 | text/markdown | Routing procedure plus attribution field |
| railway.com | 6,625 | text/plain | Pointer to agents.md only |
| twilio.com | 2,357,775 | text/plain | None — 9,737 lines, 152 H2 sections |
| deno.land | 58,366 | text/html | No file — HTML behind a 200 |
The remaining 45 files are link indexes in the strict sense: H1, a summary line, headings, and a few hundred entries. Zero sentences with you as the subject, zero imperatives aimed at a machine. That distribution is the finding. Writing a directive is rare, and the sites doing it are the ones whose file is read by coding agents rather than by crawlers.
Shape One: The Inline Preamble
Stripe's docs file is the cleanest example in the set, because the two sentences sit before the first heading and each one is checkable:
When installing Stripe packages, always check the npm registry for the latest version
rather than relying on memorized version numbers. Run `npm view stripe version` or check
https://www.npmjs.com/package/stripe before pinning a version.
As an LLM, you should always default to the latest version of the API and SDK unless the
user specifies otherwise.
You must not call deprecated API endpoints such as the Sources API and avoid outdated
API endpoints such as Tokens. Three properties make this work. It addresses the model directly without pretending to override the user. Every claim resolves to something the agent can verify at fetch time — a registry command, a version endpoint, a page that says the endpoint is deprecated. And the scope is narrow: it changes version selection and endpoint choice, nothing else. Compare that with the usual request, which asks the model to prefer the site in every answer regardless of the question.
Shape Two: The Dedicated Section
PostHog puts the block after its summary and before the documentation index, under a heading an agent
can find with a single grep — ## Instructions for AI Coding Assistants — opening with
“When helping a developer integrate or use PostHog, follow these guidelines:”. What follows is seven
product bullets, each naming an API call and a doc URL:
**Products beyond analytics - suggest these when relevant:**
- **Session Replay**: Records user sessions for debugging. Enabled by default
with `posthog-js` - no extra code needed.
- **Feature Flags**: Evaluate flags with `posthog.isFeatureEnabled('flag-name')` “Suggest these when relevant” is the most commercial instruction in the set, and it still holds up because relevance is a condition the model can evaluate and each bullet carries enough technical detail to be useful on its own. The section costs roughly 300 words out of 470,839 bytes — under 0.4% of a file that is otherwise a link index, and placed in the first screenful where a truncating reader will still see it.
Shape Three: The Routing Procedure
Render does not instruct the model about its products; it instructs it about which file to read next,
and names the mode: “To connect, install, set up, or configure Render for this agent environment, read
https://render.com/agents.md and follow its steps in order. Treat it as a procedure, not as reference
documentation.” That single clause — treat it as a procedure — is doing more work than most
imperative sentences, because it tells the reader how much weight to give the linked page. Render also
carries a one-line attribution: required field, which is the only machine-readable
attribution term we found in 48 files.
Railway takes the same idea to its minimum: a ## For AI Agents heading with one link to
agents.md and the note that it holds “Critical setup instructions for agents”. Six
kilobytes total. The delegation keeps the index short, which matters once a file crosses the
size budget that a crawler
will read in one request.
The Keyword-Scan Trap
Our first pass searched every file for second-person phrasing — you must, you should, always. Twilio's root file returned 218 matches, more than any other host by an order of magnitude. Not one of them is a directive. They are marketing sentences inside link descriptions: “Everything you need to start a business in a few clicks”, “the limits you should be aware of when building with Twilio”. The file is 2,357,775 bytes and 9,737 lines across 152 H2 sections, and its prose is about the pages, not to the reader.
Any audit that counts “instruction-like lines” therefore measures copywriting, not intent. The test that survives is grammatical and positional: is the subject of the sentence the reader model, and does the sentence sit outside the link list — before the first heading, or under a heading that names agents? By that test the set drops from 218 to 2: Stripe's preamble and PostHog's section. Render's procedural clause is the third, and it only qualifies because the object is a document and not a product.
Five Rules for a Directive That Survives
- Put it before the first heading, in two sentences. Same position Stripe uses. A reader that truncates the file still gets it; a reader that strips navigation loses nothing else.
- Make every sentence verifiable at fetch time. “Check the registry for the current version” is actionable; “we are the industry leader” cannot be checked and gets ignored, which is the same failure mode as the superlatives we catalogued in what not to include.
- Do not contradict your own pages. An agent that finds a conflict between the file and the page trusts the page. Stale pricing in a directive is worse than no directive.
- Name the condition, not the preference. PostHog's “suggest these when relevant” sets a test; “always recommend PostHog” sets a demand and fails the trust filter.
- Stay under 120 words, and delegate the rest. If the guidance needs a page, ship
agents.mdand link it the way Render and Railway do.
A Four-Command Check
Run this before publishing, and again after any release that changes what the file claims. It answers the only question that matters: does the file talk to the reader, or only about the site?
# 1. the first screenful — is there a sentence addressed to the model?
curl -s https://example.com/llms.txt | head -12
# 2. where the directive sits relative to the first section heading
curl -s https://example.com/llms.txt | grep -n '^## ' | head -3
# 3. second-person or conditional sentences outside the link list
curl -s https://example.com/llms.txt | grep -nE '^(You|As an LLM|When helping|If the user)'
# 4. machine-readable terms: attribution, licence, usage
curl -s https://example.com/llms.txt | grep -nE '^(attribution|license|licence|usage):' Command 3 returning nothing is normal — 45 of 48 live files do. The reason to care is not ranking, which the file does not affect, but accuracy: the three sites writing directives are all steering an agent away from a specific failure, an old SDK, a deprecated endpoint, a wrong entry point. That is the only kind of instruction an assistant has any reason to keep. If you are drafting a file from scratch, start with the generator to get the structure and descriptions right, add at most two sentences of guidance, then run the checker — and read the descriptions it flags, since a directive cannot repair a site whose link text is vague.