llms.txt Maintenance: Keeping the Map Honest
Publishing the file takes ten minutes. The interesting question is the one nobody writes about: what happens to it over the next two years, while your URLs, your product names and your documentation structure keep moving.
Staleness Is a Content Bug, Not a Delivery Bug
Delivery problems — the wrong content type, a rewrite rule, a blocked crawler — are loud and
fixable in an afternoon; they are covered in
the delivery-level troubleshooting guide.
Staleness is quiet. The file still returns 200, the format is still valid, and every
link is still syntactically perfect — but three of them now 404, and the H2 section called
Pricing points at a page you retired last spring. An agent that follows your map and hits
404s learns that your curated file is less reliable than your sitemap, which is the opposite of the
file's purpose.
There is a second, slower cost: summaries drift. If the blockquote under your H1 still describes a product you repositioned six months ago, that sentence travels with every retrieval that touches your domain. Curation is a claim about your site, and claims expire.
What Actually Changes (and What Each Change Forces)
Most site changes need no edit at all. These six do:
| Change | Edit required |
|---|---|
| A new top-level section or product line | Add an H2 and its 3–6 links; update the summary if the positioning moved |
| URL restructuring or a rename | Replace the URL — do not keep the old one alongside the redirect |
| A page in the file was deleted | Remove the line; leave the section if it still has three or more links |
| A page is now your best entry point | Move it into the first section — order is editorial information |
| A new language or regional site | Decide root file versus per-language files before adding URLs |
| Content moved to a subdomain you do not advertise | Point at the canonical host, never the alias an agent cannot index |
Nothing on that list requires a new format, a version bump or a migration. It is editing a markdown file you already own, which is why the expensive part is remembering to do it.
Layer 1: Make Dead Links Fail Your Build
The single highest-value habit is a link check that runs on every deploy and turns a 404 into a failed pipeline. A file that lists twenty URLs is small enough for a plain shell script — no dependency, no framework:
#!/usr/bin/env bash
# scripts/check-llms-links.sh — exits 1 if any URL in llms.txt is not 200
fail=0
while read -r url; do
code=$(curl -s -o /dev/null -w '%{http_code}' -L -A "GPTBot/1.5" "$url")
if [ "$code" != "200" ]; then
echo "DEAD LINK $code $url"
fail=1
fi
done < <(grep -oE 'https?://[^ )]+' public/llms.txt | sort -u)
exit $fail
Run it locally before you push, and once after every deploy against the production host. The
-L flag matters: it follows redirects, so a link that only works through a hop shows up
as a pass. If you would rather not allow redirects at all — several engines will not follow them for
this file — drop -L and treat a 301 as a failure too.
Add the same script to your CI workflow and to a weekly scheduled run. Weekly is enough: a link that breaks on Monday and is reported on Friday costs you four days of failed fetches, not four months.
Layer 2: Generate What Can Be Generated, Curate the Rest
Split the file into the part a machine can keep current and the part only you can write. Sitemap
derived lists of every URL belong to the automated side; the /generator builds exactly
that from a sitemap.xml if you want a starting point. The header summary, the section
names and the ordering are the editorial side, and they should be reviewed by a human — automation
that refreshes the whole file nightly tends to produce an alphabetical dump that loses the curation
that made the file useful in the first place. The
nine best-practice rules
are mostly about that editorial layer.
Layer 3: A Quarterly Review You Can Actually Finish
Ten minutes, once a quarter, in this order:
- Run the link check and delete or fix every failure.
- Read the summary line aloud. Would a stranger understand what the site is today?
- Check the first section: is your single best page still in it?
- Compare against the sitemap — did a whole section appear that the file never learned about?
- Confirm the file still returns
200withtext/plainfrom the canonical host. - Check the cache headers on that response, so you know how long a wrong version persists.
- If you measure fetches at all, compare the last quarter's numbers against the previous one — see how to measure llms.txt traffic.
When Not to Touch It
Not every week is a version. Adding a blog post that is not one of your best pages is not a reason to
edit the file — that is what sitemap.xml is for, and the two files have different jobs;
the comparison covers
the split. Constant churn also fights caching: agents and proxies that hold a copy of your file have
to reconcile changes, and a file that reorders itself weekly is noise. Keep the URL fixed, keep
absolute links, and let the file improve by accretion.
Once the automation is in place, staleness stops being a memory problem. The build fails, you fix the link, and the map stays honest. If you have not published a file yet, start with the generator, then push the production URL through the checker before you automate anything.