Maintenance · September 2026

llms.txt Maintenance: Keeping the Map Honest

Publishing the file takes ten minutes. The interesting question is the one nobody writes about: what happens to it over the next two years, while your URLs, your product names and your documentation structure keep moving.

Staleness Is a Content Bug, Not a Delivery Bug

Delivery problems — the wrong content type, a rewrite rule, a blocked crawler — are loud and fixable in an afternoon; they are covered in the delivery-level troubleshooting guide. Staleness is quiet. The file still returns 200, the format is still valid, and every link is still syntactically perfect — but three of them now 404, and the H2 section called Pricing points at a page you retired last spring. An agent that follows your map and hits 404s learns that your curated file is less reliable than your sitemap, which is the opposite of the file's purpose.

There is a second, slower cost: summaries drift. If the blockquote under your H1 still describes a product you repositioned six months ago, that sentence travels with every retrieval that touches your domain. Curation is a claim about your site, and claims expire.

What Actually Changes (and What Each Change Forces)

Most site changes need no edit at all. These six do:

Change Edit required
A new top-level section or product line Add an H2 and its 3–6 links; update the summary if the positioning moved
URL restructuring or a rename Replace the URL — do not keep the old one alongside the redirect
A page in the file was deleted Remove the line; leave the section if it still has three or more links
A page is now your best entry point Move it into the first section — order is editorial information
A new language or regional site Decide root file versus per-language files before adding URLs
Content moved to a subdomain you do not advertise Point at the canonical host, never the alias an agent cannot index

Nothing on that list requires a new format, a version bump or a migration. It is editing a markdown file you already own, which is why the expensive part is remembering to do it.

Layer 1: Make Dead Links Fail Your Build

The single highest-value habit is a link check that runs on every deploy and turns a 404 into a failed pipeline. A file that lists twenty URLs is small enough for a plain shell script — no dependency, no framework:

#!/usr/bin/env bash
# scripts/check-llms-links.sh — exits 1 if any URL in llms.txt is not 200
fail=0
while read -r url; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -L -A "GPTBot/1.5" "$url")
  if [ "$code" != "200" ]; then
    echo "DEAD LINK  $code  $url"
    fail=1
  fi
done < <(grep -oE 'https?://[^ )]+' public/llms.txt | sort -u)
exit $fail

Run it locally before you push, and once after every deploy against the production host. The -L flag matters: it follows redirects, so a link that only works through a hop shows up as a pass. If you would rather not allow redirects at all — several engines will not follow them for this file — drop -L and treat a 301 as a failure too.

Add the same script to your CI workflow and to a weekly scheduled run. Weekly is enough: a link that breaks on Monday and is reported on Friday costs you four days of failed fetches, not four months.

Layer 2: Generate What Can Be Generated, Curate the Rest

Split the file into the part a machine can keep current and the part only you can write. Sitemap derived lists of every URL belong to the automated side; the /generator builds exactly that from a sitemap.xml if you want a starting point. The header summary, the section names and the ordering are the editorial side, and they should be reviewed by a human — automation that refreshes the whole file nightly tends to produce an alphabetical dump that loses the curation that made the file useful in the first place. The nine best-practice rules are mostly about that editorial layer.

Layer 3: A Quarterly Review You Can Actually Finish

Ten minutes, once a quarter, in this order:

When Not to Touch It

Not every week is a version. Adding a blog post that is not one of your best pages is not a reason to edit the file — that is what sitemap.xml is for, and the two files have different jobs; the comparison covers the split. Constant churn also fights caching: agents and proxies that hold a copy of your file have to reconcile changes, and a file that reorders itself weekly is noise. Keep the URL fixed, keep absolute links, and let the file improve by accretion.

Once the automation is in place, staleness stops being a memory problem. The build fails, you fix the link, and the map stays honest. If you have not published a file yet, start with the generator, then push the production URL through the checker before you automate anything.