Should a news site publish an llms.txt file?

Only if the readers you are writing it for are admitted by your robots.txt. On October 2, 2026 we fetched /llms.txt across 30 major publisher domains and not one returned a 200 — most answered 404 and several answered with a WAF block instead. In the same week, the robots.txt files of 18 news domains disallowed ClaudeBot and CCBot in 17 cases each. An index that curates pages for crawlers you have blocked is a file with no recipient. Publishers that allow answer-engine crawlers to reach their reporting have a real case for shipping one; publishers that closed the site to unnamed crawlers have already made the opposite decision.

Should llms.txt list individual news articles?

No. A newsroom publishes dozens to hundreds of URLs a day, most with a shelf life measured in hours, and a curated index of them is stale before the next editorial meeting. List section and topic hubs, evergreen explainers, and the policy pages that are stable for years — corrections policy, ownership and funding, licensing terms, archives policy. If you need per-article machine-readable structure, that belongs in your feeds, your article markup and your news sitemap, not in a Markdown index.

Does an llms.txt file get news content cited in AI answers?

It is not the driver. Independent crawl studies reported during 2026 found that almost no AI fetches target the file at all — Ahrefs, analysing 137,210 domains in May 2026, reported that 97% of published llms.txt files received zero traffic, with GPTBot and coding assistants accounting for most of the fetches that did occur. For publishers, citation share comes from being crawlable, from reporting that answers a question in liftable form, and increasingly from licensing agreements. The file is a cheap namespace to hold, not a distribution channel.

Publishing · October 2026

llms.txt for News and Media Publishers

Publishers are the vertical where llms.txt is most often simply absent — and where the decision is genuinely harder than on a documentation site. Here is what 30 news domains serve today, why the missing file is often deliberate, and the structure that works if you decide to ship one anyway.

What Publisher Domains Actually Serve Today

We fetched /llms.txt across 30 news and media domains from this server on October 2, 2026. Not one returned a 200. A representative slice of the responses:

Response	Sites
404 Not Found	nytimes.com, bbc.com, theguardian.com, cnn.com, npr.org, techcrunch.com, theverge.com, wired.com, latimes.com, vox.com, businessinsider.com, forbes.com, nbcnews.com
403 / 401 / 402 / 406 (bot or paywall challenge)	ft.com, wsj.com, bloomberg.com, politico.com, theatlantic.com, axios.com, engadget.com, reuters.com, usatoday.com, time.com
Redirect to `www`, then no response	washingtonpost.com
200 OK (controls, not newsrooms)	github.com, shopify.com, netflix.com

Read that table carefully before drawing a conclusion. A 404 is proof of absence. A 401, 402, 403 or 406 is not: it is an edge rule answering the request before any file is served, which is why a WAF-blocked publisher may be perfectly capable of serving Markdown to a client it recognizes. The honest summary is that newsrooms publish essentially no llms.txt files today, and several of them would refuse to hand one to an anonymous fetch.

Why the File Is Missing, Usually on Purpose

llms.txt is a navigation aid for readers you have already admitted. Most large newsrooms spent 2024–2026 closing the door instead. Parsing the robots.txt of 18 news domains on the same day, with a specific disallow rule taking precedence over the * group, gives this picture:

Crawler	Domain-group	Disallowed by	Not named
`ClaudeBot`	training	16 of 18 explicitly, plus reuters.com via `*`	scmp.com
`CCBot`	training	16 of 18 explicitly, plus reuters.com via `*`	scmp.com
`GPTBot`	training	10 explicitly, plus reuters.com via `*`	theguardian.com, nypost.com, elpais.com, lemonde.fr, scmp.com
`OAI-SearchBot`	search / answers	8 explicitly, plus reuters.com via `*`	theguardian.com, techcrunch.com, theverge.com, apnews.com and others

Two configurations stand out. reuters.com closes the entire site to unnamed crawlers with a single wildcard rule, then adds a named block for Applebot-Extended on top — of the eight AI agents we tested, not one is admitted, file or no file. theverge.com is the rare opposite: it names GPTBot with an explicit allow and then blocks ClaudeBot, PerplexityBot, Google-Extended, CCBot and Bytespider — a hand-drawn line between one training crawler and the rest. Both sites are making a policy choice; neither is an accident. The vocabulary for those choices is in robots.txt for AI crawlers.

If You Do Ship One: Index Sections, Not Stories

The one thing a newsroom must not do is generate the file from the article feed. Story URLs are published hundreds of times a day, get retitled, re-slugged and syndicated, and a large share of them sit behind a registration or paywall. An index of stories is stale within hours and actively harmful when an assistant follows a link into a paywall page.

What a publisher file should index is the part of the site that behaves like a reference: section fronts, topic hubs, explainers and evergreen backgrounders, and the masthead pages that describe how the newsroom works. Those URLs live for years and answer most of the questions an assistant has about your publication.

The Six Buckets That Fit a Newsroom

Section and topic hubs. One link per desk — world, business, technology, climate — plus the topic hubs worth naming. Evergreen topic pages beat chronological category listings, which churn.
Explainers and backgrounders. The evergreen pieces that answer a standing question. These are the publisher's closest equivalent to documentation.
Standards and trust pages. Editorial standards, corrections policy, ownership and funding, ethics guidelines, contact for the newsroom. Rarely linked, frequently asked about.
Access terms. Licensing and syndication terms, paywall and registration policy, archive availability, permissions contact. If an assistant is deciding whether it may quote you at length, this is the page it should read.
Adjacent products. Newsletters, podcasts, apps, data projects — with a link each.
Optional. Live blogs, editions and special projects, marked so a client under a tight context budget can skip them.

# The Daily Example

> The Daily Example is an independent newsroom covering regional politics,
> energy and public finance, publishing about 40 stories a day plus
> newsletters and a weekly data project. Free to read; archive access
> requires registration. Licensing terms: /licensing/

## Sections

- [World](https://example.com/world/): international news and analysis
- [Business](https://example.com/business/): markets, companies, public finance
- [Climate & Energy](https://example.com/climate/): transition policy, grids, utilities

## Explainers

- [How the regional grid is priced](https://example.com/explainers/grid-pricing/): standing reference on tariff design
- [Reading a municipal budget](https://example.com/explainers/municipal-budget/): annual primer, updated each spring

## Newsroom standards

- [Editorial standards](https://example.com/standards/): sourcing, verification and AI policy
- [Corrections](https://example.com/corrections/): what we correct and how to submit one
- [Ownership and funding](https://example.com/about/funding/): who pays for this newsroom
- [Licensing](https://example.com/licensing/): syndication, archive and AI-use terms

## Optional

- [Live blog](https://example.com/live/): rolling coverage, replaced daily
- [Data projects](https://example.com/data/): interactive datasets and methodology notes

Paywalls, Syndication and Duplicate Copies

Never list a walled article URL in the content sections. An agent that fetches it receives a registration interstitial, and some clients will treat the visible teaser as the page's content — you end up quoted from your own paywall screen. Link the free hub, the explainer, or the licensing page that explains what a machine reader may have instead.

Syndication creates the second problem: the same story on partner sites and aggregators. The summary blockquote is the right place to state the canonical host ("the version of record is published here"), and each listed URL should be your own copy. Consistency with canonicals and hreflang matters more here than on any other site type, and the same discipline applies to publisher editions in other languages.

One boundary to keep straight: llms.txt does not grant or withhold access, and it does not opt anything out of training. It is an index. The permission decisions live in robots.txt and the licensing terms the file points to.

Make the File Reachable, Then Verify It

If your robots.txt admits OAI-SearchBot and blocks GPTBot, your llms.txt is written for the first reader only, and it must not be blocked at the edge for the agents you did admit. Three checks catch nearly every failure:

# 1. Is the file served as text, or does the CDN answer with HTML?
curl -sI https://example.com/llms.txt | grep -iE "^(HTTP|content-type|location)"

# 2. Fetch it the way the admitted crawler would
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://example.com/llms.txt

# 3. Every listed URL should answer 200 without a redirect chain
curl -s https://example.com/llms.txt | grep -oE "https://[^)]+" | while read u; do
  printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u"
done | grep -v "^200"

Then paste the result into the llms.txt checker for structure, and keep the file on the same review cycle as your editorial standards — a section that closes, an explainer that gets retired and a licensing page that moves are all quiet link rot, which the maintenance pass is designed to catch. Building a first draft from a topic-hub sitemap with the generator works fine, as long as the feed of daily stories is left out of it.

All status codes and robots.txt groups referenced were captured from this server on October 2, 2026, with a desktop browser user agent. Edge bot rules change without notice, so re-check your own paths rather than treating this table as current configuration for any site named. A 4xx status may indicate an edge rule rather than a missing file.