Is Content-Signal part of the llms.txt spec?

No. Content-Signal is a robots.txt extension proposed through the Content Signals Policy, whose documentation Cloudflare's own robots.txt points at (contentsignals.org and the IETF draft draft-romm-aipref-contentsignals). llms.txt is a separate convention that describes what your site contains. One states how crawled content may be used (search, ai-input, ai-train); the other describes which URLs are worth reading. They live in different files and are read by different code paths.

Does a Content-Signal line block AI crawlers?

It does not. Content-Signal is a preference declaration, not an access rule, and a crawler that ignores it still gets your pages unless robots.txt disallows them. In our October 26, 2026 probe none of the six hosts that declare a signal had a single user-agent group returning Disallow: / — they trained preference through the signal instead of blocking. If enforcement matters to you, pair the signal with an actual Disallow rule or edge-level bot controls, and expect the signal to be advisory either way.

Which file should I ship first, llms.txt or a Content-Signal?

llms.txt, because it is what an assistant reads to answer a question about you today, while a content signal only constrains what a crawler may do with content it has already fetched. The observed pattern in our sample is that sites ship the index first: all six signal-declaring hosts already served a working llms.txt. Add the signal once your access rules are decided, and keep the two consistent — do not declare ai-train=no in robots.txt while an llms.txt invites bulk reading of every page with no usage note.

GEO · October 2026

Content Signals vs llms.txt: Two Files, Two Different Permissions

llms.txt tells an assistant what your site contains. A Content-Signal line in robots.txt tells a crawler what it may do with what it already fetched. They are routinely described as alternatives, which they are not. We pulled both files from 30 hosts on October 26, 2026 to see how they actually coexist.

The 30-Host Result

The probe requested https://<host>/robots.txt and https://<host>/llms.txt with a plain GET and a descriptive user agent, recording status, byte count and content type. The 30 hosts are a mix of developer platforms, SaaS brands and publishers:

Outcome	Hosts	Detail
Serve a real llms.txt	16 / 30	200 with `text/plain` or `text/markdown`
Declare a Content-Signal	6 / 30	Cloudflare, Supabase, Render, Twilio, Webflow, GitBook
Signal + working llms.txt	6 / 6	No host declares a signal without an index
Use `use=` (content-use field)	0 / 6	The newer retention field is not in production yet
No llms.txt at all	14 / 30	10× 404, plus 401, 403 (×2) and 429 behind HTML bodies

Two numbers matter here. Signal adoption (6) is a third of index adoption (16), and the overlap is total: the six sites that declare usage preferences are the same six that already publish a curated index. Nobody in this sample runs the permission layer without the description layer.

What the Six Files Actually Say

Only two of the six withhold training rights. Twilio and GitBook both go search-yes, ai-input-yes, ai-train-no:

# GitBook, https://www.gitbook.com/robots.txt
# Content signals declare how this site's content may be used.
# search=yes    — may be used to build a search index and link back
# ai-input=yes  — may be used as input for AI answers that cite this site
# ai-train=no   — may not be used to train generative AI models
# See https://contentsignals.org

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Cloudflare, Supabase, Render and Webflow declare the opposite — ai-train=yes, search=yes, ai-input=yes — which is worth pausing on, because the Content Signals Policy's own scale story is different: Cloudflare reported applying search=yes, ai-train=no to the managed files of more than 3.8 million customer domains. Its corporate zone grants training while its default grant to customers denies it. There is no contradiction in that, but it means "the industry default" is not a single value you can infer from any one file.

All four permissive files also write their fields in the same order, while the two restrictive ones differ from each other — Twilio leads with ai-train=no, GitBook ends with it. Since the draft defines a comma-delimited set, order should not matter; parsers that match lines textually will still treat them differently.

Where the Line Sits in the File

Placement varies just as much. GitBook and Twilio put the signal inside the User-agent: * group, next to their Allow rules. Cloudflare.com places it after eight per-bot Allow groups, at top level, wrapped in its own comment header. Render repeats the identical line inside a named bot group. That variance is the practical hazard: a robots.txt parser built to read only User-agent/Allow/Disallow skips the line entirely and reports a clean file, so your own monitoring will not tell you whether the signal is being seen.

Cloudflare's documentation also describes content-use, an optional fourth field taking a use= value that states what a crawler may keep after access. None of the six production files we fetched used it.

Preference, Not Enforcement

The most consistent finding is negative: across the six signal-declaring hosts, not one had a single user-agent group whose rule was Disallow: /. Not Twilio, which denies training. Not GitBook, which denies training and invites AI answers. They chose a declaration over a block, and they left the generic Allow: / in place. If you want a crawler kept out, only a disallow or an edge rule does that — the same distinction that separates allow, block and feed decisions from a file that merely describes your site.

The other thing missing from all 16 live llms.txt files is a usage statement. We read the license-shaped lines in each one — Stripe, GitHub, Cloudflare, PostHog, Twilio, Framer and ReadMe all contain the word, 24 matches in total — and every hit was an article title about business licences or a product with "training" in its name. No file states terms. Licensing lives in robots.txt; description lives in llms.txt. Keep that split and the two files stop contradicting each other.

Shipping Both Without Contradicting Yourself

# robots.txt — permission layer, one line, one group
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

# llms.txt — description layer, generated from your sitemap
# Example Domain

> One sentence on what this site is and who it is for.

## Docs
- [Getting started](https://example.com/docs/start.md): What the first hour looks like

Generate the index with the llms.txt generator, validate it with the checker, then confirm both layers are live — status is not enough, read the body:

# the signal must be in the file that ships
curl -s https://example.com/robots.txt | grep -i 'content-signal'

# and the index must be text, not an HTML page
curl -s -D - -o /tmp/l.txt https://example.com/llms.txt | grep -iE '^(HTTP|content-type)'
head -n 3 /tmp/l.txt

# cross-check: nothing you list should be disallowed
curl -s -o /dev/null -w '%{http_code}\n' https://example.com/docs/start.md

Decide the usage question once, write it in one place, and let the index do the describing. If you are still choosing your crawler stance, the robots.txt comparison covers the access rules and the crawler data covers who actually shows up.

Probes run October 26, 2026 with a plain GET against 30 hosts. Signal lines quoted verbatim: cloudflare.com, supabase.com, render.com, twilio.com, webflow.com, gitbook.com. robots.txt and llms.txt change without notice; re-fetch your own pair after any CDN or bot-management change.