Does blocking GPTBot stop ChatGPT from citing my site?

No. GPTBot is OpenAI's training crawler, not its answer engine. Citations in ChatGPT search come from OAI-SearchBot and, for live fetches, ChatGPT-User. Blocking only GPTBot opts your pages out of future training data while leaving your citation surface intact.

Should I allow or block AI crawlers in robots.txt?

Decide per crawl class rather than per company. Allow the retrieval agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want to be cited in AI answers, and block the training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent) if you do not want your content in training corpora. Most publishers end up with exactly that split.

How does llms.txt differ from robots.txt for AI engines?

robots.txt grants or denies permission to fetch; llms.txt curates which pages are worth reading once fetching is allowed. They are complementary: a permissive robots.txt plus a well-structured /llms.txt file gives agents both access and a map. Generate the map from your sitemap and validate it before publishing.

AI crawlers & robots.txt · September 2026

robots.txt for AI Crawlers: Who to Allow and Who to Block

robots.txt was designed to referee search engines. In 2026 it is refereeing a dozen AI agents from six different companies — and getting the policy right depends on one thing most guides skip: which class of crawler you are talking to. Here is the token list, two pastable policies, and how your llms.txt file picks up where robots.txt stops.

The Three Classes of AI Crawler

Every AI user agent worth configuring falls into one of three job descriptions. The class decides what blocking actually costs you:

Class	What it does	Cost of blocking
Training	Collects pages for model pre-training corpora	None for today's citations — it affects future model weights
Search index	Indexes pages so the assistant can retrieve and cite them	Your pages stop being discoverable in AI answers
User-triggered	Fetches one page live when a user pastes a URL or asks about it	Often unavoidable — these behave like a browser on a user's behalf

That split is the whole game. Publishing teams that block "AI" wholesale by blocking one or two famous tokens usually block the harmless training crawler while leaving the retrieval agents untouched — or the reverse, and quietly remove themselves from AI answers.

The 2026 User-Agent Token Reference

These are the tokens site owners ask about most. "Honours robots.txt" reflects each operator's published documentation, not a guarantee — verify against your own logs.

Token	Operator	Class
`GPTBot`	OpenAI	Training
`OAI-SearchBot`	OpenAI	Search index
`ChatGPT-User`	OpenAI	User-triggered
`ClaudeBot`	Anthropic	Training
`Claude-SearchBot`	Anthropic	Search index
`Claude-User`	Anthropic	User-triggered
`PerplexityBot`	Perplexity	Search index
`Perplexity-User`	Perplexity	User-triggered
`Google-Extended`	Google	Training opt-out token
`CCBot`	Common Crawl	Training corpus
`meta-externalagent`	Meta	Training

Two traps hide in that table. First, Google-Extended is not a crawler at all — it is a robot token that opts you out of Gemini and Vertex training while leaving Googlebot, Search and AI Overviews completely unaffected. You cannot opt out of AI Overviews without blocking Google Search itself. Second, Anthropic formalised the split in February 2026, documenting ClaudeBot, Claude-SearchBot and Claude-User separately and warning that blocking Claude-SearchBot reduces your site's visibility in its search results. Meanwhile OpenAI's user-triggered ChatGPT-User is documented as fetching on a user's behalf, which makes it effectively unmanageable from robots.txt.

Two robots.txt Policies You Can Paste Today

Policy A — Get cited, do not get trained

The split most publishers want: retrieval allowed, training denied.

# Retrieval agents: allow
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /

# Training crawlers: deny
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /

# Google/Gemini training opt-out token
User-agent: Google-Extended
Disallow: /

Policy B — Maximum visibility

If your goal is AI citations above all — documentation, tools, service pages — allow the agents that can recommend you, and keep them out of private paths only:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /checkout/

Rules for a specific token always override the wildcard group for that bot, so add a bot's group deliberately rather than relying on User-agent: *. And always test the published file, not the draft — curl https://yourdomain.com/robots.txt is the only version agents see.

Cloudflare Content Signals: Usage Terms Inside robots.txt

Since Cloudflare introduced its Content Signals Policy, robots.txt has grown a second layer that describes how content may be used, not just whether it may be fetched. Three signals are defined — search (building a search index), ai-input (feeding content into models for live answers) and ai-train (training or fine-tuning) — and sites can declare preferences with comma-delimited yes/no values:

# Content Signals Policy
User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Cloudflare's documentation states that domains without a robots.txt of their own may be served a default signal set of search permitted and AI training declined, and that content signals are a stated reservation of rights rather than a technical enforcement layer — enforcement still happens at the edge, through bot management rules. Treat signals as documentation of intent that strengthens your position; treat the allow/disallow rules above as the part that actually changes behaviour.

Where llms.txt Fits: Permission vs Curation

robots.txt answers "may you read this?" llms.txt answers "what is worth reading?" Once you have allowed the retrieval agents, an /llms.txt file gives them a curated map: your best pages, one-line descriptions, and Markdown links that skip the HTML scaffolding. For large sites, llms.txt v2 formalises subpath files such as /docs/llms.txt and an llms-full.txt dump for full-context reads — Cloudflare's own developer docs run exactly this hub-and-spoke pattern.

The practical sequence is: audit your robots.txt groups, allow the retrieval tokens you want, then publish the map. Generate a first draft from your sitemap.xml with the llms.txt generator, validate it with the checker, and compare the two files properly in llms.txt vs robots.txt. Remember that permission plus a map still leaves you at square one if agents never fetch: our crawl-data breakdown in Which AI Engines Actually Read llms.txt? shows how uneven that coverage still is in 2026.

Verify the Bots With Your Own Logs

Every claim above is testable on your own server. The monthly log check:

# Which AI agents actually reached you?
grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot" access.log | awk {print $1, $12} | sort | uniq -c | sort -rn | head -20

Then confirm identity — user-agent strings are trivially spoofed. The standard check is a double reverse-DNS lookup: resolve the requesting IP to a hostname, resolve the hostname back to an IP, and compare against the operator's published ranges. If the values do not match, it was not that bot. Run this audit once a month, and you will know within a single cycle which of your robots.txt rules are reshaping real traffic — and whether the llms.txt file you published is being read at all.