Site root files · Search & AI discovery · Updated 2026

llms.txt vs robots.txt vs sitemap.xml: What's the Difference?

Three tiny files, three very different jobs. robots.txt controls which crawlers may access your site, sitemap.xml tells search engines which URLs exist, and llms.txt tells AI engines what's worth reading.

Open the root directory of almost any serious website and you will find a small set of plain files: robots.txt, sitemap.xml, and — increasingly — llms.txt. They look similar — tiny, text-based files in the same folder, served to automated visitors — so it's tempting to assume they do the same job. They don't: each is read by a different consumer and answers a different question.

Below we compare all three and show how they work together.

What Is robots.txt? — Crawler Permission Control

robots.txt is the oldest of the three, dating back to the Robots Exclusion Protocol of 1994. It is a plain-text file placed at the root of a domain (https://example.com/robots.txt) that tells compliant crawlers which parts of the site they may or may not crawl.

Who reads it: search-engine crawlers — Googlebot, Bingbot, YandexBot, DuckDuckBot — and increasingly AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot. Any compliant crawler checks robots.txt before it fetches anything else on your domain.

What it does: robots.txt does not rank, index, or describe anything — it is a permission system. Rules like Disallow: /admin/ stop compliant crawlers from requesting those paths, keeping private sections out of the crawl. It can also point crawlers at your sitemap via a Sitemap: directive.

User-agent: *
Disallow: /admin/
Disallow: /private/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

One crucial caveat: robots.txt only governs well-behaved crawlers, and blocking a URL does not remove it from Google — it can even leave a bare URL-only result with no snippet. For true removal, use a noindex meta tag or HTTP header instead.

What Is sitemap.xml? — The Complete URL Inventory

sitemap.xml is an XML file that follows the Sitemap Protocol from sitemaps.org. It is an inventory of your site: a machine-readable list of every URL you want search engines to know about, optionally annotated with <lastmod>, <changefreq>, and <priority> hints.

Who reads it: search-engine indexers. Google, Bing, and others check the sitemap (typically referenced from robots.txt or submitted via Search Console) and use it to discover and prioritize new or rarely linked URLs.

What it does: a sitemap has no effect on rankings — it improves discovery. Pages that would otherwise wait weeks to be found through internal links get crawled quickly, and a fresh <lastmod> helps crawlers notice updates. Large sites split sitemaps into multiple files via a sitemap index.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-08-01</lastmod>
  </url>
  <url>
    <loc>https://example.com/docs/getting-started</loc>
    <lastmod>2026-08-10</lastmod>
  </url>
</urlset>

Keep in mind a sitemap is a machine inventory, not a recommendation: it lists every URL with no notion of what is most important for a human or an AI to read first.

What Is llms.txt? — A Content Index Built for AI Engines

llms.txt is the newest of the three. Proposed by AI researcher Jeremy Howard in 2024 and specified at llmstxt.org, it is a small Markdown file at the root (https://example.com/llms.txt) that gives AI engines a curated, human- and LLM-readable index of your content. The file opens with an H1 title and an optional blockquote summary, followed by sections of Markdown links, each with a short description. It is designed to fit in a model's context window, so an agent can understand your whole site from one small file.

Who reads it: AI engines and agents — ChatGPT, Claude, Gemini, Perplexity, coding assistants, and retrieval tools. Adoption is accelerating: Chrome Lighthouse audits for it, platforms like Mintlify and GitBook generate it automatically, and OpenAI, Anthropic, and Google publish llms.txt files for their own docs.

What it does: where robots.txt says what not to crawl and sitemap.xml lists every URL, llms.txt tells an AI engine what the site is about and which pages to read first. It is curation, not inventory: you order links by importance, add descriptions, and point agents at clean Markdown versions of your pages. There is no ranking or crawling behavior attached — it is simply a content index.

# Example Docs

> Example is a developer platform for building serverless applications in Python and TypeScript.

## Important links

- [Quick start](https://example.com/docs/quickstart.md): Set up your first app in 10 minutes
- [API reference](https://example.com/docs/api.md): Complete REST API reference

## Optional

- [Changelog](https://example.com/changelog.md): Release notes for all past versions

llms.txt vs robots.txt vs sitemap.xml: Side-by-Side Comparison

Here is the full comparison. The short version: robots.txt manages crawler access, sitemap.xml feeds indexing, and llms.txt serves AI engines — three different consumers, three different jobs.

Can You Use robots.txt, sitemap.xml, and llms.txt Together?

Absolutely. None of these files substitutes for another — they work best as a pipeline answering three different questions:

Crawler flow: robots.txt → sitemap.xml → pages. AI flow: llms.txt → linked Markdown pages. robots.txt points crawlers at the sitemap via a Sitemap: directive; llms.txt is served directly to AI engines.

A practical setup for a modern site:

• Point crawlers at your sitemap with a Sitemap: directive in robots.txt, so discovery is automatic.
• Allow GPTBot, ClaudeBot, and PerplexityBot instead of blanket-blocking them, then control what they consume with a well-curated llms.txt.
• Link to llms.txt from your homepage or footer, and keep it in sync with your sitemap — add new guides to both files.
• Use robots.txt to protect genuinely private paths, and noindex — not robots.txt — to remove indexed pages from Google.

Common Misconceptions About These Three Files

• “llms.txt will boost my Google rankings.” — No. Google does not read llms.txt for ranking; the file targets AI engines such as ChatGPT, Claude, and Perplexity. For better Google rankings, invest in content quality, internal links, and Core Web Vitals — and treat llms.txt as a separate channel for AI visibility.
• “robots.txt removes pages from Google.” — No. robots.txt blocks crawling, not indexing. A blocked URL can still appear in results. Use noindex for removal and robots.txt for crawl control.
• “llms.txt is just a sitemap in Markdown.” — No. A sitemap is an exhaustive machine inventory; llms.txt is a curated, ordered recommendation with a summary and descriptions. Sitemaps are read by indexers; llms.txt by agents that reason about your content.
• “A complete sitemap.xml is enough for AI engines.” — Increasingly not. AI engines can crawl sitemaps, but increasingly look for llms.txt by convention because it tells them what matters in one small file — different consumers, complementary files.

FAQ: llms.txt vs robots.txt vs sitemap.xml