Why is my llms.txt file being ignored?

Most often because of placement or format: agents fetch /llms.txt at the root over HTTPS, and the file must start with an H1 on the very first line, use absolute URLs, and link to pages that actually return 200. The second reason is expectation — most AI crawlers such as GPTBot and ClaudeBot still fetch your HTML pages directly, so llms.txt is a control and context layer rather than a guaranteed citation lever.

Do AI engines actually read llms.txt?

Partly. Adoption is real and growing — about 5.6% of the top 10,000 sites published a valid file as of June 2026, and Chrome Lighthouse audits for one by default. But crawler-log analyses (Limy measured 500M+ LLM bot events) show GPTBot, ClaudeBot, PerplexityBot and Google-Extended mostly crawl HTML directly. Treat llms.txt as the fastest path for agents that do check it, not as the only path to your content.

How do I check if my llms.txt is valid?

Fetch it with curl and confirm HTTP 200 at the exact root URL, then run the content through a validator. A valid file has an H1 with your site name on the first line, an optional blockquote summary, and H2 sections of markdown links with absolute URLs. The free validator at llmstxtgenerator.dev checks all of this against the llmstxt.org spec and scores your file 0–100.

Common errors · August 2026 · AI readiness

6 llms.txt Mistakes That Make AI Engines Ignore Your File

You published /llms.txt, so why do AI engines seem not to care? In most cases the file is technically "there" but fails in one of six ways: wrong placement, format errors, dead links, a site-wide dump, marketing language, or the wrong success metric. Here is how to find and fix each one.

Mistake 1: The file is where agents can't find it

AI agents look for llms.txt at the root of your domain over HTTPS — the same place as robots.txt. If your file only works on www while the crawler checks the apex domain, or a redirect chain swallows the request, the agent gets a 404 and moves on. The llmstxt.org spec also allows subpath files such as /docs/llms.txt, which cover the pages under their path — useful when you only control a directory, but a root file is what most agents check first.

Serve the file at https://yourdomain.com/llms.txt — lowercase, exact name, root path.
Make sure HTTPS redirects work before that URL, with no infinite loops.
Confirm the status code, not just that the file exists on disk:

curl -I https://example.com/llms.txt
# expect: HTTP/2 200
curl -s https://example.com/llms.txt | head -5

Mistake 2: The format fails validation

The spec is deliberately small, and the most common failures are the simplest. The H1 with your site name must be the very first line — the only required element. A byte-order mark, an HTML comment, or a blank line pushed in front of it by your CMS is enough to break the file for strict validators.

First line: # Your Site Name, nothing before it.
Blockquote summary directly after the H1.
Absolute URLs only: https://... — never /about.
One H1; markdown link list items under H2 headings such as ## Important links.

Run every revision through a validator and fix everything reported as an error, not just the warnings. The free validator checks exactly these rules — missing H1, H1 not on the first line, relative URLs, unrecognized lines — and scores the result 0–100.

Mistake 3: The links lead nowhere useful

llms.txt is a list of links. If those links 404, redirect to a login wall, or point at HTML pages buried in navigation and cookie banners, an agent that follows them gets no clean context. Links rot fastest after a redesign — a fact every site with an old llms.txt learns the hard way.

Link to markdown versions of pages when you can; both .md and .html.md URL forms are valid in the v2 spec.
Verify every listed URL returns 200, and re-check after each deploy.
Add a short description after each link so an agent knows what the page is about before fetching it.

Mistake 4: You dumped the whole site in

sitemap.xml lists every URL for search crawlers; llms.txt should curate the handful of pages that best represent your site for an AI agent that reads the file inside its context window. Hundreds of links dilute the signal — an agent skims, and your five most important pages disappear into the noise. See llms.txt vs sitemap.xml for the full comparison.

	llms.txt	sitemap.xml
Purpose	Curated index for AI agents	Complete URL list for search crawlers
Typical size	5–15 links	Thousands of URLs
Readers	AI engines and agents	Google and other search engines
Detail	One-line description per link	Metadata and priorities

Keep the file under a few dozen links. Secondary material can live in an Optional section, which agents may skip — put your core value in the first sections.

Mistake 5: You wrote marketing copy, not context

Agents summarize from what the file actually says. "Industry-leading," "next-generation," and "empowering businesses" carry zero information — an AI model has no way to verify them, so it ignores them. Write like you are explaining the site to a smart colleague:

# Acme Inc
> Acme is the industry-leading provider of next-generation
> solutions that empower businesses worldwide.

Compare that with a summary an agent can actually use:

# Acme Inc
> Acme builds payroll software for US small businesses,
> with 40,000 customers and a 12-person support team.

The same rule applies to link descriptions: "About us" is weaker than "About: founding story, team, and 12 years of payroll history." Specificity is what makes an agent confident enough to cite you.

Mistake 6: You expect rankings — and never re-check

In June 2026, Google updated its AI optimization guide to state plainly that llms.txt has no effect — positive or negative — on Search rankings or AI Overviews. Independent data points the same way: ALLMO pulled 94,614 cited URLs from 11,867 AI answers across five AI platforms and found exactly one /llms.txt page among them (0.001%), and Limy's analysis of 500M+ LLM bot traffic events shows GPTBot, ClaudeBot, PerplexityBot and Google-Extended overwhelmingly crawl HTML directly and skip the file. We covered the evidence in detail in does llms.txt affect SEO?.

None of that makes llms.txt useless — it is a control layer. Adoption is real and rising (about 5.6% of the top 10,000 sites had a valid file in June 2026, and Shopify pushed one to every store by default in 2026), and Chrome Lighthouse audits for the file by default. The mistake is judging it by ranking metrics instead of by whether it stays correct. Every quarter, and after every site change, run this checklist:

curl -I returns 200 at the exact root URL.
First line is still the H1 with your site name.
All links are absolute and return 200.
The file is under a few dozen links and still curated.
Summary and descriptions are factual, not hype.
Re-validated after every deploy.

Fix the six mistakes above and your llms.txt does its real job: it gives agents that do check it a fast, clean, accurate path to your best content. Generate a compliant file from your sitemap with the generator, then confirm every point above with the validator.