How to Measure Whether AI Engines Read Your llms.txt
Publishing /llms.txt is the easy half. Proving anyone read it is the half that
decides whether you keep maintaining the file. Here is a three-layer measurement stack —
raw logs, crawler behaviour, and AI referral traffic — that separates what actually happened
from what your analytics dashboard implies happened.
Why llms.txt Measurement Is Different
Most web measurement assumes a browser: JavaScript loads, a session starts, a referrer gets
recorded. AI crawlers do none of that. When OAI-SearchBot fetches your
/llms.txt, no GA4 session exists, no cookie is set, and no channel group fires.
The only evidence is a line in your access log.
That single fact reorganises the whole reporting problem into three independent questions:
- Was the file read? Direct fetches of
/llms.txt— the only signal that proves the file works. - Is your content being read? AI crawler hits on your pages, which happen with or without an llms.txt file.
- Is it producing anything? Humans arriving from AI assistants in GA4.
These signals move on different timelines and disagree for months — heavy crawler traffic with zero AI referrals is normal. Report them separately and you will never confuse a crawl with a citation.
Layer 1 — Count Direct llms.txt Fetches
On your own server, the file is one path. Filter for it and group by user agent:
# Did any AI agent request the file?
grep -i "llms.txt" /var/log/nginx/access.log
# Group the hits by crawler identity
grep -i "llms.txt" access.log | awk '{print $1, $12}' | sort | uniq -c | sort -rn
# Include rotated logs from the last month
zgrep -i "llms.txt" /var/log/nginx/access.log.*.gz | grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot"What you are looking for is a small number of non-zero hits, not volume. The baseline across the wider web is grim: our breakdown in Which AI Engines Actually Read llms.txt? covers the crawler audits showing that most published files are never requested even once. Two hits a month is a working file; zero after eight weeks means the file is being skipped for a reason — usually placement, an unparsable format, or links the agent cannot resolve.
User-agent strings are trivially spoofed, so confirm identity before reporting: resolve the requesting IP to a hostname, resolve that hostname back to an IP, and compare against the operator's published ranges. If they do not match, it was not that bot.
Layer 2 — Track AI Crawler Hits on Content Pages
llms.txt is a map. Whether agents follow it shows up as requests to the pages it lists. Pull the top paths per crawler token:
# Top pages fetched by AI agents this month
grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot" access.log \
| awk '{print $12}' | sort | uniq -c | sort -rn | head -20
This is the most useful diagnostic in the stack, because it tests curation directly. If the
paths agents fetch match the pages you listed in /llms.txt, the file is working
as a prioritisation layer. If they keep fetching pages you deliberately left out — category
archives, paginated tag pages, thin landing pages — the file is not steering anything yet,
and the fix is structural, not a matter of publishing it again.
Layer 3 — Read AI Referral Traffic in GA4
On May 13, 2026, Google added an AI Assistant entry to GA4's Default Channel Group. Sessions whose referrer matches a recognised AI assistant are tagged automatically:
| Field | Value assigned |
|---|---|
| Default channel group | AI Assistant |
| Session medium | ai-assistant |
| Campaign | (ai-assistant) |
Find it in Reports → Acquisition → Traffic acquisition with Session default channel group as the primary dimension. Two caveats matter before you put the number in a report. First, Google's live Default Channel Group documentation lists ChatGPT, Gemini, Deepseek, Copilot and Grok, and states the channel excludes AI Overviews and AI Mode — so if you want Claude, Perplexity or AI Mode measurement, you still need a custom channel group with a regex on session source above the generic Referral channel. Second, this channel counts humans who clicked through. A busy AI crawler fleet produces exactly zero of these sessions.
Layer 4 — Use Platform-Side Tooling Where You Have It
If your site sits behind Cloudflare, AI Crawl Control exposes a crawler-level view you cannot get from GA4: which AI services are accessing the domain, request patterns over time, whether a crawler respects your robots.txt, and per-crawler enforcement rules. Its Metrics tab also reports a Content Format breakdown — what content types AI systems request versus what your origin serves — and the changelog documents an option that redirects verified AI training crawlers to a page's canonical URL while humans and search crawlers see the original.
Cloudflare's older "Block AI bots" toggle is documented as deprecating on September 15, 2026, so plan a migration if your enforcement depends on it. And these dashboards report what reached the edge — not what an assistant chose to cite. Useful governance, not a truth machine.
A 30-Day Benchmark Table
Put the three signals in one table and review monthly. This is the whole reporting system:
| Signal | Where to read it | What good looks like |
|---|---|---|
| llms.txt fetches | Access logs, filtered on llms.txt | Any non-zero count from a verified AI agent |
| Crawler hits on listed pages | Access logs, grouped by user agent and path | Top paths overlap the sections in your file |
| AI referral sessions | GA4 AI Assistant channel, plus a custom channel group | Any stable, non-seasonal trickle worth segmenting |
Before you spend a quarter tracking it, spend five minutes validating the file itself. Run it through the llms.txt checker to confirm each link resolves, every entry carries a description, and the header block follows the spec. If you are rebuilding from scratch, the generator turns a sitemap into a first draft. A file that fails validation will never produce a log line worth reporting — and no dashboard will tell you why.