Should a forum or community site publish an llms.txt file?

Yes, but not as a page list. On a user-generated site the valuable content is fragmented across millions of changing URLs, so an exhaustive index is worthless within a week. The two community sites that do serve a real file in our October 17, 2026 sample of 34 hosts — dev.to (2,835 bytes) and forum.openwrt.org (1,720 bytes) — both publish conduct rules and a handful of query endpoints, and neither lists a single individual thread. That is the correct shape: one H1, a blockquote summary, a few sentences telling an agent what it may and may not do on behalf of a member, then links to search, category and policy pages an agent can use to find live discussions on its own.

Does llms.txt work if the community platform returns a 403 to crawlers?

No, and this is the dominant failure mode for forums. Twenty of the 34 community hosts we probed returned 403 for /llms.txt, and thirteen of them answered with a byte-identical 295-byte HTML body naming Discourse's own address and the error code d/t/44262 — an IP-level restriction from a datacenter range, not an editorial decision by each forum. The practical consequence is asymmetric reachability: an IDE agent running on a residential connection may read the file fine while a cloud crawler behind a datacenter IP is refused. If your forum is the one serving the file, test it from a second network and a second user agent before assuming it is published.

What should never appear in a forum's llms.txt?

Thread URLs, user profile pages, private-category routes and search-result URLs. Topic URLs rot faster than any other kind of URL on the web — threads get merged, split, renamed, locked and moved between categories — so a file listing them starts decaying the day you ship it. Profile pages carry personal data, and a file is public. Restricted or logged-in categories must not be advertised at all, because you are telling an agent a path exists that you cannot let it read. List the query interface instead: a search route, a tag or category filter, the latest-activity feed and the sitemap. Those four links stay correct for years.

Community & UGC · October 2026

llms.txt for Forums and Community Platforms

A forum is the one site type where the content changes faster than any file can describe it, and it is also the one where an agent can do real damage by posting. We fetched /llms.txt from 34 forum, community and user-generated-content hosts: two serve a real file, twenty are refused at the edge, and the two live files contain zero thread URLs between them. Here is the full sample and the shape that works.

What 34 Community Hosts Serve Today

On October 17, 2026 we requested /llms.txt from 34 community hosts with a research user agent, redirect following and a 15-second timeout. Two produced a real plain-text file, one answered 200 with HTML behind the path, twenty refused the request before the origin saw it, six returned a 404 and five never completed a response at all.

Response	Count	Examples
`200` with a real text file	2	dev.to (2,835 bytes), forum.openwrt.org (1,720 bytes)
`200`, but HTML behind the path	1	community.atlassian.com (80,936 bytes of app shell)
`403` from the edge	20	13 × a 295-byte Discourse body (meta.discourse.org, discuss.python.org, community.openai.com, forums.swift.org, discuss.elastic.co, community.sonarsource.com and seven more); 2 × 536 bytes (forum.cursor.com, community.spiceworks.com); 3 × Cloudflare JS challenge, 5,570–5,596 bytes (stackoverflow.com, meta.stackexchange.com, forums.raspberrypi.com); reddit.com (190,240 bytes); community.instructure.com (10 bytes, `text/plain`)
`404`	6	news.ycombinator.com (146-byte body), quora.com (77,465 bytes after one redirect), discourse.org (45,548 bytes), community.home-assistant.io, forum.manjaro.org, discuss.frappe.io (0 bytes)
No response at all	5	lobste.rs, community.anthropic.com, discuss.nextcloud.com, community.figma.com, forum.arduino.cc

Two details are worth separating from the headline count. A 404 is not one thing: Hacker News answers with 146 bytes while Quora answers with 77,465 bytes of HTML and Discourse's marketing site with 45,548 — three sites, three different amounts of page behind a missing file. And three of the six 404s return a zero-byte body, which is the same forum software answering consistently and cheaply.

Why the 403 Wall Is the Real Story

Of the twenty refusals, thirteen returned a body of exactly 295 bytes. It is the same bytes on every host:

<html><body><h1>403 Forbidden</h1>
<p>Your IP (Internet Address) has been temporarily restricted from using this service.</p>
<p>If you feel this is an error, please contact <a href="mailto:[email protected]">[email protected]</a> and mention error code <tt>d/t/44262</tt>.</p>
</body></html>

That body names the forum platform's own support address, so the block is applied at the platform or hosting layer, not chosen forum by forum. It also means reachability is asymmetric in a way that is easy to miss: a coding agent running on a laptop over a residential connection may read a file that a cloud crawler on a datacenter IP is refused. If you self-host one of these forums, test /llms.txt from a second network and a second user agent before you conclude it is live. The same caution applies to the three Cloudflare challenges in the table — a 200-shaped JS interstitial is not a file, as we documented in the delivery-level fix list.

Inside the Two Live Files

Both real files in the sample are small, structurally correct and completely uninterested in the site's own threads. dev.to is a Forem community; forum.openwrt.org runs Discourse and serves what looks like the platform's default template with its own URL substituted — an important detail, because Discourse announced in October 2026 that it is shipping a default llms.txt for all sites, described in its own announcement thread as "very conservative yet powerful".

Host	Bytes	H2	Links	Root-relative	Optional	llms-full.txt
dev.to	2,835	3	7	0	yes	404 (5,077 bytes)
forum.openwrt.org	1,720	3	10	9	yes	404 (24,490 bytes)

dev.to's three sections are API, Policies and Optional, and its seven links are all absolute: the OpenAPI description at dev.to/api/v1/openapi.json, the developer docs, robots.txt, terms, code of conduct, contact and privacy. The OpenWrt file inverts that ratio — one absolute link (the setup page for the Discourse MCP server) and nine root-relative ones (/search, /filter, /latest, /categories, /sitemap.xml, /about, /guidelines, /tos, /privacy). Relative links resolve against the host that served the file, which is fine at the root and ambiguous anywhere else, so read the discovery guide before copying that pattern into a subpath file.

The number that matters most is the one that is missing: neither file lists a single topic, thread, post or member profile. Both describe how to query the community instead of enumerating it — which is the only approach that survives a forum's edit rate.

What a Community Site Should Put in the File

Six blocks, in this order, for a site whose content is written by its users:

One scope sentence in the blockquote — what the community is about and who it serves, in the terms a member would use.
Agent conduct rules as prose — the part no page list replaces. No autonomous account creation, no impersonation, no posting unless a human explicitly asked for that exact action.
Fetch discipline — fetch only the pages the user's request needs, no bulk download, and stop on 429 for the Retry-After interval.
Query endpoints, not threads — search, tag or category filter, latest-activity feed, sitemap.
Policy pages — guidelines, terms, code of conduct, moderation policy, privacy.
Optional — a bulk-access or contact route, and an About page for human context.

If you also expose a write API, name the fields an agent must set. dev.to goes further than any file we have measured in this sample and documents an ai_disclosure_level value with three permitted values — no_ai, some_ai and fully_autonomous — states that reviewing an AI draft does not make it human-authored, and warns that omitting the field records not_disclosed. That is a contract, not a description, and it belongs in an llms.txt precisely because it applies at the moment of the write.

# Example Community Forum

> A support and discussion forum for Example's self-hosted tools, where
> members ask questions, share configurations and report bugs.

This is a community for human discussion. Do not impersonate people or create
accounts autonomously. Only post, edit, react or take any other write action when
a human member explicitly asks you to perform that exact action.

Fetch only the pages needed for the user's request. Do not bulk-download topics or
crawl search result pages. Respect robots.txt and this site's crawler controls: if a
request returns 429, wait for the Retry-After interval before retrying.

When quoting a member's post, keep the attribution and the thread link together,
and never present forum opinions as Example's official position.

## Search and browse
- [Search the forum](/search): Full-text search across public topics.
- [Filter topics](/filter): Query by category, tag or status.
- [Latest activity](/latest): What is being discussed right now.
- [Categories](/categories): Subject areas and their descriptions.
- [Sitemap](/sitemap.xml): Public topic URLs, subject to robots.txt.

## Policies
- [Community guidelines](/guidelines): What is on topic and what is not.
- [Terms of service](/tos): Member obligations and account rules.
- [Code of conduct](/code-of-conduct): Behaviour and enforcement.
- [Moderation policy](/moderation): How flags, locks and removals work.

## Optional
- [API terms](/api-terms): Bulk access, rate limits and attribution.
- [Report an automation problem](/contact): Contact for crawler issues.

What must never appear in that file is easier to state: thread URLs, member profile pages, restricted or invite-only category routes, search-result URLs, moderation queues and anything that resolves behind a login. A thread URL is the fastest-decaying URL on the web — topics get merged, split, renamed, locked and moved — and a profile page is one public line away from exposing personal data. The exclusion list is set out in more detail in what not to include.

The robots.txt Cross-Check Nobody Runs

We fetched /robots.txt from all 34 hosts. Ten answered 200, twenty returned the same refusal as /llms.txt and four never answered. Of the ten readable files, exactly two name any AI crawler token, and the pair makes an instructive contrast:

quora.com names nine AI crawler tokens — GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, CCBot's peer set and others — and the four best-known of them each carry a blanket Disallow: /. Its preamble prohibits using Quora content to train AI models without a contractual licence. Its /llms.txt is a 77,465-byte HTML 404. No file, no permission: consistent.
forum.openwrt.org publishes an llms.txt that tells agents to respect robots.txt, and the robots.txt is 28 lines of blanket Disallow: / — including OAI-SearchBot, ChatGPT-User, Amazonbot and meta-externalagent. So the map is published and the door is closed to four of the crawlers most likely to walk through it.

Both of those are defensible choices. The failure is making them without noticing, which is why the token list in the AI crawler reference is worth reading before you publish either file.

MCP, Delivery and the Three-Command Check

The OpenWrt file does something no other community file in the sample does: instead of relying on a page list, it points at a local Discourse MCP server as the preferred interface, because MCP tools are permission-aware and can act within a member's own rights rather than scrape. That is the read-layer versus action-layer split we covered in llms.txt vs MCP, and for a community it is a better answer than any index: the file says what the community is and which doors exist, the MCP server enforces what may actually be done.

Neither host serves llms-full.txt (both 404, with 5,077 and 24,490 bytes of HTML behind the path), which is the correct call for user-generated content — a full dump of threads would be stale before it finished uploading, and it would duplicate member text that already lives on pages with canonical URLs.

# 1. Status, content type and redirect chain — 403 or a JS challenge means the file is unreachable
curl -sI -L -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://forum.example.com/llms.txt | grep -iE "^(HTTP|content-type|location)"

# 2. Count how many links are root-relative and whether any thread URL crept in
curl -sL https://forum.example.com/llms.txt | grep -oE "\]\([^)]+\)" | wc -l
curl -sL https://forum.example.com/llms.txt | grep -iE "^\- \[.*\]\(/(t|topic|u|users|members)/"

# 3. Confirm every target still answers, resolving relative links against the site root
curl -sL https://forum.example.com/llms.txt | grep -oE "\]\((/[^)]+|https?://[^)]+)\)" | \
  sed -E "s|^\]\((/.*)\)$|https://forum.example.com\1|; s|^\]\((.*)\)$|\1|" | while read u; do
    printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u"
  done | grep -v "^200"

Command two is the one that catches a forum file going wrong: it resolves a non-zero result the moment someone appends topic links to a file that should only describe query endpoints. Draft the structure with the llms.txt generator, then run it through the validator before you deploy, and re-run command one from a second network — on a community site the edge block, not the copy, is what usually decides whether an agent ever reads your file.

Status codes, content types, redirect counts, byte sizes, section counts, link counts and on-page strings referenced were captured from this server on October 17, 2026 using a research user agent with redirect following and a 15-second timeout, including full reads of the two live files and 34 /robots.txt requests. Edge bot rules, generated files and platform policies change without notice, so re-check your own paths rather than treating this table as current configuration for any host named.