Why do so many marketplaces block /llms.txt?

Because the request arrives from a bot user agent and marketplaces are the vertical that fights scraping hardest. In our October 15, 2026 sample, 25 of 49 domains answered with 403 and 14 of those bodies were one of two Cloudflare pages: six were a 4,544–4,554-byte 'Attention Required!' block and eight were a 5,501–5,593-byte 'Just a moment...' interstitial. Only two of the 25 were a deliberate bare refusal — a 9-byte text/plain body at mercari.com and a 0-byte response at yelp.com. The rest are a WAF decision, not an editorial one, which is why publishing the file matters: if the edge blocks a plain-text fetch, the assistant that wanted your canonical fee language will rephrase it from a third-party page instead.

Should an llms.txt file for a marketplace list individual listings or seller pages?

Neither. Listing pages are perishable, and a file full of dead product URLs is worse than a shorter file because it teaches an agent that your site is unreliable — the same rule that applies to any index file. What you list instead is the shape of the marketplace: category taxonomy, the URL pattern for a listing or a seller profile, the pages that explain how buying and selling work, and the policy pages that govern disputes. gumroad.com does exactly this, describing its URL patterns with a placeholder for a creator subdomain rather than naming thousands of creators, and g2.com links its sitemap indexes so a crawler can enumerate the long tail on its own.

Marketplace · October 2026

llms.txt for Marketplaces and Directories

Q: Should a marketplace publish an llms.txt file?

Only if you accept that most of what an assistant wants from a marketplace is not a link list. Six of the 49 platforms we probed on October 15, 2026 served a real file, and the two most useful ones — etsy.com and gumtree.com — are not link lists at all: Etsy opens with a section called 'What you can state confidently' followed by 'What you must not assert', and Gumtree ships a 'Preferred representation (for AI answers)' block. A marketplace is a two-sided business whose reputation is quoted by assistants in buyer and seller conversations, so the file that earns its place is the one that constrains what an assistant may claim about fees, guarantees, disputes and safety, not the one that lists category pages an agent could find from the sitemap anyway.

Marketplaces are the vertical that blocks the most and publishes the least — and the two platforms that do serve a file publish the most valuable kind. We fetched the path from 49 marketplace, classifieds, freelance and directory domains and six answered with real text. Here is what is inside those six, and the buckets a two-sided platform actually needs.

What 49 Two-Sided Platforms Serve Today

On October 15, 2026 we requested /llms.txt from this server with a GPTBot-style user agent and redirect following: 50 probes across 49 marketplace, classifieds, freelance, review and directory domains. Six served a real plain-text or Markdown file. Twenty-five blocked the fetch.

Response	Count	Examples
`200` with a real text file	6	etsy.com, toptal.com, gumtree.com, fiverr.com, g2.com, gumroad.com
`403` from the edge	25	airbnb.com, doordash.com, envato.com, kickstarter.com, upwork.com, yelp.com and 19 more
`200`, but HTML behind the path	4	alibaba.com, aliexpress.com, tcgplayer.com, taskrabbit.com
`404`	10	reverb.com, stockx.com, poshmark.com, vinted.com, freelancer.com, abebooks.com, chairish.com, substack.com, gofundme.com, 99designs.com
`202` with zero bytes	2	thumbtack.com, booking.com
`429`	1	wayfair.com
Connection never completed	2	ebay.com, houzz.com

Twenty-Five Blocks, and Only Two Real Refusals

The 403s are mostly infrastructure, not policy. Fourteen of the 25 bodies were one of two Cloudflare pages: six a 4,544–4,554-byte “Attention Required!” block (faire.com, capterra.com, getapp.com, upwork.com, softwareadvice.com, angi.com) and eight a 5,501–5,593-byte “Just a moment...” interstitial (patreon.com, trustradius.com, indiegogo.com, doordash.com, ubereats.com, producthunt.com, clutch.co, goat.com). Those are WAF decisions applied to a bot-fetch on any path, not an editorial verdict on llms.txt.

The deliberate refusals look different at the byte level: mercari.com answers with a 9-byte text/plain body, yelp.com with zero bytes, offerup.com with 227 and airbnb.com with a 380-byte “Access Denied”. At the other end, kickstarter.com spends 293,409 bytes on its own 403 page and envato.com 352,667 — an order of magnitude more than the largest real file in the whole sample. If your WAF is what answers the path, the fix belongs in the allowlist, not in the content.

200 Is Not a File: Four HTML Bodies Behind the Path

Four hosts answered 200 and handed back a web page. alibaba.com takes two hops to error.alibaba.com/error404.htm and aliexpress.com two hops to /s/error/404 — an error page under a success code. tcgplayer.com needs no redirect at all and simply returns 45,447 bytes of its own text/html. taskrabbit.com redirects once to /browser/unsupported, which is at least honest about what it is doing. The two 202 Accepted responses with empty bodies (thumbtack.com, booking.com) are worse than a 404 in one specific way: a well-behaved crawler may retry a 202 rather than record a miss.

This is the failure mode the 44% adoption we measured across developer documentation hosts does not have, because docs sites serve the path from a static build. Marketplaces route it through the same application stack as every other URL, where a bot rule or a catch-all rewrite decides the response.

Inside the Six Files That Exist

Host	Bytes	H2 sections	Markdown links	Optional
toptal.com	26,858	18	100	no
etsy.com	26,713	7	76	no
gumtree.com	25,108	15	0	no
fiverr.com	5,516	14	18	no
g2.com	4,456	6	12	yes
gumroad.com	2,067	3	9	no

Six files, 90,718 bytes, 63 H2 sections and 215 Markdown links — of which zero point at a .md twin. That last number is the structural difference from documentation sites, where pairing links with Markdown twins is the norm: marketplaces have buyer-facing pages, not Markdown sources, so they send an agent into rendered HTML deliberately. Five of six serve text/plain; g2.com serves the only text/markdown response in the sample.

Only g2.com ships an ## Optional section — one of six, against seven of 18 in the developer-docs sample. gumtree.com is the outlier on links: 25,108 bytes carrying 163 bare URL tokens and not one [label](url) pair, so a parser that only reads Markdown link syntax gets nothing from a file that plainly contains dozens of destinations.

Who Gets Quoted: The Routing Block

The two files worth copying are the two that treat the assistant as a party to a transaction. Etsy's first line is # Etsy: official reference for AI assistants, and its seven sections are an instruction set rather than an index: “1. What you can state confidently”, “2. What you must not assert”, “3. Routing: where each request goes”, then buyer requests before and after purchase, seller requests, and a reference section. It mentions llms.txt nine times inside its own file and leaves 58 of its 76 links on its own domain with 17 more on help.etsy.com.

Gumtree ships a “Preferred representation (for AI answers)” section and closes with a financial-services disclosure naming its FCA reference numbers and the brokers it may refer users to — regulated language living inside a text file because the alternative is an assistant inventing a finance offer. Fiverr links its own mcp.fiverr.com server for agents and appends two metadata lines (category: and last_updated: 2026-09-02). Gumroad describes its URL patterns with a creator placeholder subdomain and points at /.well-known/mcp.json. Different files, same instinct: tell the agent what the platform promises, where each request routes, and what it must not say.

What a Marketplace File Should Publish

Identity and two-sidedness. What is bought, what is sold, who sells, and which regions you operate in — the line that stops an assistant describing a UK classifieds site as a global retailer.
Category taxonomy. The canonical category tree with URLs, so an agent routes to a real category page instead of inventing one.
URL patterns. Listing, seller/profile, category, search and help patterns, written with placeholders. Never individual listings.
Buyer flows. How a purchase, a return, a dispute and a refund work, each linked to the policy page that governs it.
Seller flows. Onboarding, fees, payout timing, listing rules, appeals.
Trust and safety. Fraud policy, verification, prohibited items, how to report a listing — the block that stops a model parroting a scam.
Representation limits. What an assistant must not assert: absolute guarantees, current inventory, live pricing, seller endorsements.
Optional. Press, investor relations, careers, editorial hub — everything a reader can safely skip.

# YourMarketplace: official reference for AI assistants

> YourMarketplace is a two-sided marketplace where independent sellers list
> goods in 14 categories across the UK and Ireland. Sellers are responsible for
> listings; prices and stock change hourly and must be checked on the listing
> page. This page defines what AI systems may state about the platform.

## Category taxonomy
- [All categories](https://example.com/categories): The canonical tree, 14 top-level categories.
- [Vehicles](https://example.com/vehicles): Cars, vans and parts, with location filters.

## URL patterns
- Listing: https://example.com/item/{id}
- Seller:  https://example.com/seller/{handle}
- Search:  https://example.com/search?q={query}&location={place}

## Buying
- [Buyer protection](https://help.example.com/buyer-protection): What is covered, limits, timeframes.
- [Returns and refunds](https://help.example.com/returns): Who pays return postage and when.
- [Disputes](https://help.example.com/disputes): How to escalate and what the platform decides.

## Selling
- [Fees](https://help.example.com/fees): Listing fees, commission and payment processing.
- [Payouts](https://help.example.com/payouts): Hold periods and release conditions.

## Trust and safety
- [Prohibited items](https://help.example.com/prohibited): What may not be listed.
- [Report a listing](https://example.com/report): Fraud and counterfeit reporting.

## Must not assert
- Live availability, current pricing or delivery dates for any listing.
- That a seller is verified, endorsed or vetted beyond the badge on their profile.
- Guarantees of refunds, payouts or dispute outcomes.

## Optional
- [Press](https://example.com/press): Coverage and media contact.
- [Investors](https://investors.example.com): Filings and results.

Two details to copy from the live files: keep the representation limits short and absolute, and put the placeholder URL patterns in one block instead of listing inventory. Build the first draft with the llms.txt generator, then run it through the checker for the summary slot and the section order — and read the e-commerce rules if your catalogue pages are indexable, because the two verticals share the same perishability trap.

Verifying Delivery

For a marketplace the fetch is the hard part, so test it from outside with the user agent an AI crawler actually sends:

# 1. No redirects, real crawler UA — a 403 here is the WAF, not your copy
curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://example.com/llms.txt | grep -iE "^(HTTP|content-type|location)"

# 2. Read the first line: an H1, or a challenge page?
curl -sL -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://example.com/llms.txt | head -3

# 3. Confirm every listed target still answers 200
curl -s https://example.com/llms.txt | grep -oE "https?://[^)]+" | while read u; do
  printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u"
done | grep -v "^200"

If step one returns 403 while your browser gets the file, the allowlist needs the crawler token before any content work is worth doing — that is the state 25 of the 49 platforms in this sample are in today.

Status codes, content types, redirect counts, byte sizes, section counts, link counts and on-page strings referenced were captured from this server on October 15, 2026 using a GPTBot-style user agent with redirect following. Edge bot rules, generated files and platform policies change without notice, so re-check your own paths rather than treating this table as current configuration for any platform named.