Should a real estate site list individual property listings in its llms.txt file?

No. A portal with 200,000 active listings would produce a file that changes every hour, cannot be diffed, and is stale the moment a property goes under contract. Document the URL pattern instead — one line for the search page, one line each for the market and the sold-price history — and let the agent fetch live inventory from those pages. The only property-sized files worth publishing are the ones behind a stable path, such as a city hub or a neighbourhood guide. The single working file in our October 8, 2026 check of 25 property domains follows exactly this rule: 78 links, 11 KB, not one individual home.

Why do so many real estate sites return 403 for /llms.txt?

Because property portals run the heaviest bot management of any consumer vertical — inventory scraping is a real attack — and that protection cannot tell an AI agent from a scraper. In our October 8, 2026 check, fourteen of twenty-five domains blocked the fetch at the edge, including Zillow, Trulia, Apartments.com, Homes.com, Compass, Zoopla, Idealista, LoopNet, Crexi, RentCafe and CBRE. We re-tested two of them with a GPTBot user agent, a ClaudeBot user agent and a Chrome user agent from the same connection: all three got 403, so the block was not user-agent based. Allow the /llms.txt path in the WAF explicitly or the file will never be read.

Do I need llms.txt if my robots.txt already blocks AI crawlers?

Then you have already made the decision, and the two files should agree. Rightmove's robots.txt disallows GPTBot sitewide with a narrow Allow for its mortgage pages, and Zillow hands a long list of AI and research agents — AI2Bot, Amazonbot, Applebot-Extended, meta-externalagent and others — a blanket Disallow. Publishing an llms.txt on top of that would be a contradiction an agent can read. Run the reverse check too: every path you list in the file must be crawlable per robots.txt. Opendoor does this correctly, disallowing /customers/, /offers/ and its seller dashboards in robots.txt while its llms.txt links only public calculators, market pages and developer resources.

Real Estate · October 2026

llms.txt for Real Estate Portals and Brokerages

Property is the vertical where the assistant's question is a listing — and a listing is the one thing your file must never contain. Here is what 25 real estate domains actually serve today, the seven buckets a portal or brokerage needs, and the robots.txt cross-check that catches a self-contradicting file.

What 25 Property Domains Serve Today

We fetched /llms.txt from this server on October 8, 2026 with a generic user agent, following redirects, across national portals, agent-brand networks, commercial brokerages, rental platforms and proptech.

Response	Count	Examples
`200`, `text/plain` — a real file	1	opendoor.com (11.0 KB, 78 links, 10 sections)
Blocked at the edge	14	zillow.com, trulia.com, apartments.com, homes.com, compass.com, zoopla.co.uk, idealista.com, century21.com, cbre.com, cushmanwakefield.com, domain.com.au, rentcafe.com, loopnet.com, crexi.com
`404`	7	redfin.com, realtor.com, immobilienscout24.de, savills.com, realestate.com.au, foxtons.co.uk, purplebricks.com
`200`, but no file behind it	2	rightmove.co.uk (147 KB HTML "page not found"), jll.com (redirected to a locale landing page, 976 KB HTML)
`405` Method Not Allowed	1	remax.com

Read the shape: fourteen of twenty-five hosts never let us see the path at all, so we cannot know whether a file exists behind the block. That is worse than a 404, because a 404 at least tells the agent to move on. A 200 with a 147 KB error page behind it (Rightmove) is worse still — a client that trusts the status code will try to parse an HTML shell as Markdown. Confirm the body, not the status line.

The Block Is Not User-Agent Based

The obvious next question is whether these blocks are aimed at AI crawlers specifically. We re-tested two of the blocked hosts from the same connection with three user agents: a GPTBot string, a ClaudeBot string, and a current desktop Chrome string. Zillow and Apartments.com answered 403 to all three, and two other hosts returned a byte-identical 5,452-byte block page. Whatever is rejecting the request is deciding on the connection and the path, not on the name the client calls itself — which means the fix is a WAF allowlist entry, not a robots.txt rewrite.

Meanwhile the policy layer says something else entirely. Rightmove's robots.txt disallows GPTBot sitewide with a narrow Allow: /mortgages/ and Allow: /mortgages/mortgage-in-principle — the same carve-out many sites give the one commercial funnel they want indexed. Zillow groups a long list of AI and research agents (AI2Bot, Amazonbot, Applebot-Extended, meta-externalagent and others) under Disallow: /, and Redfin names FireCrawlAgent. For these companies the absent file is a decision, not an oversight: robots.txt for AI crawlers is where their answer lives.

The Seven Buckets a Property File Needs

Identity and coverage. Brand, primary domain, and the markets you actually operate in — one metro list, not a link per city page. This is the line that keeps an assistant from recommending you for a city where you have no inventory.
Search and browse entry points. For-sale search, for-rent search, new builds, and sold-price history as four separate links. These are the durable URLs; the URLs below them are not.
Valuation and market data. Home value estimator, mortgage calculator, affordability tools, and any public price index or market report. Opendoor puts its calculators in their own H2 section and links the estimator first.
Transaction flows. How to sell, how to buy, how to make an offer, financing and title partners. Process pages, never the gated dashboards behind them.
For agents and partners. Agent directory or find-an-agent, join or franchise pages, referral and partner programs — a genuinely distinct audience in this vertical, and the one bucket most brokerages do publish in some form.
Policy and compliance. Terms, privacy, accessibility statement, fair-housing statement, and your MLS or IDX attribution page. An assistant quoting a listing site is a compliance surface, so hand it the sources it should cite.
Machine-readable targets. Sitemap index, listings sitemap, any public PDF or OpenAPI document, and the developer portal. Opendoor ends its file with an ## Optional section holding exactly this — developer resources, an OpenAPI document, robots.txt and two sitemaps.

A Copyable Skeleton

Eleven stable links beat two thousand listing URLs. Start here, then add sections only where your site genuinely has a durable hub for them:

# Northline Realty

> Northline Realty is a residential brokerage operating in Austin, Denver and
> Seattle. It publishes for-sale and rental search, neighbourhood guides,
> market reports and a home valuation tool for those three metros.

## Search
- [Homes for sale](https://www.example.com/homes-for-sale): Search active listings by city and ZIP.
- [Homes for rent](https://www.example.com/homes-for-rent): Rental inventory and lease terms.
- [Sold home prices](https://www.example.com/sold): Closed sale prices by neighbourhood.

## Market data
- [Home valuation](https://www.example.com/home-value): Estimate a property's current value.
- [Monthly market report](https://www.example.com/market-report): Median price and days-on-market by metro.

## Working with us
- [How selling works](https://www.example.com/sell): Listing process, timeline and fees.
- [Find an agent](https://www.example.com/agents): Licensed agents by metro.
- [Join Northline](https://www.example.com/careers/agents): Recruitment for licensed agents.

## Policies
- [Fair housing](https://www.example.com/fair-housing): Equal housing opportunity statement.
- [Terms](https://www.example.com/terms) · [Privacy](https://www.example.com/privacy) · [Accessibility](https://www.example.com/accessibility)

## Optional
- [Sitemap index](https://www.example.com/sitemap.xml)
- [Listings sitemap](https://www.example.com/listings.xml)

What Must Never Appear

Individual listings, street addresses and unit numbers — including pre-market and coming-soon inventory, which is a confidentiality problem rather than a freshness one.
Live prices, price-change history per home, or "from $X" figures. Every number gets a link.
Anything you licence rather than own: MLS and IDX feeds, syndicated photos, third-party valuation model output.
Gated account paths — offer portals, seller dashboards, buyer financing status, saved-search pages. Opendoor disallows /customers/, /offers/ and /buyer/offers/ in robots.txt and keeps every one of them out of the file; do the same consistency pass.
Client records or anything from a transaction: names, offer amounts, inspection reports, mortgage pre-approvals.
Agent mobile numbers and personal inboxes. Point at the agent profile page instead.

The mechanics are the same as any commercial site — the buckets and the leak list are covered in what not to include in llms.txt, and the listing-pattern argument is the one we made for e-commerce stores, where a SKU catalogue plays the role of an inventory feed.

Size, Subpaths and the Delivery Check

A three-metro brokerage needs 5–12 KB. A national portal with franchise groups is the case for subpath files rather than one large root file: keep the root file as an index of sections, and expand the parts that have their own stable home, such as a commercial division or an international desk. The byte-to-token arithmetic and the split-versus-delete tradeoff are in how long should an llms.txt be.

Then run three commands from any machine, using a real browser user agent:

# 1. Status, content type, byte count — a 403 or a text/html here ends the exercise
curl -sIL -A "Mozilla/5.0" https://www.example.com/llms.txt | grep -Ei "HTTP/|content-type|content-length"

# 2. Confirm the body is Markdown, not a WAF page or an HTML 404 shell
curl -sL -A "Mozilla/5.0" https://www.example.com/llms.txt | head -5

# 3. Cross-check: disallowed paths must not be listed in the file
comm -12 <(curl -sL https://www.example.com/robots.txt | awk '/^Disallow:/ {print $2}' | sort -u) \
         <(curl -sL https://www.example.com/llms.txt | grep -oE "https://[^) ]+" | sed 's#https://[^/]*##' | sort -u)

Step 3 is the one real estate sites get wrong: a file that links a path your own robots.txt disallows is a contradiction a careful agent will notice. Paste the file into the validator for structure and link checks first, or build the skeleton from your sitemap with the generator and then delete every listing-level URL it imports. Serving it correctly once it exists is a separate problem — content type, encoding and caching are covered in the delivery guide.

The Short Version

One of twenty-five property domains publishes a real file today, and half the rest will not even answer the path. That is a thin field, and the entry bar is a small, curated index: identity and coverage, four search entry points, calculators and market data, process pages, agent recruitment, compliance, and a machine-readable tail of sitemaps. No listings, no prices, no gated paths — and one Allow rule in the WAF so the agents you wrote it for can actually fetch it.