Should a job board publish an llms.txt file?

Yes, but not as a list of jobs. 13 of the 31 employment hostnames we probed on October 30, 2026 served a real file, and the ones an agent can use are the shortest: ZipRecruiter spends 5,956 bytes documenting URL patterns and permission to build them, while SmartRecruiters spends 643,513 bytes dumping every page on the marketing site. Live postings belong in an API or an MCP endpoint, because a curated index of individual jobs is stale within a day and teaches an assistant that your links rot. What belongs in the file is the shape of the platform: category taxonomy, the URL pattern for a job, a company page and a salary page, plus the policy pages that govern applications and pay.

Why do so many employment sites block /llms.txt?

Because the request arrives from a bot user agent and job boards are the vertical that fights scraping hardest — the listings are the product and competitors scrape them. In our October 30, 2026 sample, 7 of 31 hostnames answered 403, and the shapes vary: reed.co.uk returns a Cloudflare challenge flagged with cf-mitigated, indeed.com a 27,441-byte HTML block, seek.com.au 49,604 bytes of its own HTML page, and cv-library.co.uk 121,080 bytes. A block is an infrastructure decision that also hides your canonical fee, application and privacy language from the models answering candidate questions, which is the reason to allowlist the path rather than the reason to skip the file.

Should an llms.txt list individual job postings?

No. Not one of the 13 live files in our sample pointed at a single job posting URL, and that is the correct outcome rather than an oversight. Postings expire in weeks, they are personalised or geo-gated, and a client that fetches 2,000 of them burns its context budget on pages that will 404 tomorrow. Document the URL pattern instead, point at the search and category surfaces that survive, and expose live inventory through an API or MCP tool — ZipRecruiter does both, declaring an Accept: text/markdown twin for job detail pages and a search_jobs MCP tool that requires no authentication.

Recruiting · October 2026

llms.txt for Job Boards and Recruiting Sites

Employment is the vertical where content expires fastest and the incentives are most conflicted: the listings are the product, so the edge fights bots while the marketing site publishes a file. We requested the path from 31 job boards, ATS vendors and recruiting platforms. Thirteen answered with real text, and two of those hold 92% of the bytes.

What 31 Employment Hosts Serve Today

On October 30, 2026 we requested /llms.txt from 31 hostnames covering job boards, applicant tracking systems and recruiting-software vendors, with redirect following and a browser-compatible user agent.

Response	Count	Hosts
`200` with a real text file	13	greenhouse.com, ziprecruiter.com, monster.com, ashbyhq.com, smartrecruiters.com, icims.com, bamboohr.com, jobvite.com, workable.com, teamtailor.com, bayt.com, remoteok.com, himalayas.app
`200`, but HTML behind the path	1	dice.com — 78,734 bytes of app shell
`403` from the edge	7	indeed.com, glassdoor.com, reed.co.uk, seek.com.au, cv-library.co.uk, stepstone.de, totaljobs.com
`404`	7	lever.co, workday.com, wellfound.com, weworkremotely.com, recruitee.com, jobs.lever.co, job-boards.greenhouse.io
`429`	1	personio.com
Connection never completed	2	naukri.com, remote.co — both an HTTP/2 `INTERNAL_ERROR` mid-stream

The misses are not uniform: 404 covers both a zero-byte answer at weworkremotely.com and a 296,328-byte HTML page at jobs.lever.co, the ATS host thousands of employer careers pages live on. The 403s split two ways — Cloudflare challenges at reed.co.uk (flagged cf-mitigated: challenge) and cv-library.co.uk, and Akamai Access Denied blocks of 378 and 383 bytes at stepstone.de and totaljobs.com: WAF rules, not an editorial verdict.

The Redirect Layer: 20 of 31 Answer With a 3xx First

Twenty hostnames responded to the first request with a redirect, usually apex to www. Following it is the only way to measure this vertical, and one chain is broken. teamtailor.com/llms.txt answers 301 to https://www.teamtailor.com/llms — the .txt is gone — and that path returns 404 with 67,366 bytes of HTML. The file exists: www.teamtailor.com/llms.txt serves 6,777 bytes to the same client. The URL a crawler discovers from your documentation dead-ends, while the URL it guesses works.

The rest are less dramatic. greenhouse.io/llms.txt redirects correctly across a domain change to www.greenhouse.com/llms.txt. lever.co answers 308 with a 15-byte text/plain body reading “Redirecting...” and its www target answers 404 with 47,515 bytes.

Two Files Hold 92% of the Bytes, and One Points at a Dev Server

The thirteen live files total 1,325,776 bytes, median 6,777 — but the mean is 101,983 because two outliers carry the rest: smartrecruiters.com at 643,513 bytes (roughly 160,878 tokens at four bytes per token) and jobvite.com at 580,156 (roughly 145,039). Both are plugin output whose first line reads “Generated by All in One SEO Pro”, and both open with a UTF-8 BOM.

SmartRecruiters' file is a site dump: 2,125 Markdown links across 25 sections, 852 into /resources, 310 into /news, 230 into /hiring-success, plus 391 into /fr and /de copies of the same pages. Jobvite's has a defect the byte count hides — all 1,928 links point at 3mw-dev-jobvite.pantheonsite.io, a Pantheon development environment, including the sitemap link. The dev host answers: a listed 2022 blog post returned 200 with 216,068 bytes, the root 200 with 291,477. Every agent that follows the production file lands on a staging copy with posts dated 2020 and 2022.

The File Worth Copying: ZipRecruiter's URL-Pattern Index

The 5,956-byte file is the one to read. It publishes ten sections and states that its URL templates are not a generator, then labels each section: Derivable means safe to build only from a value that came back in your own results, Curated means the slug is editorial and guessing will 404. That vocabulary answers a question agents otherwise answer by brute force, and the file follows it with an explicit ban on speculative fetching — “do not build URLs from user input and fetch them speculatively”.

Underneath sit five job-search patterns (title, title plus state, title plus city, state-only, near-me), five salary patterns, two or three worked examples each, and three declarations most boards skip: a text/markdown twin for job and directory pages when the request carries Accept: text/markdown — the layer described in Markdown for Agents vs llms.txt — plus a streamable-HTTP MCP endpoint with one read-only, unauthenticated search_jobs tool. The short file describes behaviour; the 643,513-byte one describes a sitemap.

What a Job Board File Should Contain

Six buckets, in this order. The first two stop an assistant inventing URLs; the rest stop it inventing policy.

Identity and scope. Who hires, who applies, which markets and languages you cover, and whether you are a board or an ATS embedded on other people's domains — the reason jobs.lever.co having no file is a real gap.
URL patterns, not postings. Job, company, category, location, salary and search, each with one placeholder and one example.
Candidate policy. Application rules, retention, resume handling, what an assistant may not do on a candidate's behalf.
Employer policy. Posting rules, pricing model, prohibited listings, pay transparency per market.
Live-inventory route. Where current jobs live — API, feed or MCP — plus the sentence saying this file is not inventory.
Optional. Blog, salary reports, press, the company's own careers page.

# ExampleBoard: official reference for AI assistants

> ExampleBoard is a job board for technology roles in the UK and Ireland,
> used by 4,200 employers. Listings change hourly; this file defines URL
> patterns and policy, not live inventory. For current postings use the API.

## URL patterns
- Job detail:  https://example.com/jobs/{slug}
- Company:     https://example.com/companies/{company}
- Category:    https://example.com/c/{category}
- Location:    https://example.com/l/{city}
- Salary data: https://example.com/salaries/{role}
- Search:      https://example.com/search?q={query}&where={place}

## Live inventory
- [Job API](https://api.example.com/v1/jobs): JSON feed, paginated, 60 req/min.
- This llms.txt does not list individual postings and must not be read as one.

## For candidates
- [How applications work](https://example.com/how-it-works): One profile, direct apply.
- [Data and privacy](https://example.com/privacy/candidates): What is stored and for how long.
- [Report a listing](https://example.com/report): Scams, misleading pay, duplicate posts.

## For employers
- [Posting rules](https://example.com/employers/rules): Pay transparency required in the UK.
- [Pricing](https://example.com/employers/pricing): Per-post and subscription.

## Optional
- [Salary reports](https://example.com/reports): Annual benchmarks by role and region.

Draft it with the llms.txt generator, check the summary slot and section order in the checker, and if your board also runs a two-sided marketplace the routing rules in llms.txt for marketplaces stack on top of this structure.

Six Defects That Repeat Across the 13 Files

No H1 at all. greenhouse.com spends 21,000 bytes and 22 H2 sections on a fact sheet opening with ## Official Information About Greenhouse — no H1, no summary, not one Markdown link, so an agent gets facts it cannot follow anywhere.
A BOM before line one. smartrecruiters.com and jobvite.com both start with one, which pushed the tooling line ahead of the H1.
Almost no .md twins. Of 4,375 Markdown links across the thirteen files, exactly one targets a .md URL (ashbyhq.com).
Optional is rare. Only ashbyhq.com and teamtailor.com use it, both last — the correct position for skippable material.
Two summaries. monster.com opens with two blockquote paragraphs, so a client lifting only the first loses the employer half of the description.
Non-spec preamble. workable.com starts with three YAML-looking keys (title, description, tags) before its H1.

Check your own delivery in four commands — the redirect chain first, because that is where this vertical fails most:

# 1. Follow the chain: where does the apex end up?
curl -sIL https://example.com/llms.txt | grep -iE "^(HTTP|location)"

# 2. Does the final target return text, or HTML?
curl -sL https://example.com/llms.txt | head -3

# 3. Is the status 403 to a crawler but 200 to your browser?
curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://example.com/llms.txt | grep -iE "^(HTTP|cf-mitigated)"

# 4. Confirm every listed target still answers 200
curl -s https://example.com/llms.txt | grep -oE "https://[^)]+" | while read u; do
  printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u"
done | grep -v "^200"

Status codes, content types, redirect targets, byte sizes, section counts, link counts and on-page strings referenced were captured from this server on October 30, 2026 with redirect following. Edge bot rules, generated files and platform policies change without notice, so re-check your own paths rather than treating this table as current configuration for any platform named.