All three files live at your site's root — /robots.txt, /sitemap.xml, /llms.txt — which makes it easy to assume they overlap. They don't. Each one answers a different question for a different kind of visitor.
| File | Answers | Audience |
|---|---|---|
robots.txt | What are you allowed to crawl? | Any bot, before it crawls anything |
sitemap.xml | What pages exist? | Search engine indexers |
llms.txt | What does this site mean? | LLMs with limited context |
The oldest of the three, dating back to the mid-90s. It's purely about access control — which paths a crawler is allowed or disallowed from fetching. It says nothing about what's actually on the page, and most bots treat it as optional guidance rather than hard enforcement.
An XML file enumerating every URL on your site, sometimes with metadata like last-modified dates. It's built for exhaustiveness — a search engine indexer wants to find every page eventually. But it's a flat list: a sitemap can't tell a crawler which five pages actually matter most, or summarize what your product does.
This is the newest of the three, and it inverts the sitemap's approach on purpose. Instead of listing everything, it curates: a short summary of what the site is, plus links to the pages that actually matter, grouped and described in plain markdown. That's a deliberate design choice — language models work with limited context windows, so a file optimized for completeness is actually worse for them than one optimized for relevance. See the spec at llmstxt.org for the full format.
Yes — they're complementary, not competing. robots.txt controls access, sitemap.xml helps traditional search indexing, and llms.txt helps AI assistants describe your site accurately instead of guessing from a noisy HTML scrape. Most sites already have the first two. Almost none have the third yet — which is exactly why adding one now is worth the ten minutes it takes.
Generate your llms.txt in under a minute
Try it free →