If you've noticed AI assistants like ChatGPT, Claude, or Perplexity citing some websites accurately and completely botching others, part of the difference often comes down to a small text file: llms.txt.
Search engines have crawled the web with fairly consistent tooling for decades — HTML, sitemaps, structured data. Large language models are different: they have limited context windows, and a typical webpage is full of navigation menus, ads, scripts, and layout markup that isn't actually the content the model needs. When an AI assistant tries to answer a question about your product using your live site, it's working with a lot of noise.
llms.txt is a proposed convention (see the spec at llmstxt.org) for giving AI systems a clean, curated entry point: a markdown file at your site's root that summarizes what the site is and links to its most important pages, without the HTML noise.
# Your Company > One-sentence summary of what you do. ## Pages - [Docs](https://yoursite.com/docs): Your documentation - [Pricing](https://yoursite.com/pricing): Plans and pricing - [API Reference](https://yoursite.com/api): Full API docs
It's deliberately simple: an H1 title, a blockquote summary, and markdown links grouped into sections. No JSON schema, no required fields beyond the title — the whole point is that it's cheap to produce and cheap for a model to parse.
sitemap.xml and semantic HTML, an accurate llms.txt increases the odds an AI assistant describes your product correctly instead of guessing from a scraped, noisy page.llms.txt at all. That's an advantage for whoever adds one now.You can write one by hand for a small site — it's just markdown. For anything with more than a handful of pages, doing it manually means keeping it in sync every time content changes, which is where it tends to go stale.
That's the specific problem AgentSEO solves: paste your URL, and it crawls your site and generates a properly formatted llms.txt for you — a free preview covers 5 pages, and $9 gets a full multi-page crawl you can drop straight into your site root.
Want an llms.txt for your own site?
Generate one free →Next: llms.txt vs robots.txt vs sitemap.xml — what's the difference?