Where it came from

Jeremy Howard proposed llms.txt in September 2024. The problem it addresses is practical. Web pages are built for people: navigation, cookie banners, scripts and layout surround the text a model actually needs. A model working within a limited context does better with a clean summary and direct links to Markdown or plain-text versions of key pages.

robots.txt tells crawlers what they may fetch. sitemap.xml lists every URL for indexing. llms.txt does a different job: it picks out the handful of pages that best explain the site.

The format

The specification keeps the structure simple:

# Example Analytics

> Example Analytics is an open-source web analytics tool that runs without cookies.

Example Analytics is self-hosted and stores data in SQLite.

## Docs

- [Installation](https://example.com/docs/install.md): requirements and a five-minute setup
- [Tracking API](https://example.com/docs/api.md): every endpoint with request examples
- [Privacy model](https://example.com/docs/privacy.md): what is stored and for how long

## Optional

- [Changelog](https://example.com/changelog.md)

The parts, in order:

  1. An H1 with the name of the project or site. This is the only required element.
  2. A blockquote with a short summary of the key information.
  3. Optional paragraphs or lists with context a model should know. These may not contain headings.
  4. H2 sections, each a list of Markdown links, optionally followed by a colon and a note on what the page contains.
  5. A section called "Optional" for links a tool can skip when space is short.

The proposal also suggests offering a clean Markdown version of important pages at the same URL with .md added. Some documentation sites go further and publish llms-full.txt, a single file with the full text of their docs. That is a convention outside the original proposal.

Who reads it today

Claims about llms.txt often run ahead of the evidence, so it is worth being precise.

  • Google Search does not use it. Google's guide to its generative AI features says you don't need AI text files or Markdown versions to appear, that Google Search ignores them, and that having one neither helps nor harms your visibility.
  • Other AI search engines. ChatGPT search, Perplexity, Claude and Copilot have not said they use llms.txt to discover or rank sources. Your visibility in AI answers still depends on being crawlable, indexed and worth citing.
  • AI agents. Chrome's Lighthouse has an Agentic Browsing category that includes an llms.txt audit. It flags a server error when the file is requested and marks a missing file as not applicable, because the file is optional. The audit treats llms.txt as a help for agents moving around a site, not as a search signal.
  • Documentation tools. Many developer-tool companies publish llms.txt and llms-full.txt so that coding assistants can load their documentation in one go. This is where the file is most clearly useful.

You can measure this on your own site. Server logs, or the Crawlers view in this site's admin console, show which bots request /llms.txt. If no AI bot fetches it for months, that tells you how much weight to put on it.

When it is worth adding

Add one if:

  • you publish documentation, an API reference, courses or other reference material that people load into AI tools;
  • you can keep it accurate as the site changes;
  • you already produce clean Markdown or text versions of key pages.

Skip it, or keep it minimal, if your site is small, mostly visual or rarely used as reference material. A stale llms.txt that points to moved pages helps no one.

Writing a good one

  • Lead with the pages that answer the questions people actually ask about your subject, not your company history.
  • Write a one-line note for every link. That note is what a model uses to decide whether to open the page.
  • Link to clean versions where you have them: Markdown, plain text or a print view.
  • Keep the main list short. If it grows past a few dozen links, move the rest under "Optional".
  • Serve it as plain text or Markdown with a 200 status, and do not block it in robots.txt. A server error is what Lighthouse flags.
  • Update it when you publish, move or retire a key page.

This site's own file is at /llms.txt and lists the four free books and every guide on the blog.

Primary sources and further reading

Book-derived explanations on this page are editorial synthesis. Platform-specific claims link to the platform's own current documentation.

Frequently asked questions

Is llms.txt a standard?

It is a community proposal with a published specification at llmstxt.org, not a standard adopted by a standards body or by the large AI search providers.

Does llms.txt replace robots.txt or the sitemap?

No. robots.txt controls access and the sitemap lists URLs for indexing. llms.txt is an optional, curated guide on top of both.

Can llms.txt block AI training?

No. Use robots.txt rules for the training crawlers, such as GPTBot, ClaudeBot and Google-Extended, to control training use.