llms.txt: What It Is and Whether It Does Anything

llms.txt is a proposed convention: a markdown file at the root of a domain that gives large language models a curated map of the site's most useful content. It is a proposal, not a standard, and no major AI system has publicly committed to reading it as a ranking or retrieval input.

That is the honest answer, and it is the one most coverage of the file omits. The file costs almost nothing to publish and carries a small option value if adoption arrives. It is not a lever, and treating it as one displaces work that does affect retrieval.

§ 01Key points

• llms.txt is a markdown index at /llms.txt listing a site's key pages with short descriptions.
• It was proposed by Jeremy Howard in 2024 as a way to fit a site's essentials inside a model's context window.
• No major provider has confirmed using it for retrieval or citation. Adoption claims should be treated as unverified.
• Publishing it is cheap and low-risk. Expecting visibility gains from it is not supported.

§ 02What the file actually is

The format is deliberately plain. An H1 with the site or project name, an optional blockquote summary, then H2 sections containing markdown link lists, each link followed by a short description. A companion convention, llms-full.txt, contains the expanded content itself rather than links.

A minimal example:

# Example Corp

> Example Corp builds inventory forecasting software for independent retailers.

## Docs
- [Getting started](https://example.com/docs/start): Install, authenticate, first forecast.
- [API reference](https://example.com/docs/api): Endpoints, auth, rate limits.

## Company
- [About](https://example.com/about): Founding, team, what the product does and does not do.

The design goal is context economy. A model asked about your product cannot crawl your site; if it can fetch one file, that file should be a high-signal index rather than a rendered homepage full of navigation chrome.

§ 03Whether anything reads it

This is the part that requires care, because the incentive to overstate is strong.

What is verifiable: a growing number of sites publish the file, and several documentation platforms generate it automatically. Directories of adopters exist.

What is not verifiable: that OpenAI, Google, Anthropic, or Perplexity fetch it, weight it, or use it during retrieval. None has documented such behavior. Server-log evidence of AI crawlers requesting /llms.txt has been reported by individual site operators, but a fetch is not evidence of use in answer composition, and the reports are anecdotal. [NEEDS SOURCE] for any claim about adoption rates or measured effect.

The comparison people reach for is robots.txt, which is also not a formal standard but is universally honored. The difference is that robots.txt constrains crawler behavior in a way crawler operators want, while llms.txt asks retrieval systems to trust a self-authored summary. Systems have obvious reasons to be cautious about the second.

Fig. 03 — What is established versus what is asserted about llms.txt.

§ 04Should you publish one

Yes, if it takes an hour and you treat it as a bet rather than a tactic. The reasoning is asymmetric cost, not expected value: the downside is an hour and a file to maintain, the upside is being already compliant if a provider adopts it.

Do not publish one if you cannot keep it accurate. A stale index that points at removed pages is worse than nothing, because the one scenario where the file matters is the scenario where something is trusting it.

§ 05A worked implementation

Generate it from the same source of truth that builds your sitemap, so it cannot drift. This site does exactly that: the route list and content data that produce sitemap.xml also produce /llms.txt.

1. 01Choose the pages. Not everything. The pages that answer a question about who you are and what you do, ten to forty of them. Skip tag archives, pagination, and thin utility pages.
2. 02Write one-line descriptions. Written for a machine deciding whether to fetch, not for a click. State what the page contains, not why it is exciting.
3. 03Group under H2s that match how someone would ask: Documentation, Frameworks, Company, Research.
4. 04Emit at build time from your route manifest, alongside the sitemap, so a new page appears in both or neither.
5. 05Add `llms-full.txt` only if the expanded content is genuinely useful as a single document. For most marketing sites it is not; for documentation, it often is.

The maintenance rule matters more than the content rule. If the file is hand-written, it will be wrong within two quarters.

§ 06What to do instead, or first

If the goal is being cited by AI systems, the interventions with a mechanism behind them are elsewhere:

• Crawler access. Verify that the AI user agents you want are permitted. See whether to block AI crawlers in robots.txt.
• Self-contained answer passages. A retrievable paragraph that answers the question with no surrounding context. This is the single highest-leverage change on most sites.
• Entity consolidation. One canonical identity, declared identically everywhere, so retrieved content resolves to you. Entity Lock Protocol™ covers the operational version.
• Structured data. Article, Organization, Person, FAQPage nodes that agree with the visible page. See schema markup for AI search.

Each of these affects a documented mechanism in how retrieval systems fetch, chunk, and attribute content. llms.txt currently affects none that anyone has demonstrated.

§ 07Serving and maintaining it

A few operational details decide whether the file is usable at all.

Location. Root only: https://example.com/llms.txt. Subdirectory copies are not part of the convention and nothing will look for them. If you run multiple subdomains, each needs its own file, because each is a separate origin.

Content type. Serve it as text/plain; charset=utf-8. Some static hosts will guess application/octet-stream for an unknown extension pattern, which prompts a download rather than a read. Check the response headers, not just the rendered output.

Crawler access. The file is worthless if robots.txt blocks the agents that might fetch it. If you are publishing llms.txt as a signal of openness while blocking every AI user agent, the two statements contradict each other and the block is the one that gets honored.

Absolute URLs. Use them in the link list. The file may be read outside the context of your domain, and a relative path resolved against the wrong base is a dead link.

Freshness. Regenerate it on every deploy. If your build emits a sitemap, add the llms.txt writer to the same script and give both the same route source. The failure mode to avoid is a file that was accurate on the day it was written and has silently described a different site ever since.

One thing worth deciding early: whether to include pages you would not want quoted. The file is a recommendation, not an access control, and listing a page does not grant permission you have not otherwise given — but it does raise the odds that the page is the one fetched. Pricing pages, policy pages, and anything with a short shelf life deserve a deliberate yes or no rather than a default include.

§ 08The honest summary

Publish it, generate it from your build, keep it accurate, and expect nothing measurable. Revisit if a provider documents support. The reason to care about the file at all is not the file — it is that the exercise of choosing forty pages and describing each in one accurate line is a useful audit of whether your site says clearly what it is.

That exercise is the same discipline as AI Visibility Architecture: decide what the machine-readable version of you should be, then make every surface agree with it. The file is one surface, and a minor one. The consistency is the point. For the wider structural view see what AI visibility is and the structured data checklist.