llms.txt

llms.txt is a Markdown text file in the root directory of a website (example.com/llms.txt) that gives AI assistants a short, curated overview of its most important content, with links. It is an open proposal from September 2024 rather than an official standard, and Google states explicitly that your site does not need one to appear in the AI features of Google Search.

What we found when we checked 1,914 publishers for llms.txt

On 11 November 2026 we checked 2,087 active publisher domains from the Linkbroker marketplace catalogue for a file at /llms.txt. Of these, 1,914 were reachable, most of them German-language sites. Only 12.9% of them have an llms.txt that follows the specification (247 of 1,914). For comparison, 94.0% have a robots.txt and 89.1% have a sitemap that crawlers can find.

A second fetch on the same day, in which we read every file, found 248 files that follow the specification. Their content shows how most of them come about:

• Often generated by a plugin: 103 of the 248 files name the plugin or shop system that created them, 79 of them Yoast SEO alone. Shopify follows with 10, Hostinger Tools and All in One SEO with 7 each.
• Link lists rather than a selection: Many files consist of automatically filled sections listing pages, posts and categories. The median file contains 26 links, and 20 files contain none at all.
• A short description is usually there: 198 files (80%) sum up the website in a blockquote, and 107 (43%) use the "Optional" section for secondary links.
• Markdown versions are rare: Only 14 files link to plain Markdown versions of their pages (.md), which the specification recommends for agents.

Another 29 files come from Rank Math and start with a note before the heading; under a strict reading of the specification they do not count. Our takeaway: for many publishers, llms.txt is a by-product of an SEO plugin rather than an editorially maintained overview. Limits of the measurement: it is a one-day snapshot, and we only checked the file in the root directory, not in subfolders.

Do Google, ChatGPT and Claude use llms.txt?

The primary sources show that the providers handle the file differently:

• Google Search: The Search Central documentation on AI features says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." This refers to AI Overviews and AI Mode (Google Search Central).
• Chrome Lighthouse: As part of its audits for AI agents, the website testing tool checks whether an existing llms.txt loads without a server error. If there is no file, the audit is marked as not applicable (Chrome for Developers). This concerns agents that read websites on behalf of users, not rankings.
• OpenAI and Anthropic: In their crawler documentation, both describe robots.txt as the way to control their bots, and llms.txt does not appear there as a signal (checked 11 November 2026). For their own developer documentation, however, OpenAI, Anthropic and Google all provide an llms.txt.

That matches what the file was designed for. Jeremy Howard, who proposed it, writes that llms.txt is mainly used at inference time, when an agent fetches information while helping a user, rather than for training models (llmstxt.org).

How to create an llms.txt file: structure and example

You can write the file in any text editor, and many SEO plugins and shop systems also generate it automatically. What matters is that it is then available at /llms.txt in the root directory. The specification sets a fixed order, and only the first heading is required:

1. An H1 with the name of the website or project, as the first line.
2. A blockquote with a short summary that contains the key information needed to understand the rest.
3. Details as paragraphs or lists, without further headings.
4. H2 sections with file lists: one link per line in the format [name](URL), optionally followed by a colon and a short note.
5. An "Optional" section for links an agent can skip when space is short.

An llms.txt may also sit in a subfolder, for example /docs/llms.txt, and then covers that area. The specification also recommends offering important pages as a clean Markdown file at the same address with the extension .md. This is what the file could look like for an online shop that ships across Europe:

# Example Garden Furniture

> Online shop for wooden garden furniture with delivery to the UK, Ireland and the EU.

## Guides

- [Caring for wood](https://www.example.com/guides/wood-care.md): oils, stains and the right time of year

## Service

- [Shipping and returns](https://www.example.com/service/shipping.md): delivery times and costs by country

## Optional

- [About us](https://www.example.com/about-us.md)

llms.txt vs robots.txt and sitemap: three files, three jobs

• robots.txt controls access: it tells crawlers which areas they may fetch. It is also where you allow or block AI crawlers.
• An XML sitemap lists all the pages search engines should index, completely and in machine-readable form.
• llms.txt makes a selection: a few especially helpful pages with a short explanation, so an agent does not have to search the whole website.

So an llms.txt neither allows nor blocks anything. Our measurement shows this too: 28 of the 248 publishers with an llms.txt block at least one training crawler in their robots.txt, yet none of them blocks the search crawlers behind ChatGPT, Claude or Perplexity. Why those crawlers matter for appearing in AI answers is explained under answer engine optimization.

Is llms.txt worth it for your website?

An llms.txt does no harm and is quick to set up. Whether it helps depends on who reads your content:

• Software, APIs and technical documentation: This is where it is most likely to pay off, because coding assistants and agents look things up on purpose.
• Shops, blogs and company websites: Not a must. If you create one, curate it: a manageable selection of your most important pages with one line of explanation each, instead of the complete post list from the plugin. Update it whenever your key pages change.
• Technically clean: The file should respond at /llms.txt with status 200 as plain text and start with a line "# Name". In our measurement, 5 servers delivered an HTML page instead.

Other things matter more for visibility in AI answers: that search crawlers may fetch your pages, that your content answers questions clearly (more on this under LLM SEO) and that other websites write about your brand. In an Ahrefs analysis of 75,000 brands, brand mentions were more closely related to appearances in Google AI Overviews (0.664) than the number of backlinks was (0.218). The study shows a correlation, not a cause.

Conclusion

llms.txt is a simple, open proposal that mainly helps AI agents look things up. It is not a ranking lever: Google explicitly does not need it for its AI features, and among our publishers it is often an automatically generated link list. What matters more is whether AI systems cite your brand, which depends on clear content and on mentions in the media that AI answers draw on. Our AI Score tells you how often an outlet is cited in answers from ChatGPT, Perplexity, Gemini, Copilot and Google AI Overviews, and you can see it for many outlets in our marketplace before you book. More on this under digital PR.

Frequently asked questions

Do I need an llms.txt file for my website?

In most cases, no: Google Search does not require it, and the major AI providers do not list it as a signal in their crawler documentation. It is most useful for software and API documentation that coding assistants look up.

Does llms.txt control AI training or block AI crawlers?

No. It is a reading list for AI assistants and contains no permissions, so you still allow or block crawlers, including those used for training, in your robots.txt.

Will an llms.txt file get my site cited more often in AI answers?

There is no evidence for that so far. Citations depend on open crawlers, clear answers on your pages and mentions on other sites, as described in the section on whether llms.txt is worth it.

Related terms: Answer engine optimization · LLM SEO · AI visibility · Indexing · Off-page SEO