robots.txt

robots.txt is a plain text file in the root directory of a website (example.com/robots.txt) that tells crawlers which URLs they may fetch and which they should leave alone. It controls crawling, not indexing: to keep a page out of Google, you use a noindex rule instead of a Disallow line.

Which crawlers 1,914 publishers block in their robots.txt

On 11 November 2026 we requested the robots.txt of 2,087 active publisher domains from the Linkbroker marketplace catalogue, most of which publish in German. 1,914 domains answered. 94.0% of them have a robots.txt (1,800 of 1,914), and 73.5% of the files we read list a sitemap.

The AI crawlers show a clear pattern:

• Most publishers keep the door open: 88.8% let in every one of the 10 AI crawlers in our test.
• Training crawlers are blocked more often: GPTBot by 6.6%, ClaudeBot by 6.0%, Google-Extended by 4.9% and Applebot-Extended by 4.7%, and the data collectors CCBot and Bytespider by 7.6% and 7.7%.
• Search crawlers are almost never blocked: OAI-SearchBot, Claude-SearchBot and PerplexityBot, the bots behind the answers in ChatGPT search, Claude and Perplexity, are shut out by 1.9% combined. Googlebot is shut out by 0.3%.
• Few files name AI bots at all: 15.9% of the robots.txt files mention at least one AI crawler by name.

The limits of this measurement: it is a one-day snapshot, we tested a typical article path, and blocks on single subfolders are not counted. Our own robots.txt on linkbroker.io blocks no crawler at all and contains a single line that points to the sitemap.

How robots.txt works: syntax and rules

A well-behaved crawler requests /robots.txt before it fetches anything else on a host and then follows the rules that apply to it. The format is standardised as the Robots Exclusion Protocol (RFC 9309), and Google documents its own reading in the robots.txt specification. The file is made of groups, and each group starts with one or more user-agent lines:

• User-agent: names the crawler, for example Googlebot or GPTBot. An asterisk (*) covers every crawler that has no group of its own.
• Disallow: blocks a path, for example Disallow: /checkout/.
• Allow: opens a subpath inside a blocked area again.
• Sitemap: gives the full URL of your XML sitemap. This line applies to all crawlers and may appear more than once.

A few details decide whether a file does what you expect. A crawler follows only the group that matches its name most specifically and ignores the others, including the * group. When rules conflict, Google applies the most specific one, which is the rule with the longest matching path. Paths are case-sensitive, * stands for any sequence of characters and $ marks the end of a URL. The file only covers the exact host it sits on, so www.example.com and shop.example.com each need their own. Google reads the first 500 KiB and does not support crawl-delay.

Robots.txt examples you can copy

Replace example.com with your domain and save the file as robots.txt in the root directory.

1. Allow everything. The right choice for most websites: every crawler may fetch every page, and the sitemap is listed.

User-agent: *

Allow: /

Sitemap: https://www.example.com/sitemap.xml

2. Block AI training, keep search and AI search open. Several user-agent lines can share one group. The search crawlers, including OAI-SearchBot, Claude-SearchBot and PerplexityBot, fall under the * group and stay allowed.

User-agent: GPTBot

User-agent: ClaudeBot

User-agent: CCBot

User-agent: Google-Extended

User-agent: Applebot-Extended

Disallow: /

User-agent: *

Allow: /

Sitemap: https://www.example.com/sitemap.xml

3. Keep crawlers out of dead ends. Useful for internal search results, the shopping cart or sorting parameters, which add no value in search. Everything else stays open.

User-agent: *

Disallow: /search/

Disallow: /cart/

Disallow: /*?sort=

Sitemap: https://www.example.com/sitemap.xml

How to block AI crawlers without losing visibility

AI companies run different bots for different jobs, and robots.txt lets you treat them separately. OpenAI describes its settings as independent of each other: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot so that your content is not used to train its models. Sites that block OAI-SearchBot are not shown in ChatGPT search answers, according to OpenAI's crawler documentation. Fetchers that act on a user's request, such as ChatGPT-User, may not follow robots.txt at all, because a person triggered the visit.

Google works differently. Google-Extended is not a crawler but a control token: it decides whether content Google has crawled may be used to train future Gemini models and to ground answers in Gemini Apps. It does not affect inclusion or ranking in Google Search, and AI Overviews and AI Mode are part of Search. Google also says it intends to support a shared machine-readable signal for AI training preferences in early 2027, so this area is worth a look every few months. If you want your brand to appear in AI answers, keep the search crawlers open, because what they cannot read, they cannot cite. More on this under AI visibility.

What robots.txt cannot do

• Remove pages from Google: a blocked URL can still show up in search results without a description if other pages link to it. Use noindex for that, and keep the page crawlable so Google can see the rule.
• Protect private content: the file is public and works as a request, not a lock. Serious crawlers respect it, but real protection needs a password.
• Fix duplicates: for duplicate content, canonical tags and redirects work better, because Google cannot read a blocked page at all.
• Hide CSS and JavaScript safely: block files Google needs to render your pages and it sees something different from your visitors.

robots.txt and llms.txt do different jobs: robots.txt grants or denies access, while llms.txt only suggests which pages an AI agent should read first.

How to test your robots.txt

Open yourdomain.com/robots.txt in a browser. The file should answer with status 200 as plain text. If it returns 404, Google crawls without restrictions. If the server returns an error (5xx), Google stops crawling the site for the first 12 hours and then falls back on the last good version for up to 30 days. In Google Search Console, the robots.txt report under Settings shows which version Google fetched last and any problems, and the URL Inspection tool tells you whether a specific URL is blocked. After a change, give it time: Google generally caches the file for up to 24 hours. A common accident is a staging file with Disallow: / that goes live with a relaunch, so check the file after every deployment. Whether your pages then make it into the index is covered under indexing.

Conclusion

A good robots.txt is usually short: everything important stays open, only real dead ends are blocked and the sitemap is listed. Blocking AI training is your own decision, but if you want to be named in AI answers, the search crawlers need access. Being named also depends on what other sites write about you, which is where digital PR comes in.

Frequently asked questions

Does blocking GPTBot hurt my Google rankings?

No. GPTBot belongs to OpenAI and has nothing to do with Google Search. What matters for Google is that Googlebot may crawl your pages, and even blocking Google-Extended does not change your rankings.

Do I need a robots.txt file?

It is not required, and without one every crawler may fetch everything. It is still worth having, because it lists your sitemap and lets you exclude areas that add nothing to search.

Can robots.txt remove a page from Google?

No, it only stops crawling. A blocked page can stay in the index if other pages link to it, so use a noindex rule on a crawlable page instead.

How do I create a robots.txt file?

Write it as plain text in a simple editor, name it robots.txt and upload it to the root directory of your domain. Many content management systems create the file automatically and let you edit it in their SEO settings.

Related terms: Indexing · llms.txt · Answer engine optimization · URL structure