Crawling

Crawling in SEO is the process in which programs such as Googlebot fetch web pages, read their content and follow the links on them to discover new and updated pages. Being crawled does not yet mean appearing in search: only afterwards does the search engine decide whether to process a page and add it to its index.

Which crawlers the publishers in our marketplace let in

On 11 November 2026 we fetched the homepage, robots.txt and sitemap of 2,087 active publisher domains from the Linkbroker marketplace catalogue with our own crawler. 1,914 domains answered, most of them German-language sites, and the result shows websites that are well prepared for crawlers:

• Sitemaps are easy to find: 89.1% offer a sitemap, either at /sitemap.xml or through a line in the robots.txt.
• Search engines are welcome: only 0.3% block Googlebot from article pages.
• AI search is welcome too: the bots that collect sources for ChatGPT search, Claude and Perplexity are shut out by 1.9% combined.
• Training is where publishers draw the line, if at all: GPTBot is blocked by 6.6% and ClaudeBot by 6.0%.

So when a publisher does block something, it usually targets the collection of training data and keeps search open. The limits: a one-day snapshot of a typical article path. 93 domains refused our requests and 61 did not respond, which is why the basis is 1,914 rather than 2,087.

How crawling works

Google describes three stages: crawling, indexing and serving results (How Google Search works). Crawling itself runs in five steps:

1. Finding URLs: crawlers start from addresses they already know and learn about new ones through links on other pages and through XML sitemaps.
2. Checking robots.txt: before fetching a page, a well-behaved crawler checks whether the site's robots.txt allows the path.
3. Fetching the page: the server answers with a status code. 200 means content, 301 means moved permanently, 404 or 410 means removed and 5xx means a server problem. If server errors pile up, Google slows down.
4. Rendering: if a page loads content with JavaScript, Google renders it much like a browser. Content that only appears after a user action is usually not seen.
5. Coming back: how often a crawler revisits a page depends on how important and how fresh it seems.

Crawl budget: who needs to care

Crawl budget is the number of URLs Google can and wants to crawl on a site in a given period. It combines two things: the crawl capacity limit, meaning how many requests your server can handle without slowing down, and crawl demand, meaning how important and how up to date Google considers your content. URLs that are more popular on the web tend to be crawled more often.

According to Google, three groups of sites benefit most from managing their crawl budget actively: very large sites with more than a million pages that change about weekly, sites with more than 10,000 pages whose content changes daily, and sites where a large share of URLs is listed as "Discovered - currently not indexed". For most other sites, an up-to-date sitemap and a regular look at the page indexing report are enough.

You can see how Google crawls your site in two places in Search Console. URL Inspection shows the date of the last crawl for a single page, and the Crawl stats report under Settings shows requests over the last 90 days with response times and status codes. Your server's log files are the most precise source, because they record every single visit.

Search crawlers vs AI crawlers

For a long time there was one main reason to crawl the web: building a search index. Today AI companies send out crawlers too, and they do different jobs:

• Search engine crawlers such as Googlebot and Bingbot build the index for classic search, which in Google also feeds AI Overviews and AI Mode.
• Training crawlers such as GPTBot, ClaudeBot and CCBot collect content that may be used to train future AI models.
• AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages so that AI assistants can show them in answers with a source link.
• User-triggered fetchers such as ChatGPT-User load a single page when someone asks about it in a chat and do not crawl the web on their own.

You can allow or block each group separately in your robots.txt. If you want your brand to be named in AI answers, keep the AI search crawlers open, because a page they cannot fetch cannot be cited. What happens after that is explained under answer engine optimization.

How to get Google to crawl your site faster

• Link important pages internally: every page that matters should be reachable within a few clicks of the homepage. Pages with no links pointing to them are hard for a crawler to find.
• Earn links from other sites: a backlink from a page Google already crawls often is one of the fastest ways for a new page to be discovered.
• Keep status codes clean: removed pages return 404 or 410, moved pages a direct redirect without detours.
• Avoid endless URL variations: filters, sorting and session parameters quickly create thousands of near-identical addresses. A clear URL structure and merged duplicate content save crawl time.
• Answer fast and reliably: the more stable your server, the more Google can fetch.
• Keep your sitemap current: list all indexable URLs, and change lastmod only when the content really changes.
• Request indexing once: URL Inspection lets you ask Google to recrawl a page, but there is a quota, and asking repeatedly does not make it faster.

Conclusion

Crawling is the precondition for everything else: what a crawler cannot find or may not fetch can neither rank nor be quoted in an AI answer. Most sites need no tricks for it, just clean technology, good internal links and pages that other sites link to. How mentions on widely read sites help with both is shown under digital PR, and what follows the crawl is covered under indexing.

Frequently asked questions

What is the difference between crawling and indexing?

Crawling means fetching a page, indexing means analysing it and storing it in the search engine's database. A page can be crawled and still not be indexed, for example if Google considers it a duplicate.

Is ChatGPT a web crawler?

ChatGPT itself is an assistant, but OpenAI runs crawlers for it: OAI-SearchBot for ChatGPT search and GPTBot for model training. ChatGPT-User fetches single pages when a user asks about them.

Does noindex save crawl budget?

No, because Google has to fetch a page to read the noindex rule. To stop Google from requesting certain URLs at all, such as endless filter pages, a robots.txt rule is the right tool.

How can I tell whether a visit really comes from Googlebot?

Run a reverse DNS lookup on the IP address: genuine Google crawlers resolve to googlebot.com, google.com or googleusercontent.com, and Google also publishes its IP ranges. The name in the user agent alone proves nothing, because any crawler can fake it.

Related terms: Indexing · llms.txt · AI visibility · Duplicate content · Website speed