AI crawlers and agents:
GPTBot, ClaudeBot, PerplexityBot and the rest
The names you will see in robots.txt, who runs each one, and what its owner says it is for.
Three kinds of visitor
AI companies now split their automated visitors by job. It helps to know which is which before you decide what to allow.
- Training crawlers collect pages that may be used to train AI models.
- Search crawlers build an index so an assistant can find and link to pages when it answers questions.
- User fetchers visit a page because a person asked the assistant to, for example "check this supplier's prices".
Some names are not crawlers at all. They are control tokens: names you can use in robots.txt to say "do not use my content for this", while the company's normal crawler does the fetching.
The list
| Name in robots.txt | Run by | What the owner says it is for |
|---|---|---|
GPTBot | OpenAI | Training. Collects content that may be used to train OpenAI's models. |
OAI-SearchBot | OpenAI | Search. Finds pages to show and link to in ChatGPT's search features. |
ChatGPT-User | OpenAI | User fetches. Visits a page when a ChatGPT user's request needs it. |
ClaudeBot | Anthropic | Training. Collects content that may be used to train Anthropic's models. |
Claude-SearchBot | Anthropic | Search. Indexes pages to improve the results Claude can give. |
Claude-User | Anthropic | User fetches. Visits a page when a Claude user asks a question that needs it. |
PerplexityBot | Perplexity | Search. Finds and links to pages in Perplexity's answers. Perplexity says it is not used to train AI models. |
Perplexity-User | Perplexity | User fetches. Visits a page when a Perplexity user asks a question that needs it. |
Google-Extended | Control token, not a separate crawler. Googlebot does the fetching. Blocking this name asks Google not to use your content for its Gemini models. Google says it does not affect your place in Google Search. | |
Applebot-Extended | Apple | Control token, not a separate crawler. Applebot does the fetching. Blocking this name asks Apple not to use your content to train its AI models. |
CCBot | Common Crawl | Builds a free, public archive of the web. Many others use that archive, including to train AI models. |
Meta-ExternalAgent | Meta | Training and related uses. Meta says it collects content for uses such as training AI models. |
Amazonbot | Amazon | General. Amazon says it uses it to improve its products and services, which can include its AI. |
Bytespider | ByteDance | ByteDance publishes very little about it. It is widely reported to collect content for AI models. |
Operators publish these names and change them. New ones appear and old ones are retired or split. Treat this page as a starting point and check each operator's own documentation before you rely on it.
Things to know before you act
User fetchers may not follow robots.txt in the same way. Because a person asked for the page, some companies say their user fetchers may not apply robots.txt rules the way a crawler does. Check each operator's documentation for its current position.
Blocking training is a separate choice from blocking search and user fetchers. You can say no to GPTBot and still allow OAI-SearchBot and ChatGPT-User. Blocking everything means assistants may not be able to read your site when a customer asks about you.
robots.txt is a request. Well-behaved crawlers follow it, but it does not physically stop anyone. Bot settings at your host or CDN can also block these visitors before robots.txt is even read. Where blocks hide
Ready to write the rules? How to allow or block AI crawlers in robots.txt