What is ClaudeBot?
ClaudeBot is a training crawler run by Anthropic. Here is what Anthropic says it is for, and how to allow or block it.
What it is for
What Anthropic says: Training. Collects content that may be used to train Anthropic's models.
Training crawlers collect pages that may be used to train AI models. Blocking one does not stop the same company's search or user fetchers.
Is ClaudeBot blocked on your site?
Free. Checks your robots.txt for ClaudeBot and 13 other AI crawlers.
How to block ClaudeBot
Add this group to the robots.txt file at the root of your site:
User-agent: ClaudeBot
Disallow: /How to allow ClaudeBot
If nothing in your robots.txt names ClaudeBot and your User-agent: * group does not say Disallow: /, it is already allowed. If an older rule blocks it, remove that group, or say so explicitly:
User-agent: ClaudeBot
Allow: /robots.txt is a request, not a lock, and a "block AI bots" setting at your host or CDN can turn ClaudeBot away whatever robots.txt says. More on writing the rules
Other Anthropic agents
- Claude-SearchBot: a search crawler
- Claude-User: a user fetcher
Operators publish these names and change them. This page reflects what Anthropic has said publicly; check Anthropic's own documentation before you rely on it. See the full AI crawler list.