What is Claude-SearchBot?
Claude-SearchBot is a search crawler run by Anthropic. Here is what Anthropic says it is for, and how to allow or block it.
What it is for
What Anthropic says: Search. Indexes pages to improve the results Claude can give.
Search crawlers build an index that an AI assistant or search product uses to find and link to pages when it answers questions. Blocking one can mean that product cannot show or link to your pages.
Is Claude-SearchBot blocked on your site?
Free. Checks your robots.txt for Claude-SearchBot and 13 other AI crawlers.
How to block Claude-SearchBot
Add this group to the robots.txt file at the root of your site:
User-agent: Claude-SearchBot
Disallow: /How to allow Claude-SearchBot
If nothing in your robots.txt names Claude-SearchBot and your User-agent: * group does not say Disallow: /, it is already allowed. If an older rule blocks it, remove that group, or say so explicitly:
User-agent: Claude-SearchBot
Allow: /robots.txt is a request, not a lock, and a "block AI bots" setting at your host or CDN can turn Claude-SearchBot away whatever robots.txt says. More on writing the rules
Other Anthropic agents
- ClaudeBot: a training crawler
- Claude-User: a user fetcher
Operators publish these names and change them. This page reflects what Anthropic has said publicly; check Anthropic's own documentation before you rely on it. See the full AI crawler list.