AI crawlers
AI crawler directory
The names you will see in robots.txt and server logs, one page each.
All crawlers
| Name | Run by | Kind |
|---|---|---|
| GPTBot | OpenAI | training crawler |
| OAI-SearchBot | OpenAI | search crawler |
| ChatGPT-User | OpenAI | user fetcher |
| ClaudeBot | Anthropic | training crawler |
| Claude-SearchBot | Anthropic | search crawler |
| Claude-User | Anthropic | user fetcher |
| PerplexityBot | Perplexity | search crawler |
| Perplexity-User | Perplexity | user fetcher |
| Google-Extended | control token, not a crawler | |
| Applebot-Extended | Apple | control token, not a crawler |
| CCBot | Common Crawl | training crawler |
| Meta-ExternalAgent | Meta | training crawler |
| Amazonbot | Amazon | search crawler |
| Bytespider | ByteDance | training crawler |
Which of these can read your site?