AI crawler study · US and international news
Does businessinsider.com block AI crawlers?
businessinsider.com blocks 3 of 14 AI crawlers in robots.txt. Checked 21 October 2026.
| Crawler | Kind | robots.txt |
|---|---|---|
| GPTBot OpenAI | Training | Allowed |
| OAI-SearchBot OpenAI | Search | Allowed |
| ChatGPT-User OpenAI | User fetcher | Allowed |
| ClaudeBot Anthropic | Training | Blocked |
| Claude-SearchBot Anthropic | Search | Allowed |
| Claude-User Anthropic | User fetcher | Allowed |
| PerplexityBot Perplexity | Search | Allowed |
| Perplexity-User Perplexity | User fetcher | Allowed |
| Google-Extended | AI-use control | Allowed |
| Applebot-Extended Apple | AI-use control | Allowed |
| CCBot Common Crawl | Training | Blocked |
| Meta-ExternalAgent Meta | Training | Allowed |
| Amazonbot Amazon | General | Allowed |
| Bytespider ByteDance | Training | Blocked |
From robots.txt rules for the homepage (RFC 9309). robots.txt is a request; this does not show what any AI company does. Part of Kavel's AI crawler study, built with AI.
Show this on a page or README
Is this your site? SiteCheck shows what else AI agents need: readable text, structured data and contact details.