AI crawler study · Lists
Sites that block ClaudeBot but allow GPTBot
28 of the 746 sites whose robots.txt we could read (3.8%). Their robots.txt asks Anthropic's training crawler (ClaudeBot) to stay out of the homepage, but does not name OpenAI's (GPTBot). Checked October 2026.
| Site | Blocks | Which |
|---|---|---|
| allrecipes.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| arstechnica.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| businessinsider.com | 3 of 14 | ClaudeBot, CCBot, Bytespider |
| epicurious.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| ft.com | 8 of 14 | ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| houstonchronicle.com | 7 of 14 | ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Applebot-Extended, CCBot, Bytespider |
| indiamart.com | 2 of 14 | ClaudeBot, Bytespider |
| investopedia.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| irishnews.com | 6 of 14 | ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent |
| lbc.co.uk | 5 of 14 | ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider |
| newyorker.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| nymag.com | 11 of 14 | ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| nypost.com | 10 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| people.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| poe.com | 12 of 14 | OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| seriouseats.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| sfchronicle.com | 8 of 14 | ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider |
| theatlantic.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| theguardian.com | 9 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| thesun.co.uk | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| thetimes.co.uk | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| theverge.com | 11 of 14 | ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| vanityfair.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| vox.com | 11 of 14 | ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| washingtonpost.com | 7 of 14 | ClaudeBot, PerplexityBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| wired.co.uk | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| wired.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| wsj.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
Other lists: Sites that block GPTBot but allow ClaudeBot · Sites that block AI training but allow AI search · Sites that block all 14 AI crawlers
From each site's robots.txt rules for the homepage, read as the robots.txt standard (RFC 9309) says. robots.txt is a request, not a lock: this shows what sites ask, not what AI companies do. Sites whose robots.txt we could not read are left out. The list of sites was picked by Kavel and is not a sample of the web. Part of Kavel's AI crawler study, built with AI.
What does your site allow? Check any site live, crawler by crawler.