AI crawler study · US and international news
Which US and international news sites block AI crawlers?
59 of 74 US and international news sites block at least one AI crawler (79.7%). Checked October 2026, 80 sites.
Blocked most often
| Crawler | Kind | Sites blocking it |
|---|---|---|
| CCBot Common Crawl | Training | 53 of 74 |
| ClaudeBot Anthropic | Training | 51 of 74 |
| Bytespider ByteDance | Training | 50 of 74 |
| PerplexityBot Perplexity | Search | 49 of 74 |
| Google-Extended | AI-use control | 45 of 74 |
| Applebot-Extended Apple | AI-use control | 43 of 74 |
| GPTBot OpenAI | Training | 42 of 74 |
| Meta-ExternalAgent Meta | Training | 38 of 74 |
| Amazonbot Amazon | General | 33 of 74 |
| ChatGPT-User OpenAI | User fetcher | 32 of 74 |
| Claude-User Anthropic | User fetcher | 29 of 74 |
| Perplexity-User Perplexity | User fetcher | 27 of 74 |
| Claude-SearchBot Anthropic | Search | 25 of 74 |
| OAI-SearchBot OpenAI | Search | 22 of 74 |
Every site
| Site | Blocks | Which |
|---|---|---|
| chicagotribune.com | 14 of 14 | all of them |
| denverpost.com | 14 of 14 | all of them |
| huffpost.com | 14 of 14 | all of them |
| nbcnews.com | 14 of 14 | all of them |
| nzherald.co.nz | 14 of 14 | all of them |
| thestar.com | 14 of 14 | all of them |
| usatoday.com | 14 of 14 | all of them |
| globeandmail.com | 13 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| nytimes.com | 13 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| cnbc.com | 12 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| cnn.com | 12 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot-Extended, CCBot, Amazonbot, Bytespider |
| economist.com | 12 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider |
| reuters.com | 12 of 14 | GPTBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| dw.com | 11 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| newyorker.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| nymag.com | 11 of 14 | ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| people.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| smh.com.au | 11 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| theage.com.au | 11 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| theatlantic.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| theverge.com | 11 of 14 | ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| vanityfair.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| vox.com | 11 of 14 | ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| wired.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| wsj.com | 11 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| buzzfeed.com | 10 of 14 | GPTBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| dallasnews.com | 10 of 14 | GPTBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider |
| deadline.com | 10 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| france24.com | 10 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, CCBot |
| hollywoodreporter.com | 10 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| nypost.com | 10 of 14 | ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| rollingstone.com | 10 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| thehill.com | 10 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| variety.com | 10 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| bloomberg.com | 9 of 14 | GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| scientificamerican.com | 9 of 14 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, CCBot |
| abc.net.au | 8 of 14 | GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| apnews.com | 8 of 14 | GPTBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Applebot-Extended, CCBot, Amazonbot |
| bostonglobe.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| csmonitor.com | 8 of 14 | GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| forbes.com | 8 of 14 | GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| newsweek.com | 8 of 14 | GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Applebot-Extended, CCBot, Amazonbot, Bytespider |
| sfchronicle.com | 8 of 14 | ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider |
| slate.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| theathletic.com | 8 of 14 | GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Amazonbot, Bytespider |
| houstonchronicle.com | 7 of 14 | ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Applebot-Extended, CCBot, Bytespider |
| motherjones.com | 7 of 14 | GPTBot, ClaudeBot, PerplexityBot, Perplexity-User, Applebot-Extended, Meta-ExternalAgent, Amazonbot |
| washingtonpost.com | 7 of 14 | ClaudeBot, PerplexityBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| theintercept.com | 6 of 14 | GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, CCBot, Amazonbot |
| aljazeera.com | 5 of 14 | GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Bytespider |
| espn.com | 5 of 14 | GPTBot, ChatGPT-User, Google-Extended, CCBot, Bytespider |
| pbs.org | 4 of 14 | GPTBot, PerplexityBot, CCBot, Bytespider |
| axios.com | 3 of 14 | CCBot, Amazonbot, Bytespider |
| businessinsider.com | 3 of 14 | ClaudeBot, CCBot, Bytespider |
| latimes.com | 3 of 14 | GPTBot, ClaudeBot, Bytespider |
| fortune.com | 2 of 14 | Google-Extended, CCBot |
| cbsnews.com | 1 of 14 | GPTBot |
| semafor.com | 1 of 14 | GPTBot |
| straitstimes.com | 1 of 14 | Google-Extended |
| ajc.com | 0 of 14 | none |
| esquire.com | 0 of 14 | none |
| foreignaffairs.com | 0 of 14 | none |
| foreignpolicy.com | 0 of 14 | none |
| foxnews.com | 0 of 14 | none |
| inquirer.com | 0 of 14 | none |
| japantimes.co.jp | 0 of 14 | none |
| nationalgeographic.com | 0 of 14 | none |
| nationalreview.com | 0 of 14 | none |
| propublica.org | 0 of 14 | none |
| reason.com | 0 of 14 | none |
| scmp.com | 0 of 14 | none |
| thedailybeast.com | 0 of 14 | none |
| time.com | 0 of 14 | none |
| vice.com | 0 of 14 | none |
Left out, because their robots.txt could not be read (bot protection, errors or no answer): cbc.ca, miamiherald.com, npr.org, politico.com, seattletimes.com, startribune.com.
From each site's robots.txt rules for the homepage, read as the robots.txt standard (RFC 9309) says. robots.txt is a request, not a lock: this shows what sites ask, not what AI companies do. Sites whose robots.txt we could not read are left out. The list of sites was picked by Kavel and is not a sample of the web. Part of Kavel's AI crawler study, built with AI.
What does your site allow? Check any site live, crawler by crawler.