Which big websites
block AI crawlers?
We checked the robots.txt rules of 945 well-known websites for 14 AI crawlers: news, retail, SaaS, finance, government and more. In progress: 484 of 945 sites checked so far (last 22 October 2026). Numbers update as the study runs.
block at least one AI crawler in robots.txt (132 of 454 sites we could read)
block AI training but leave every AI search crawler open (38 sites)
turned away our test request identifying as GPTBot or ClaudeBot (see the limits below)
Blocked most often
| Crawler | Kind | Sites blocking it |
|---|---|---|
| CCBot Common Crawl | Training | 23.3% (106) |
| ClaudeBot Anthropic | Training | 22.5% (102) |
| Bytespider ByteDance | Training | 20.5% (93) |
| GPTBot OpenAI | Training | 20% (91) |
| Applebot-Extended Apple | AI-use control | 19.8% (90) |
| PerplexityBot Perplexity | Search | 19.6% (89) |
| Meta-ExternalAgent Meta | Training | 19.2% (87) |
| Google-Extended | AI-use control | 16.5% (75) |
| Amazonbot Amazon | General | 15% (68) |
| OAI-SearchBot OpenAI | Search | 11.7% (53) |
| ChatGPT-User OpenAI | User fetcher | 11% (50) |
| Claude-User Anthropic | User fetcher | 10.4% (47) |
| Perplexity-User Perplexity | User fetcher | 9.9% (45) |
| Claude-SearchBot Anthropic | Search | 9.3% (42) |
By kind of site
| Sites | Checked | Block any AI crawler | Block GPTBot |
|---|---|---|---|
| US and international news | 77 | 75.3% | 53.2% |
| UK news | 64 | 75% | 59.4% |
| Ecommerce brands | 81 | 9.9% | 7.4% |
| UK trade suppliers | 52 | 9.6% | 3.8% |
| SaaS and developer tools | 108 | 9.3% | 2.8% |
| UK retail | 72 | 4.2% | 1.4% |
No to training, yes to AI search
These sites block at least one AI training crawler (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider) but leave every AI search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) open.
aliexpress.com · amplitude.com · asda.com · axios.com · bostonglobe.com · businessinsider.com · calendly.com · campaignlive.co.uk · canva.com · cbsnews.com · espn.com · figma.com · fortune.com · freshbooks.com · github.com · lakeland.co.uk · latimes.com · loom.com · machinemart.co.uk · marketingweek.com · moneyweek.com · newstatesman.com · petsathome.com · postman.com · prospectmagazine.co.uk · prweek.com · radiotimes.com · rexel.co.uk · semafor.com · slate.com · speedyservices.com · theregister.com · theweek.co.uk · toolstation.com · tortoisemedia.com · vinted.com · warbyparker.com · zalando.com
How we checked, and the limits
- For each site we read robots.txt and apply the rules for the homepage, crawler by crawler, following the robots.txt standard (RFC 9309). A crawler is "blocked" if robots.txt asks it not to read the homepage.
- We also ask for the homepage once as GPTBot and once as ClaudeBot. Our requests come from Cloudflare's network, not from OpenAI or Anthropic, so a firewall that checks where crawlers really come from may refuse our test and still let the real crawler in.
- 30 sites are left out of the robots.txt figures: 20 answered robots.txt with a server error (often bot protection refusing our checker) and 10 could not be reached.
- robots.txt is a request, not a lock. This shows what sites ask, not what AI companies do.
- The list of sites was put together by Kavel; it is not a statistical sample of the web. This page was built and written with AI.