AI crawler study · Lists
Sites that block AI training but allow AI search
69 of the 746 sites whose robots.txt we could read (9.2%). They block at least one AI training crawler (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider) and none of the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Checked October 2026.
| Site | Blocks | Which |
|---|---|---|
| airbnb.com | 3 of 14 | GPTBot, ClaudeBot, Applebot-Extended |
| alibaba.com | 2 of 14 | GPTBot, Google-Extended |
| aliexpress.com | 6 of 14 | GPTBot, ClaudeBot, Google-Extended, CCBot, Amazonbot, Bytespider |
| amplitude.com | 1 of 14 | Bytespider |
| asda.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| axios.com | 3 of 14 | CCBot, Amazonbot, Bytespider |
| bbcgoodfood.com | 4 of 14 | GPTBot, CCBot, Meta-ExternalAgent, Bytespider |
| bostonglobe.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| businessinsider.com | 3 of 14 | ClaudeBot, CCBot, Bytespider |
| calendly.com | 1 of 14 | CCBot |
| campaignlive.co.uk | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| canva.com | 6 of 14 | GPTBot, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| cbsnews.com | 1 of 14 | GPTBot |
| coursera.org | 1 of 14 | Meta-ExternalAgent |
| descript.com | 2 of 14 | Amazonbot, Bytespider |
| dictionary.com | 3 of 14 | GPTBot, Applebot-Extended, CCBot |
| ea.com | 2 of 14 | Meta-ExternalAgent, Bytespider |
| espn.com | 5 of 14 | GPTBot, ChatGPT-User, Google-Extended, CCBot, Bytespider |
| figma.com | 5 of 14 | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider |
| foodnetwork.com | 1 of 14 | Meta-ExternalAgent |
| fortune.com | 2 of 14 | Google-Extended, CCBot |
| freshbooks.com | 1 of 14 | Bytespider |
| genius.com | 8 of 14 | GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| github.com | 1 of 14 | Bytespider |
| goodreads.com | 2 of 14 | GPTBot, CCBot |
| healthline.com | 7 of 14 | GPTBot, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| indiamart.com | 2 of 14 | ClaudeBot, Bytespider |
| lakeland.co.uk | 1 of 14 | Bytespider |
| latimes.com | 3 of 14 | GPTBot, ClaudeBot, Bytespider |
| lonelyplanet.com | 1 of 14 | GPTBot |
| loom.com | 1 of 14 | GPTBot |
| machinemart.co.uk | 11 of 14 | GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| marketingweek.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| mcmaster.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| medium.com | 6 of 14 | GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider |
| moneyweek.com | 4 of 14 | Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider |
| netflix.com | 6 of 14 | Perplexity-User, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| newstatesman.com | 2 of 14 | GPTBot, Meta-ExternalAgent |
| nintendo.com | 7 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider |
| petsathome.com | 6 of 14 | Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| postman.com | 1 of 14 | Bytespider |
| prospectmagazine.co.uk | 2 of 14 | GPTBot, Google-Extended |
| prweek.com | 2 of 14 | GPTBot, Google-Extended |
| radiotimes.com | 5 of 14 | GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider |
| rexel.co.uk | 1 of 14 | Bytespider |
| semafor.com | 1 of 14 | GPTBot |
| skyscanner.net | 1 of 14 | Meta-ExternalAgent |
| slate.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| snapchat.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| speedyservices.com | 3 of 14 | GPTBot, ClaudeBot, Amazonbot |
| techcrunch.com | 7 of 14 | GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider |
| techradar.com | 4 of 14 | Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider |
| temu.com | 1 of 14 | Bytespider |
| theregister.com | 9 of 14 | GPTBot, ClaudeBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| thesaurus.com | 3 of 14 | GPTBot, Applebot-Extended, CCBot |
| theweek.co.uk | 4 of 14 | Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider |
| tomshardware.com | 4 of 14 | Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider |
| toolstation.com | 1 of 14 | Meta-ExternalAgent |
| tortoisemedia.com | 4 of 14 | GPTBot, ChatGPT-User, Google-Extended, CCBot |
| tripadvisor.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| trustpilot.com | 4 of 14 | Perplexity-User, Applebot-Extended, Amazonbot, Bytespider |
| vinted.com | 5 of 14 | GPTBot, ClaudeBot, Google-Extended, CCBot, Amazonbot |
| warbyparker.com | 2 of 14 | CCBot, Bytespider |
| webmd.com | 3 of 14 | GPTBot, ClaudeBot, CCBot |
| which.co.uk | 1 of 14 | Bytespider |
| who.int | 1 of 14 | CCBot |
| wpp.com | 2 of 14 | GPTBot, CCBot |
| zalando.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| zoominfo.com | 4 of 14 | CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
Other lists: Sites that block ClaudeBot but allow GPTBot · Sites that block GPTBot but allow ClaudeBot · Sites that block all 14 AI crawlers
From each site's robots.txt rules for the homepage, read as the robots.txt standard (RFC 9309) says. robots.txt is a request, not a lock: this shows what sites ask, not what AI companies do. Sites whose robots.txt we could not read are left out. The list of sites was picked by Kavel and is not a sample of the web. Part of Kavel's AI crawler study, built with AI.
What does your site allow? Check any site live, crawler by crawler.