AI crawler study · B2B marketplaces and distributors
Which B2B marketplaces and distributors sites block AI crawlers?
3 of 22 B2B marketplaces and distributors sites block at least one AI crawler (13.6%). Checked October 2026, 43 sites.
Blocked most often
| Crawler | Kind | Sites blocking it |
|---|---|---|
| GPTBot OpenAI | Training | 2 of 22 |
| ClaudeBot Anthropic | Training | 2 of 22 |
| Google-Extended | AI-use control | 2 of 22 |
| Bytespider ByteDance | Training | 2 of 22 |
| Applebot-Extended Apple | AI-use control | 1 of 22 |
| CCBot Common Crawl | Training | 1 of 22 |
| Meta-ExternalAgent Meta | Training | 1 of 22 |
| Amazonbot Amazon | General | 1 of 22 |
| OAI-SearchBot OpenAI | Search | 0 of 22 |
| ChatGPT-User OpenAI | User fetcher | 0 of 22 |
| Claude-SearchBot Anthropic | Search | 0 of 22 |
| Claude-User Anthropic | User fetcher | 0 of 22 |
| PerplexityBot Perplexity | Search | 0 of 22 |
| Perplexity-User Perplexity | User fetcher | 0 of 22 |
Every site
| Site | Blocks | Which |
|---|---|---|
| mcmaster.com | 8 of 14 | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| alibaba.com | 2 of 14 | GPTBot, Google-Extended |
| indiamart.com | 2 of 14 | ClaudeBot, Bytespider |
| 1688.com | 0 of 14 | none |
| avnet.com | 0 of 14 | none |
| bidfood.co.uk | 0 of 14 | none |
| dhgate.com | 0 of 14 | none |
| digikey.com | 0 of 14 | none |
| ec21.com | 0 of 14 | none |
| henryschein.com | 0 of 14 | none |
| jaggaer.com | 0 of 14 | none |
| made-in-china.com | 0 of 14 | none |
| mckesson.com | 0 of 14 | none |
| mscdirect.com | 0 of 14 | none |
| octopart.com | 0 of 14 | none |
| protolabs.com | 0 of 14 | none |
| staples.com | 0 of 14 | none |
| sysco.com | 0 of 14 | none |
| tti.com | 0 of 14 | none |
| uline.com | 0 of 14 | none |
| usfoods.com | 0 of 14 | none |
| xometry.com | 0 of 14 | none |
Left out, because their robots.txt could not be read (bot protection, errors or no answer): alliedelec.com, arrow.com, booker.co.uk, bunzl.com, cardinalhealth.com, europages.com, farnell.com, fastenal.com, globalsources.com, grainger.com, kompass.com, lyreco.com, motion.com, mouser.com, newark.com, officedepot.com, rs-online.com, thomasnet.com, tradeindia.com, wesco.com, zoro.com.
From each site's robots.txt rules for the homepage, read as the robots.txt standard (RFC 9309) says. robots.txt is a request, not a lock: this shows what sites ask, not what AI companies do. Sites whose robots.txt we could not read are left out. The list of sites was picked by Kavel and is not a sample of the web. Part of Kavel's AI crawler study, built with AI.