Guides · Research
Do UK trade suppliers
block AI crawlers?
We checked 24 big UK trade and industrial suppliers on 21 October 2026. Most leave the door open to AI crawlers. A few have made deliberate choices.
The short answer
Of the 15 robots.txt files we could read, 13 block no AI crawler at all. One blocks only Meta's crawler. One blocks AI training crawlers and the fetchers AI assistants use when a person asks about a page, but leaves AI search crawlers open.
What we checked
We used our AI crawler checker on 24 well-known UK trade, industrial and tool suppliers. For each site it reads the homepage rules in robots.txt for 14 named AI crawlers (see the AI crawler directory), and asks for the homepage once as GPTBot and once as ClaudeBot to see whether the server turns the request away.
| Result | Sites |
|---|---|
| robots.txt read | 15 |
| Of those, block no AI crawler | 13 |
| Of those, block at least one AI crawler | 2 |
| robots.txt answered with a server error | 4 |
| No robots.txt returned | 5 |
| Server turned away our test request identifying as GPTBot or ClaudeBot | 7 |
Two deliberate choices
Toolstation blocks one AI crawler by name: Meta-ExternalAgent. Every other AI crawler we check is allowed.
Machine Mart blocks 11 of the 14, including GPTBot, ClaudeBot and CCBot, and the user fetchers ChatGPT-User, Claude-User and Perplexity-User. It leaves OAI-SearchBot, Claude-SearchBot and PerplexityBot open: the crawlers that build AI search indexes. In effect: no to training, yes to being found in AI search. That is a coherent policy, and one most sites have not thought about yet.
What this does not show
- Nine results are probably our checker being refused. Four sites answered robots.txt with a server error and five returned none. Big sites' bot protection often refuses unfamiliar requests, so we left these nine out of the headline instead of counting them as "blocks everything" or "has no rules".
- "Turned away our test" is not "blocks GPTBot". Our test requests come from Cloudflare's network, not from OpenAI or Anthropic. A firewall that checks where a crawler really comes from would refuse our test whatever it does with the real crawler.
- Homepage only. We checked the rules that apply to each homepage, not every page.
- Rules, not behaviour. robots.txt is a request. This shows what each site asks, not what any AI company does.
Why it matters for trade suppliers
Buyers are starting to use AI assistants to find suppliers and ask for quotes. Being readable is the first step; answering their requests quickly is the next. If you supply the trade, check your own site with the AI crawler checker and decide what you want to allow: our guide How to allow or block AI crawlers in robots.txt shows how.
Checked 21 October 2026 with Kavel's AI crawler checker. Sites checked: Screwfix, Toolstation, Travis Perkins, Jewson, Selco, Wickes, B&Q (diy.com), RS (uk.rs-online.com), Farnell (uk.farnell.com), CPC (cpc.farnell.com), Cromwell, Würth UK, Accu, CEF, City Plumbing, Plumbbase, Huws Gray, Lyreco, Viking, Machine Mart, Axminster Tools, Power Tool World, Toolstop and UK Fasteners. Written with AI from the checker's results.