AI crawler study · Government
Does nationalarchives.gov.uk block AI crawlers?
nationalarchives.gov.uk blocks none of the 14 AI crawlers we check in robots.txt. Checked 22 October 2026.
| Crawler | Kind | robots.txt |
|---|---|---|
| GPTBot OpenAI | Training | Allowed |
| OAI-SearchBot OpenAI | Search | Allowed |
| ChatGPT-User OpenAI | User fetcher | Allowed |
| ClaudeBot Anthropic | Training | Allowed |
| Claude-SearchBot Anthropic | Search | Allowed |
| Claude-User Anthropic | User fetcher | Allowed |
| PerplexityBot Perplexity | Search | Allowed |
| Perplexity-User Perplexity | User fetcher | Allowed |
| Google-Extended | AI-use control | Allowed |
| Applebot-Extended Apple | AI-use control | Allowed |
| CCBot Common Crawl | Training | Allowed |
| Meta-ExternalAgent Meta | Training | Allowed |
| Amazonbot Amazon | General | Allowed |
| Bytespider ByteDance | Training | Allowed |
The server turned away our test request identifying as GPTBot. Our test came from Cloudflare's network, not from the AI company, so the real crawler may be treated differently.
From robots.txt rules for the homepage (RFC 9309). robots.txt is a request; this does not show what any AI company does. Part of Kavel's AI crawler study, built with AI.
Show this on a page or README
Is this your site? SiteCheck shows what else AI agents need: readable text, structured data and contact details.