AI crawler study · US and international news
Does scientificamerican.com block AI crawlers?
scientificamerican.com blocks 9 of 14 AI crawlers in robots.txt. Checked 21 October 2026.
| Crawler | Kind | robots.txt |
|---|---|---|
| GPTBot OpenAI | Training | Blocked |
| OAI-SearchBot OpenAI | Search | Blocked |
| ChatGPT-User OpenAI | User fetcher | Blocked |
| ClaudeBot Anthropic | Training | Blocked |
| Claude-SearchBot Anthropic | Search | Blocked |
| Claude-User Anthropic | User fetcher | Blocked |
| PerplexityBot Perplexity | Search | Blocked |
| Perplexity-User Perplexity | User fetcher | Allowed |
| Google-Extended | AI-use control | Blocked |
| Applebot-Extended Apple | AI-use control | Allowed |
| CCBot Common Crawl | Training | Blocked |
| Meta-ExternalAgent Meta | Training | Allowed |
| Amazonbot Amazon | General | Allowed |
| Bytespider ByteDance | Training | Allowed |
The server turned away our test request identifying as GPTBot and ClaudeBot. Our test came from Cloudflare's network, not from the AI company, so the real crawler may be treated differently.
From robots.txt rules for the homepage (RFC 9309). robots.txt is a request; this does not show what any AI company does. Part of Kavel's AI crawler study, built with AI.
Show this on a page or README
Is this your site? SiteCheck shows what else AI agents need: readable text, structured data and contact details.