AI crawler study · AI companies
Does together.ai block AI crawlers?
together.ai blocks 1 of 14 AI crawlers in robots.txt. Checked 22 October 2026.
| Crawler | Kind | robots.txt |
|---|---|---|
| GPTBot OpenAI | Training | Allowed |
| OAI-SearchBot OpenAI | Search | Allowed |
| ChatGPT-User OpenAI | User fetcher | Allowed |
| ClaudeBot Anthropic | Training | Allowed |
| Claude-SearchBot Anthropic | Search | Allowed |
| Claude-User Anthropic | User fetcher | Allowed |
| PerplexityBot Perplexity | Search | Allowed |
| Perplexity-User Perplexity | User fetcher | Allowed |
| Google-Extended | AI-use control | Blocked |
| Applebot-Extended Apple | AI-use control | Allowed |
| CCBot Common Crawl | Training | Allowed |
| Meta-ExternalAgent Meta | Training | Allowed |
| Amazonbot Amazon | General | Allowed |
| Bytespider ByteDance | Training | Allowed |
From robots.txt rules for the homepage (RFC 9309). robots.txt is a request; this does not show what any AI company does. Part of Kavel's AI crawler study, built with AI.
Show this on a page or README
Is this your site? SiteCheck shows what else AI agents need: readable text, structured data and contact details.