AI crawler study · Lists
Sites that block all 14 AI crawlers
19 of the 746 sites whose robots.txt we could read (2.5%). Their robots.txt asks every one of the 14 AI crawlers we check to stay out of the homepage. Checked October 2026.
| Site | Blocks | Which |
|---|---|---|
| chicagotribune.com | 14 of 14 | all of them |
| dailymail.co.uk | 14 of 14 | all of them |
| denverpost.com | 14 of 14 | all of them |
| huffpost.com | 14 of 14 | all of them |
| inews.co.uk | 14 of 14 | all of them |
| meta.ai | 14 of 14 | all of them |
| metro.co.uk | 14 of 14 | all of them |
| nbcnews.com | 14 of 14 | all of them |
| newcivilengineer.com | 14 of 14 | all of them |
| newscientist.com | 14 of 14 | all of them |
| nursingtimes.net | 14 of 14 | all of them |
| nzherald.co.nz | 14 of 14 | all of them |
| pinterest.com | 14 of 14 | all of them |
| reddit.com | 14 of 14 | all of them |
| thestar.com | 14 of 14 | all of them |
| thisismoney.co.uk | 14 of 14 | all of them |
| usatoday.com | 14 of 14 | all of them |
| x.com | 14 of 14 | all of them |
| yelp.com | 14 of 14 | all of them |
Other lists: Sites that block ClaudeBot but allow GPTBot · Sites that block GPTBot but allow ClaudeBot · Sites that block AI training but allow AI search
From each site's robots.txt rules for the homepage, read as the robots.txt standard (RFC 9309) says. robots.txt is a request, not a lock: this shows what sites ask, not what AI companies do. Sites whose robots.txt we could not read are left out. The list of sites was picked by Kavel and is not a sample of the web. Part of Kavel's AI crawler study, built with AI.
What does your site allow? Check any site live, crawler by crawler.