Which AI crawlers
can read your site?
Enter your website. We read your robots.txt and show, name by name, which AI crawlers it lets in. We also ask your homepage for a page as GPTBot and ClaudeBot to see if your server or firewall turns them away.
We read only your public robots.txt and homepage. This checks the rules your site publishes. It does not test real AI assistants or show what any of them will do.
Write your AI rules for robots.txt
Tick the crawlers you want to keep out, then copy the lines into your robots.txt. Training crawlers collect pages to train AI models. Search crawlers build the index that AI search answers from. User fetchers visit a page when someone asks an assistant about it.
robots.txt is a request, not a lock. Most large AI companies say their crawlers follow it; check each company's own documentation, and check any "block AI bots" setting at your host or CDN, which can override it.
What the results mean
Allowed: nothing in your robots.txt asks this crawler to stay away from your homepage. If some paths are excluded (for example /admin/), they are listed under the result. Blocked: your robots.txt asks this crawler not to read your homepage, either by name or through a rule for all crawlers (User-agent: *).
Read more: the AI crawler list and how to allow or block AI crawlers.