Should you let AI crawlers use your site?
A plain decision guide
There is no right answer for every business. What each kind of crawler does, what you gain and give up either way, and how to put your choice into robots.txt.
Three different jobs
AI companies now send different automated visitors for different jobs. You can decide on each one separately.
- Training crawlers collect pages that may be used to train AI models. Examples: GPTBot, ClaudeBot and CCBot.
- Search crawlers build an index so an assistant can find and link to pages when it answers a question. Examples: OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- User fetchers visit a page because a person asked an assistant to look at it. Examples: ChatGPT-User, Claude-User and Perplexity-User.
Google and Apple use control tokens instead. Blocking Google-Extended asks Google not to use your content for its Gemini models, and Google says it does not affect your place in Google Search. Blocking Applebot-Extended asks Apple not to use your content to train its AI models. The AI crawler directory has a page for each name, with what its owner says it is for.
The trade-offs
Training crawlers
- If you allow them, your public pages may become part of what future models learn from. Some businesses are happy with that, especially for basic facts about what they sell.
- If you block them, crawlers that follow robots.txt should stay out. That may matter if your writing, photos or research are what you sell.
- Worth knowing: a block applies from the next time a crawler reads your robots.txt. It does not remove anything collected before. Nobody can say whether allowing training makes an assistant more likely to mention you.
Search crawlers
- If you allow them, assistants that use their index can find your pages and may link to them.
- If you block them, those assistants may not find you, or may not link to you, when someone asks about what you do.
- Worth knowing: allowing them does not mean you will be shown. How each assistant chooses its sources is not public, and it changes.
User fetchers
- If you allow them, an assistant can read your page when a customer asks it to check your prices, products or opening hours.
- If you block them, the assistant may not be able to answer that customer from your own pages.
- Worth knowing: because a person asked for the page, some operators say their user fetchers may not apply robots.txt the way a crawler does. Check each operator's own documentation.
Every crawler also uses some of your server's capacity. If crawling ever slows your site, your host's logs show who visits and how often.
Questions to help you decide
- Is your site mainly there to bring in customers, or is the content itself what you sell?
- When a customer asks an assistant about you, do you want it to be able to read your prices and contact details?
- Is there content you would rather not see reused, such as articles, guides, photos or designs?
- Is anything on your site private? If so, it belongs behind a login, whatever you decide here.
One middle position is to block training and allow search and user fetchers. It will not suit everyone, but it shows the choice is not all or nothing. If you have questions about copyright or how others may use your content, those are legal questions, so ask an adviser.
Put your decision into robots.txt
robots.txt lives at the root of your site (yourbusiness.com/robots.txt). Each group names one or more crawlers, then says what they may read. A crawler follows the one group that best matches its name and ignores the rest.
Block training, allow search and user fetchers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: *
Allow: /Allow everyone
User-agent: *
Allow: /Keep training crawlers out of one section
This asks the named crawlers to stay out of /articles/ only. Everything else stays open to them, and to everyone else.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /articles/
User-agent: *
Allow: /For more examples, including how to block every AI crawler you can name, and the mistakes that undo the rules, see How to allow or block AI crawlers in robots.txt.
Check it worked
- Read the live file. Open yourbusiness.com/robots.txt in a browser.
- Check your host or CDN. Many have a "block AI bots" setting that can turn crawlers away whatever robots.txt says, or change the file for you.
- Run a check. The free AI crawler checker shows, name by name, which AI crawlers your robots.txt lets in.
- Look again now and then. Operators add, rename and retire crawler names.
No promises either way
robots.txt is a request, not a lock. Well-behaved crawlers follow it, but it does not stop anyone who chooses to ignore it. Allowing AI crawlers does not guarantee that any assistant will find, mention or recommend you. Blocking them does not guarantee your content will never be used. Choose on purpose, write your reasons down, and check your settings match.