How to allow or block
AI crawlers in robots.txt

How robots.txt rules work

robots.txt lives at the root of your site (yourbusiness.com/robots.txt). It is made of groups. Each group starts with one or more User-agent lines naming who it is for, followed by Allow and Disallow lines saying which paths they may read.

• A crawler uses the one group that best matches its name and ignores the rest. If you have a group for GPTBot, GPTBot ignores your User-agent: * group completely.
• * means everyone not named elsewhere.
• Disallow: / blocks the whole site. An empty Disallow: blocks nothing.
• The most specific path wins. Disallow: / with Allow: /blog/ lets the crawler read your blog and nothing else.
• Each host needs its own file. shop.yourbusiness.com does not read the file on yourbusiness.com.

For the names and what each one does, see the AI crawler list.

Example 1: block training, allow search and user fetchers

This asks training crawlers to stay out, while assistants can still find your pages and read them when a customer asks.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Allow: /

OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User are not named, so they fall under * and are allowed. Blocking Google-Extended does not block Googlebot, so normal Google Search is not affected.

Example 2: allow everyone

User-agent: *
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml

The Sitemap line is optional but useful. It points crawlers to a list of your pages.

Example 3: block all AI crawlers you can name

There is no single "AI" switch in robots.txt. You list each name. Leave out User-agent: * blocks unless you also want to block ordinary search engines.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Think before you use this one. It also asks assistants not to read your site when a customer asks about you.

Common mistakes

• A stray User-agent: * with Disallow: /. Often left over from when the site was being built. It blocks everyone, including search engines.
• A named group that forgets your other rules. If you add a group for one crawler, it no longer reads your * group. Copy any rules you still want it to follow into its own group.
• Host or CDN settings that override robots.txt. Many hosts and CDNs have a "block AI bots" setting. Cloudflare, for example, has one, and can also add AI rules to your robots.txt for you. These can block a crawler at the server even when robots.txt says yes, or change the file without you editing it.
• The file in the wrong place. It must open at yourbusiness.com/robots.txt, not in a folder.
• Typos in names. A misspelt name matches nothing, so the rule does nothing.

robots.txt is a request, not a lock

Well-behaved crawlers follow robots.txt. It does not stop anyone who chooses to ignore it, and some user fetchers may treat it differently because a person asked for the page. If you need to keep something private, put it behind a login.

After any change, check the live file. Then check your host's bot settings match what you meant.