Cloudflare's AI bot blocking:
how to check what it blocks on your site
If your site runs through Cloudflare, AI crawlers may be turned away before they ever read your robots.txt. Here is where to look, what the settings do, and how to check the result.
Why this matters
Cloudflare can answer a visitor before your own server sees the request. So it can turn an AI crawler away even when your robots.txt says "welcome". It can also add AI rules to your robots.txt without you editing the file.
Neither is wrong, if you chose it. The problem is a block nobody chose.
Does your site use Cloudflare?
If you manage your domain in a Cloudflare dashboard, it does. Otherwise, ask whoever set up your site, or look up your domain's name servers with a free DNS lookup tool. Name servers ending in cloudflare.com mean Cloudflare runs your DNS.
The settings that can block AI crawlers
Cloudflare renames and moves these settings often. The names below are the ones in its documentation in October 2026. Most are on the Security Settings page of the dashboard, when you filter by Bot traffic.
AI bot policies
Cloudflare groups AI traffic by what it does.
- Search: crawlers that index your content so they can answer questions about it later.
- Agent: automated visits acting in real time for a person, such as chat fetchers and browser agents.
- Training: crawlers that take your content to train or fine-tune a model.
For each one you can block on all pages, block only on pages that show adverts, or allow. Cloudflare's documentation says that from 15 September 2026, new domains get defaults that block Training and Agent bots on pages that show adverts, while Search stays allowed. It also counts crawlers used for both search and training as training. Cloudflare decides automatically which pages show adverts, so check rather than assume.
The older "Block AI bots" switch
Older setups may have a single Block AI bots switch. Cloudflare's documentation marks it as retired from 15 September 2026, in favour of the AI bot policies above. If it was turned on in the past, check what your policies show now.
AI Crawl Control
AI Crawl Control has its own section in the dashboard. It lists the AI crawlers that visit your site, how many requests each made, how many were unsuccessful, and how often each one broke your robots.txt rules. You can set each crawler to Allow or Block. A block here is enforced by Cloudflare, not just requested.
Cloudflare's documentation says that on the free plan it identifies AI crawlers by their user agent, the name each one sends with its requests.
Managed robots.txt
Cloudflare can write AI rules into your robots.txt for you. If you already have a robots.txt, it merges its lines with yours. If you do not have one, it creates one.
Its documentation also says that some free-plan sites without their own robots.txt show a Cloudflare "Content Signals" policy when a crawler asks for the file. Content signals are lines that state your preferences for search, AI input and AI training. So your live robots.txt may contain text you never wrote.
Bot Fight Mode
Bot Fight Mode (on the free plan) and Super Bot Fight Mode (on paid plans) challenge traffic that looks like known bots. Cloudflare's documentation says you cannot make exceptions to Bot Fight Mode with your own rules. If it is on, check how it treats the crawlers you want to let in.
How to check what is being blocked
- Read your live robots.txt. Open yourbusiness.com/robots.txt in a browser. Look for a section marked as Cloudflare managed content, or Content-signal lines you did not write.
- Look at the bot settings. In the dashboard, open Security Settings and filter by Bot traffic. Note the AI bot policies, Bot Fight Mode, and any older Block AI bots switch.
- Open AI Crawl Control. Note the action set for each crawler, and whether requests are failing.
- Check the events. Security Analytics, on the Events tab, shows requests that were blocked or challenged, and which setting did it.
- Run an outside check. Our free AI crawler checker reads your robots.txt for 14 AI crawlers, and asks your homepage for a page as GPTBot and as ClaudeBot.
One caution about outside checks, including ours. Our test requests come from Cloudflare's network, not from OpenAI or Anthropic. Cloudflare can check where a crawler really comes from, so it may treat our test differently from the real crawler. Treat a refused test as a strong hint to look at your settings, not as proof. The dashboard is the final word.
How to change it
Decide first what you want. Training, search and fetches for a user are separate choices. For what each kind of crawler does, see OAI-SearchBot vs GPTBot vs ChatGPT-User and the AI crawler list.
- To stay readable by AI search and assistants but refuse training, allow Search and Agent and block Training. Remember that Cloudflare counts mixed-use crawlers as training.
- To decide crawler by crawler, use AI Crawl Control.
- Make robots.txt and Cloudflare agree. For a crawler that follows the rules, whichever one says no wins. A block at Cloudflare stops the request whatever robots.txt says. A robots.txt "no" asks the crawler not to come at all.
- Check again after any change.
If someone else manages your site, ask them to show you these settings. To write the rules yourself, see How to allow or block AI crawlers in robots.txt.
Not on Cloudflare? Other hosts and CDNs have similar settings. See where blocks hide.
Written with AI. Setting names and behaviour were checked against Cloudflare's developer documentation in October 2026. Cloudflare changes these often, so check its current documentation before you rely on them.