Does your website block
ChatGPT and other AI agents?
Plenty of sites block AI agents without anyone deciding to. Here are the three places a block can hide, and how to check each one.
1. robots.txt
robots.txt is a small text file at the root of your site (yourbusiness.com/robots.txt) that tells automated visitors what they may read. Well-behaved crawlers follow it. A rule like this blocks everything for everyone:
User-agent: *
Disallow: /And a rule like this blocks one named AI crawler from your whole site:
User-agent: GPTBot
Disallow: /AI companies publish the names their crawlers use. Examples include GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-User (Anthropic), PerplexityBot (Perplexity) and CCBot (Common Crawl). Google uses a separate token, Google-Extended, for its AI products. These names change, so check each company's own documentation before you rely on a list.
How to check: open yourbusiness.com/robots.txt in a browser and look for "Disallow: /" under "User-agent: *" or under any of those names.
2. A "noindex" instruction
A page can carry an instruction asking search engines and similar tools not to list it, either as a meta tag in the page or as a header sent by the server. It is easy to leave one switched on after a site is built, for example from a "discourage search engines" setting during development.
How to check: view your homepage's source (right-click, View page source) and search for "noindex".
3. Bot protection at your host or CDN
This is the one people miss. Hosting companies and CDNs offer firewall and bot-protection settings, and some now offer, or switch on by default, options that block AI crawlers. These work before your site is even reached, so robots.txt can say "welcome" while the server says "no".
How to check: look in your host's or CDN's security or bot settings for anything mentioning AI bots, AI crawlers or scrapers. SiteCheck also requests your homepage as an AI agent and tells you if your server refuses it or shows a challenge page.
Should you let them in?
That is a real choice, and it is yours. Some businesses block AI crawlers because they do not want their content used to train models. Others want AI assistants to find and recommend them. The companies often separate these uses: a crawler that collects training data is not the same as an assistant fetching your page because a customer just asked about you. Blocking one does not have to mean blocking the other.
The worst position is not knowing. Decide on purpose, then check your settings match.