What is GPTBot?
GPTBot is a training crawler run by OpenAI. Here is what OpenAI says it is for, and how to allow or block it.
What it is for
What OpenAI says: Training. Collects content that may be used to train OpenAI's models.
Training crawlers collect pages that may be used to train AI models. Blocking one does not stop the same company's search or user fetchers.
Is GPTBot blocked on your site?
Free. Checks your robots.txt for GPTBot and 13 other AI crawlers.
How to block GPTBot
Add this group to the robots.txt file at the root of your site:
User-agent: GPTBot
Disallow: /How to allow GPTBot
If nothing in your robots.txt names GPTBot and your User-agent: * group does not say Disallow: /, it is already allowed. If an older rule blocks it, remove that group, or say so explicitly:
User-agent: GPTBot
Allow: /robots.txt is a request, not a lock, and a "block AI bots" setting at your host or CDN can turn GPTBot away whatever robots.txt says. More on writing the rules
Other OpenAI agents
- OAI-SearchBot: a search crawler
- ChatGPT-User: a user fetcher
Operators publish these names and change them. This page reflects what OpenAI has said publicly; check OpenAI's own documentation before you rely on it. See the full AI crawler list.