OAI-SearchBot vs GPTBot
vs ChatGPT-User
OpenAI sends three different visitors to websites. Each has its own job and its own name in robots.txt. Here is what each one does, in OpenAI's words, and how to set each one.
The short version
| Name | Job | What OpenAI says blocking it means |
|---|---|---|
GPTBot | Training. Collects content that may be used to train OpenAI's generative AI models. | Your content should not be used for that training. |
OAI-SearchBot | Search. Finds pages to show and link to in ChatGPT's search features. | Your site will not be shown in ChatGPT search answers, though it can still appear as a navigational link. |
ChatGPT-User | User fetches. Visits a page when a ChatGPT user's request needs it. | Because a user started the visit, robots.txt rules may not apply. |
The three are set separately. You can say no to one and yes to the others.
GPTBot: training
OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models. Disallowing GPTBot tells OpenAI that your content should not be used for that training.
GPTBot has nothing to do with whether ChatGPT can find or link to you in search. Blocking it is a choice about training only.
Read more: What is GPTBot? and How to block GPTBot (and when you should not).
OAI-SearchBot: search
OAI-SearchBot finds pages so ChatGPT can show and link to them in its search features. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
OpenAI also says it can take about 24 hours after you change robots.txt for its search systems to adjust. So do not expect a change to show the same minute.
Allowing OAI-SearchBot does not mean ChatGPT will show, quote or recommend your site. OpenAI does not publish how it chooses sources. Allowing it only means you have not asked to be left out.
Read more: What is OAI-SearchBot?
ChatGPT-User: when a person asks
ChatGPT-User is not a crawler in the usual sense. OpenAI says it is not used to crawl the web automatically. It visits a page when a person using ChatGPT, or a custom GPT, asks something that needs that page. For example, a customer pastes your web address and asks for your opening hours.
Two points in OpenAI's documentation matter here.
- robots.txt may not apply. Because a user started the visit, OpenAI says robots.txt rules may not apply to ChatGPT-User.
- It is not the search switch. OpenAI says ChatGPT-User is not used to decide whether content may appear in search. Use OAI-SearchBot for that.
If you need to keep pages away from every visitor, put them behind a login. robots.txt will not do that job.
Read more: What is ChatGPT-User?
Three common choices
Each example is a complete robots.txt. Add your own other rules and your Sitemap line as needed.
1. Allow all three
User-agent: *
Allow: /None of OpenAI's names appears, so all three follow the general rule. Check that your User-agent: * group does not say Disallow: /.
2. No training, but stay open to search and user fetches
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /Your pages stay open to ChatGPT search and to fetches when a customer asks, and you have asked OpenAI not to use them for training. It keeps your two choices, search and training, apart.
3. Ask all three to stay out
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /
User-agent: *
Allow: /This asks OpenAI to leave your site out of training and out of ChatGPT search answers. Remember that OpenAI says robots.txt may not apply to ChatGPT-User. Think about this one. If a customer asks ChatGPT about you, it may have little from your own site to go on.
Things to check
- One crawl can serve two jobs. OpenAI says that if a site allows both GPTBot and OAI-SearchBot, it may use the results of just one crawl for both purposes, to avoid crawling twice. If you want search but not training, block GPTBot by name.
- Your host may decide first. A "block AI bots" setting at your host or CDN can turn these visitors away before robots.txt is read. See how to check Cloudflare's AI bot settings.
- Names and rules change. OpenAI has added names over time and revised its documentation. Check OpenAI's own crawler pages before you rely on any list, including ours.
- Check the visitor is real. Anyone can send a request that calls itself GPTBot. OpenAI publishes the IP addresses its visitors use, so your host or developer can check.
- Other companies split things too. Anthropic has separate names for training, search and user fetches. Perplexity has separate names for search and user fetches. See the AI crawler list.
Which should you choose?
Start with what you want. If you want customers' assistants to be able to read your site, keep OAI-SearchBot and ChatGPT-User open. If you do not want your content used for training, block GPTBot. If you want nothing to do with OpenAI at all, block all three, and accept that ChatGPT may then know less about you.
The worst position is not knowing. Decide on purpose, then check your live robots.txt and your host's settings say the same thing. For the wider choice across all AI companies, see How to allow or block AI crawlers in robots.txt.
Written with AI. OpenAI's descriptions were checked against its published crawler documentation in October 2026.