How to edit robots.txt on WordPress
for AI crawlers

How WordPress handles robots.txt

If your site has no physical robots.txt file, WordPress generates a virtual one. Yoast's help page says the default is:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Creating a real file, or editing it through an SEO plugin, replaces that default. Keep those lines unless you know why not.

Option 1: Yoast SEO

1. In the WordPress dashboard go to Yoast SEO, then Tools, then File editor.
2. If there is no robots.txt yet, click Create robots.txt file.
3. Add your rules, then save.

Yoast notes that if you create the file with its plugin it writes a block that allows all search engines and adds your sitemap link. The File editor does not appear if your WordPress install has disabled file editing (some hosts do this). Yoast says to enable file editing or edit the file through FTP, and that your host can help.

Option 2: another SEO plugin

Many SEO plugins have their own robots.txt editor, in a different place (for example under their tools or general settings). Look for "robots.txt" in the plugin's menu. Use only one tool to edit it, so two plugins do not fight over the file.

Option 3: a file on your server

Create a plain text file named robots.txt, put your rules in it and upload it to the folder your website runs from (the same place as wp-config.php). It overrides the virtual file. If you are unsure where to upload, your web host can tell you.

What to add

Add the AI crawler groups below the default group, not in place of it.

Which crawlers to name

Most AI crawlers publish a name you can put in robots.txt. Their owners describe different purposes, and you can treat them differently:

• Training crawlers (for example GPTBot, ClaudeBot, CCBot): collect pages that may be used to train models.
• Search crawlers (for example OAI-SearchBot, Claude-SearchBot, PerplexityBot): find pages to show or link in an AI assistant's search answers. Blocking these may make your business less likely to be named there.
• Control tokens (Google-Extended, Applebot-Extended): they do not crawl; they tell Google or Apple whether content already crawled may be used for certain AI purposes. Google says Google-Extended does not affect Google Search ranking.

The lines below block the training-related names and leave search alone. Change them to suit your business. Our AI crawler list says what each name is for, and the AI crawler checker shows what your robots.txt says today.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

What this does not do

• robots.txt is a request that well-behaved crawlers follow. It does not technically stop anything, and it does not remove content already collected.
• It does not control whether a person using an AI assistant can have it fetch your page; those visits use their own names (for example ChatGPT-User, Claude-User).
• A firewall or bot-protection setting on your host can turn crawlers away whatever robots.txt says. The checker shows both.

After you save

Open yoursite.com/robots.txt in a browser to see the live file, then run the checker to confirm the names are blocked and nothing you wanted allowed got caught. Not sure whether to block at all? Read allowing or blocking AI crawlers in robots.txt.

Written with AI from the platforms' own help pages (read 26 October 2026) and checked against them. Menus and options change: if yours differ, follow the platform's current help page. This is general information, not advice.