What is CCBot?
CCBot is a training crawler run by Common Crawl. Here is what Common Crawl says it is for, and how to allow or block it.
What it is for
What Common Crawl says: Builds a free, public archive of the web. Many others use that archive, including to train AI models.
Training crawlers collect pages that may be used to train AI models. Blocking one does not stop the same company's search or user fetchers.
Is CCBot blocked on your site?
Free. Checks your robots.txt for CCBot and 13 other AI crawlers.
How to block CCBot
Add this group to the robots.txt file at the root of your site:
User-agent: CCBot
Disallow: /How to allow CCBot
If nothing in your robots.txt names CCBot and your User-agent: * group does not say Disallow: /, it is already allowed. If an older rule blocks it, remove that group, or say so explicitly:
User-agent: CCBot
Allow: /robots.txt is a request, not a lock, and a "block AI bots" setting at your host or CDN can turn CCBot away whatever robots.txt says. More on writing the rules
Operators publish these names and change them. This page reflects what Common Crawl has said publicly; check Common Crawl's own documentation before you rely on it. See the full AI crawler list.