What is Bytespider?
Bytespider is a training crawler run by ByteDance. Here is what ByteDance says it is for, and how to allow or block it.
What it is for
What ByteDance says: ByteDance publishes very little about it. It is widely reported to collect content for AI models.
Training crawlers collect pages that may be used to train AI models. Blocking one does not stop the same company's search or user fetchers.
Is Bytespider blocked on your site?
Free. Checks your robots.txt for Bytespider and 13 other AI crawlers.
How to block Bytespider
Add this group to the robots.txt file at the root of your site:
User-agent: Bytespider
Disallow: /How to allow Bytespider
If nothing in your robots.txt names Bytespider and your User-agent: * group does not say Disallow: /, it is already allowed. If an older rule blocks it, remove that group, or say so explicitly:
User-agent: Bytespider
Allow: /robots.txt is a request, not a lock, and a "block AI bots" setting at your host or CDN can turn Bytespider away whatever robots.txt says. More on writing the rules
Operators publish these names and change them. This page reflects what ByteDance has said publicly; check ByteDance's own documentation before you rely on it. See the full AI crawler list.