Guides › AI crawlers
How to block Bytespider in robots.txt
Bytespider is ByteDance's crawler (the TikTok parent company), widely reported to collect data for ByteDance's AI products.
Block Bytespider from your whole site
User-agent: Bytespider Disallow: /
Block only part of your site
User-agent: Bytespider Disallow: /blog/ Disallow: /docs/private/
Notes
Several site operators and CDN providers have reported that Bytespider doesn't reliably honour robots.txt. If it ignores your rules, block it at the firewall or CDN by user agent. Cloudflare's bot-management and AI-crawler controls can do this on any plan.
A crawler follows only the most specific group that matches its name. If you add a User-agent: Bytespider group, it stops reading your User-agent: * rules entirely. Repeat any rules you still want it to follow inside its own group.
robots.txt is a request, not an access control. Reputable crawlers honour it; to enforce a block, use your CDN or firewall. Changes are usually picked up within about a day, because crawlers cache robots.txt.
Check it
Run your domain through the checker. It shows whether Bytespider and 11 other AI crawlers are allowed or blocked, and which line decides it.