Guides › AI crawlers
How to block CCBot in robots.txt
CCBot is the crawler of Common Crawl, a non-profit that publishes free web crawl datasets. Those datasets are widely used to train large language models.
Block CCBot from your whole site
User-agent: CCBot Disallow: /
Block only part of your site
User-agent: CCBot Disallow: /blog/ Disallow: /docs/private/
Notes
Blocking CCBot prevents future Common Crawl snapshots from including your pages. Copies already published in past datasets stay out there.
A crawler follows only the most specific group that matches its name. If you add a User-agent: CCBot group, it stops reading your User-agent: * rules entirely. Repeat any rules you still want it to follow inside its own group.
robots.txt is a request, not an access control. Reputable crawlers honour it; to enforce a block, use your CDN or firewall. Changes are usually picked up within about a day, because crawlers cache robots.txt.
Check it
Run your domain through the checker. It shows whether CCBot and 11 other AI crawlers are allowed or blocked, and which line decides it.