Guides › AI crawlers
How to block ClaudeBot in robots.txt
ClaudeBot is Anthropic's crawler that collects public web content that may be used to train Anthropic's models.
Block ClaudeBot from your whole site
User-agent: ClaudeBot Disallow: /
Block only part of your site
User-agent: ClaudeBot Disallow: /blog/ Disallow: /docs/private/
Notes
Anthropic also lists Claude-User (fetches a user asks for) and Claude-SearchBot (search quality). Each one respects its own group, so list every agent you want to restrict. Older robots.txt files sometimes target anthropic-ai or Claude-Web. Keeping those lines does no harm, but ClaudeBot is the current training crawler.
A crawler follows only the most specific group that matches its name. If you add a User-agent: ClaudeBot group, it stops reading your User-agent: * rules entirely. Repeat any rules you still want it to follow inside its own group.
robots.txt is a request, not an access control. Reputable crawlers honour it; to enforce a block, use your CDN or firewall. Changes are usually picked up within about a day, because crawlers cache robots.txt.
Check it
Run your domain through the checker. It shows whether ClaudeBot and 11 other AI crawlers are allowed or blocked, and which line decides it.