Guides › AI crawlers
robots.txt template to block AI crawlers (and keep search engines)
This template blocks the main AI training and data crawlers we know of and leaves normal search crawling alone. Several User-agent lines in a row share the rules that follow them.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent User-agent: Amazonbot Disallow: / User-agent: * Allow: /
What this keeps working
- Google Search (Googlebot) and Bing (Bingbot) crawling and ranking.
- Link previews (facebookexternalhit, Twitterbot, Slackbot, LinkedInBot), since those aren't listed.
- AI search agents that you might want for referral traffic, such as OAI-SearchBot and PerplexityBot. Add them to the list if you want to opt out of those too.
Trade-offs
Blocking Google-Extended or Applebot-Extended only opts you out of model training; it doesn't affect search. New crawlers appear regularly, so review this list every few months. Our monitoring plan re-checks your robots.txt daily and tells you when it changes.
Check your site now