Guides › AI crawlers
How to block GPTBot in robots.txt
GPTBot is OpenAI's crawler that collects publicly available pages that may be used to train OpenAI's generative AI models.
Block GPTBot from your whole site
User-agent: GPTBot Disallow: /
Block only part of your site
User-agent: GPTBot Disallow: /blog/ Disallow: /docs/private/
Notes
OpenAI runs separate user agents for separate jobs. OAI-SearchBot is used for ChatGPT search results, and ChatGPT-User makes fetches a user asks for inside ChatGPT. Blocking GPTBot doesn't block the other two. If you want to stay visible in ChatGPT search but opt out of training, block GPTBot and allow OAI-SearchBot:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /
OpenAI documents its crawlers and their IP ranges at platform.openai.com/docs/bots.
A crawler follows only the most specific group that matches its name. If you add a User-agent: GPTBot group, it stops reading your User-agent: * rules entirely. Repeat any rules you still want it to follow inside its own group.
robots.txt is a request, not an access control. Reputable crawlers honour it; to enforce a block, use your CDN or firewall. Changes are usually picked up within about a day, because crawlers cache robots.txt.
Check it
Run your domain through the checker. It shows whether GPTBot and 11 other AI crawlers are allowed or blocked, and which line decides it.