Guides › AI crawlers
How to block Google-Extended in robots.txt
Google-Extended is a robots.txt control token, not a separate crawler. Google's normal crawlers still fetch your pages, and Google-Extended controls whether that content may be used to train and ground Gemini models.
Block Google-Extended from your whole site
User-agent: Google-Extended Disallow: /
Block only part of your site
User-agent: Google-Extended Disallow: /blog/ Disallow: /docs/private/
Notes
Important: disallowing Google-Extended does not affect Google Search crawling, indexing or ranking. To block Search, you'd have to block Googlebot, which almost nobody should do. Because Google-Extended is a token, you won't see it in your server logs. Fetches still appear as Googlebot.
A crawler follows only the most specific group that matches its name. If you add a User-agent: Google-Extended group, it stops reading your User-agent: * rules entirely. Repeat any rules you still want it to follow inside its own group.
robots.txt is a request, not an access control. Reputable crawlers honour it; to enforce a block, use your CDN or firewall. Changes are usually picked up within about a day, because crawlers cache robots.txt.
Check it
Run your domain through the checker. It shows whether Google-Extended and 11 other AI crawlers are allowed or blocked, and which line decides it.