Guides › robots.txt basics
Noindex in robots.txt doesn't work: what to do instead
Since 1 September 2019, Google ignores Noindex:, Nofollow: and Crawl-delay: lines in robots.txt. They were never part of the standard.
Use one of these instead
- Meta tag in the page's
<head>:<meta name="robots" content="noindex"> - HTTP header (works for PDFs and images too):
X-Robots-Tag: noindex - Password protection or a 404/410 status for content that should disappear.
- Search Console Removals tool for urgent, temporary hiding (about 6 months).
The classic mistake
Don't disallow a page in robots.txt and add noindex. If Google can't crawl the page, it never sees the noindex. If other sites link to it, the URL can stay indexed with no snippet ("Indexed, though blocked by robots.txt"). Allow crawling until the page drops out, then block it if you need to.
Check your site now