Guides › Search Console errors

Fix 'Blocked by robots.txt' in Google Search Console

"Blocked by robots.txt" means Googlebot found the URL but a Disallow rule stopped it from being crawled, so the page isn't indexed.

Steps

1. Enter your domain and the blocked path in the checker. It shows the exact rule and line number deciding Googlebot's verdict.
2. Check for over-broad prefixes: Disallow: /p blocks /products, /pricing and everything else starting with /p.
3. Check for a staging-site leftover: Disallow: / under User-agent: *, which is often shipped by accident when a site launches.
4. Fix the rule, then open Search Console › Settings › robots.txt and request a recrawl of the file. Use URL Inspection › Request indexing for key pages.

Google retired the old robots.txt Tester in late 2023. The Settings › robots.txt report shows the version Google last fetched, along with any fetch errors.

Related