Guides › robots.txt basics
What happens when robots.txt returns 404, 403 or 500
| robots.txt response | Google's behaviour |
|---|---|
| 200 OK | Rules are parsed and followed. |
| 3xx redirect | Follows up to 5 hops. After that it's treated as a 404. |
| 4xx (404, 403, 410…) | Treated as if no robots.txt exists, so everything may be crawled. (429 is treated like a 5xx.) |
| 5xx / timeout | Treated as full disallow, so crawling stops. Google uses a cached copy for up to 30 days. If the file stays unreachable for longer, Google treats the site as having no restrictions as long as the site itself is reachable. |
Why this matters
A misconfigured firewall or WAF that returns 403 or 503 to bots on /robots.txt is a common silent problem. A 503 halts crawling site-wide, and a 403 quietly removes every restriction you set. The checker fetches your robots.txt live and flags both. The monitoring plan re-checks daily, because these errors often appear after a CDN or hosting change.
Check your site now