Guides › robots.txt basics
Allow vs Disallow: which rule wins in robots.txt?
When several rules in a group match a URL, Google (and RFC 9309) use the most specific rule, meaning the one with the longest path. If an Allow and a Disallow are equally long, Allow wins. The order of lines in the file doesn't matter.
User-agent: * Disallow: /shop/ Allow: /shop/sale/
| URL | Matching rules | Result |
|---|---|---|
| /shop/shoes | Disallow: /shop/ (6) | Blocked |
| /shop/sale/boots | Disallow: /shop/ (6), Allow: /shop/sale/ (11) | Allowed (longer) |
| /about | none | Allowed |
Equal-length tie
User-agent: * Disallow: /page Allow: /page
/page is allowed, because Allow wins the tie. With wildcards the "length" is the length of the pattern as written, so /*.pdf counts as 6 characters.
Other crawlers
Older or simpler crawlers may use first-match-wins instead. If you need consistent results everywhere, avoid overlapping rules where you can. The checker shows which line decided each crawler's verdict, using Google's longest-match logic.
Check your site now