Guides › robots.txt basics
robots.txt wildcards: how * and $ work
Google, Bing and RFC 9309 support two special characters in Allow/Disallow paths:
*matches any sequence of characters, including none.$at the end means "the URL ends here".
| Rule | Blocks | Doesn't block |
|---|---|---|
Disallow: /*.pdf$ | /files/a.pdf | /files/a.pdf?dl=1 |
Disallow: /*.pdf | /a.pdf, /a.pdf?dl=1, /a.pdfx | /a.PDF (case-sensitive) |
Disallow: /*? | any URL with a query string | /page |
Disallow: /*?sort= | /shoes?sort=price | /shoes?color=red&sort=price (the parameter follows &, not ?) |
Disallow: /*sort= | sort= in any position | /sorted-list is still allowed, because there's no "=" |
Disallow: /blog/$ | /blog/ only | /blog/post-1 |
Trailing * is redundant
Rules are prefixes already, so Disallow: /private* behaves exactly like Disallow: /private.
Test your patterns against a real path with the checker: enter a path and see the verdict for each crawler.
Check your site now