Guides › robots.txt basics
robots.txt syntax: every directive explained
robots.txt is a plain-text file served at the root of a host (https://example.com/robots.txt). It applies only to that exact protocol, host and port. blog.example.com needs its own file. The format is standardised as RFC 9309 (2022).
| Directive | Meaning | Supported by |
|---|---|---|
User-agent | Starts a group: which crawler(s) the following rules apply to. * means any crawler without a more specific group. | All |
Disallow | Path prefix the crawler shouldn't fetch. An empty value means nothing is disallowed. | All |
Allow | Path prefix explicitly allowed. Used to carve exceptions out of a Disallow. | Google, Bing, most modern crawlers (RFC 9309) |
Sitemap | Absolute URL of a sitemap. Not tied to a group; it can appear anywhere. | Google, Bing, Yandex, others |
Crawl-delay | Seconds between requests. | Bing, Yandex. Ignored by Google. |
Noindex | Not a valid robots.txt rule. | Google dropped unofficial support in Sept 2019 |
A complete example
# Default rules User-agent: * Disallow: /cart/ Disallow: /search Allow: /search/help # Opt out of AI training User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
Rules that trip people up
- Paths are case-sensitive:
Disallow: /Admindoesn't block/admin. - Rules match by prefix:
Disallow: /blogalso blocks/blog-postand/blogroll. Use/blog/to target the folder. - A crawler uses only the most specific matching group and ignores
*once it has its own group. - Rules before the first
User-agentline are ignored. - Google reads only the first 500 KiB.
See also: Allow vs Disallow precedence and wildcards.
Check your site now