Guides › robots.txt basics
Crawl-delay in robots.txt: who honours it
Crawl-delay: 10 asks a crawler to wait 10 seconds between requests.
| Crawler | Honours Crawl-delay? |
|---|---|
| Googlebot | No. Google adjusts crawl rate automatically and slows down when your server returns 500, 503 or 429. |
| Bingbot | Yes (Bing Webmaster Tools also has crawl control) |
| YandexBot | Historically yes. Yandex now recommends the crawl-rate setting in Yandex Webmaster. |
Slowing Google down
To slow Google down temporarily, return HTTP 503 or 429 to Googlebot during overload. Google reduces its crawl rate in response. Don't do this for more than a day or two, or pages may drop out of the index.
Large crawl-delay values can stop a crawler covering a big site at all (10 s means at most 8,640 URLs per day), so use small values.
Check your site now