Guides
AI crawlers
How to block GPTBot in robots.txtHow to block ClaudeBot in robots.txtHow to block Google-Extended in robots.txtHow to block PerplexityBot in robots.txtHow to block CCBot in robots.txtHow to block Applebot-Extended in robots.txtHow to block Bytespider in robots.txtHow to block meta-externalagent in robots.txtrobots.txt template to block AI crawlers (and keep search engines)
robots.txt basics
robots.txt syntax: every directive explainedAllow vs Disallow: which rule wins in robots.txt?robots.txt wildcards: how * and $ workNoindex in robots.txt doesn't work: what to do insteadCrawl-delay in robots.txt: who honours itWhat happens when robots.txt returns 404, 403 or 500
Search Console errors
Fix 'Indexed, though blocked by robots.txt'Fix 'Blocked by robots.txt' in Google Search ConsoleFix 'Sitemap could not be read' / 'Couldn't fetch'
Sitemaps
XML sitemap format: a minimal valid exampleSitemap index files: when and how to split sitemapsSitemap limits: 50,000 URLs and 50 MBlastmod, priority and changefreq: what Google actually usesImage sitemaps: syntax and when you need oneWhere to put your sitemap and how to submit it