robots.txt
robots.txt is a crawler-control file at the root of an origin, such as https://example.com/robots.txt. It tells compliant crawlers which paths they should avoid requesting. It is not a security boundary.
XML sitemaps
An XML sitemap lists URLs you want crawlers to discover. It is useful for large sites, new sites and complex internal structures. Keep it focused on canonical, indexable URLs.
They solve different problems
robots.txt controls crawler access to paths; a sitemap provides URLs for discovery. Avoid listing URLs in a sitemap while simultaneously blocking them unless you understand the consequences.
Common mistakes
- Blocking the entire site with
Disallow: /. - Assuming robots.txt removes a URL from search results.
- Listing redirected or non-canonical URLs in the sitemap.
- Forgetting to update the sitemap after a migration.
- Using robots.txt to protect private information.
Migration checklist
- Build the sitemap from intended canonical URLs.
- Review robots.txt before launch.
- Test representative URLs.
- Confirm redirects and canonicals.
- Monitor indexing after launch.
Use the SEO Checker to check crawlability and related Lighthouse findings.