HTTPS and status codes
Important URLs should resolve consistently over HTTPS and return the status you expect. Avoid unnecessary redirect chains. A URL returning an error cannot serve as a normal indexed document.
robots.txt
robots.txt provides crawler instructions at the origin root. It is useful for crawl control but is not an access-control mechanism. Never use it to protect private information.
XML sitemaps
An XML sitemap lists URLs you want crawlers to discover. Keep it accurate: include canonical, indexable URLs and remove obsolete or redirected entries.
Canonical URLs
The canonical link element identifies the preferred URL for substantially duplicate or equivalent pages. Use absolute, consistent URLs and avoid unrelated canonical targets.
Indexability
Check noindex, robots directives, response codes and canonical signals together. A page can be reachable while still being deliberately excluded from indexing.
Internal linking
Important pages should be reachable through normal links. Use descriptive anchor text and a sensible information architecture.
Migration checklist
- Map old URLs to intended new URLs.
- Test representative redirects.
- Check canonicals after launch.
- Submit the new sitemap.
- Monitor indexing and errors after the change.
Run the Ingenix SEO Checker against representative URLs before and after a migration.