26 guides

Crawling & Indexing

Understand how pages are discovered, which technical directives they expose and where crawl paths or indexing signals need attention. Use this group when pages disappear from search, a migration changes URLs, or a site has crawl and indexability uncertainty.

Problem fixes and how-to guides

Fix guidePage not indexed by Google: how to find the cause and fix itWork through access, directives, canonicalization and quality in that order. Most pages are not indexed because of a blocking…Fix guideHow to find and remove an unwanted noindex tagA noindex directive can come from a meta tag or an X-Robots-Tag HTTP header. Find the source, remove it from the template and…Fix guideBlocked by robots.txt: how to unblock important pagesIf a rule in robots.txt matches an important URL, crawlers will not fetch it. Find the matching rule, narrow or remove it, and…Fix guideSitemap could not be read or has errors: how to fix itCheck that the sitemap URL loads with 200, is valid XML, lists only canonical URLs and stays within size limits, then resubmit it.Fix guideWrong canonical URL: fix Duplicate, Google chose different canonicalA canonical tag is a hint. Make internal links, redirects, sitemap entries and the canonical tag all agree on one preferred URL.Fix guideHreflang errors: how to fix language and region targetingEach language version must list all alternates, including itself, and every alternate must link back. Use valid language and…Fix guideOrphan pages: how to find and fix pages with no internal linksOrphan pages have no internal path from the rest of the site. Find them by comparing crawl results with your sitemap, then link…Fix guideDuplicate title tags and meta descriptions: how to fix themGive each page a title and description that describe its own purpose. Prioritize pages that target different topics but share…Fix guideFaceted navigation and URL parameters: stop crawl wasteFilters and sort options can create thousands of near-duplicate URLs. Decide which combinations deserve to be indexed, then…GuideCrawl budget for small websites: what actually mattersFor most small and medium sites, crawl budget is not the first problem to solve. Make important URLs discoverable, fast enough…GuideHow to analyze Googlebot crawl logsServer access logs show which URLs a crawler requested, when it requested them and what response the server returned. They help…GuideJavaScript SEO: how to check rendered page contentA page can return HTML successfully while important content or links appear only after JavaScript runs. Compare the initial…GuideCanonical tags vs redirects: when to use eachA redirect sends visitors and crawlers to another URL. A canonical annotation signals a preferred version while leaving the…GuideInternational SEO: plan language and regional URL variantsInternational SEO requires clear URL targeting, language signals and useful localized content. Hreflang annotations help…

Tool guides