Collect the URL versions that need comparison
Find duplicates that a simple navigation crawl may miss.
Use this for: Sites with multiple routes, parameters or suspected repeated content.
Combine a current page crawl with your CMS page list or sitemap. Add known alternate hosts, protocol versions, old routes, print pages and parameter examples that are relevant to your site. Search Console duplicate indexing reasons can suggest further candidates; they are not a complete inventory.
Keep the review bounded. You need representative examples of each pattern, not an unlimited crawl of every filter combination. Compare excluded or unfetched URLs separately so they do not disappear from the conclusion.
Hypothetical example: A shop crawl finds /products/clay-pot/, while the CMS also exposes it beneath /sale/clay-pot/. Add that second route to the comparison.
- Gather the public page inventory and known alternate address patterns.
- Add candidate pairs from sitemap, CMS and indexing reports where available.
- Mark which patterns were fetched and which still need access or investigation.
More help and optional notes
If it fails: Add representative missing variants and fetch their actual content.
Retest: Check that each known pattern has a comparable page pair or an explicit unresolved access state.
Not applicable: No public or ready-to-test content exists in scope.
Keep in mind: Blocked: You cannot obtain the page inventory or access the suspected variants. Passing this discovery task does not mean the site is free of duplicates.
Optional: the page or URL pair you checked, what happened and the next action. Keep confidential data out of shared exports.
Changing this result updates your checkmark and progress immediately. Notes are optional.

