Crawlability Checklist

Use this for: Important public pages intended to be discoverable.

List the important published service, product and guide URLs in your website editor. For a small site, start at the homepage and follow its relevant category, hub and contextual links. Compare the pages you can reach with your list; do not type a destination into the address bar to count it as internally linked.

For a larger site, compare that list with your crawler’s homepage-started results. A missing page is a candidate orphan, not a confirmed one: exclusions, JavaScript settings or an incomplete crawl can hide an existing link.

Open each candidate and its intended parent page. If there is no useful link path, add a descriptive link in the relevant hub, category or related guide. A sitemap can help discovery but does not create that navigation path.

1. Export or list the important published URLs in your website editor.
2. From the homepage, follow the relevant hub/category links and compare reachable pages with your list; use a crawler for a larger review.
3. Add the missing contextual link and follow it from that parent; rerun the affected crawl if you used one.

Use this for: Menus, cards and contextual links to public pages.

Right-click an important navigation or card link and choose Inspect. The link should use an anchor element with an href that resolves to the intended page, such as <a href="/services/roof-repair/">Roof repair</a>. A button whose script changes pages may work for a person without supplying a reliable crawlable link.

Ask the developer to retain that real destination in the component. JavaScript can insert crawlable anchors; confirm that the resulting link exists in rendered HTML instead of assuming all JavaScript links fail.

1. Inspect the menu or card element and locate its href.
2. Open the destination directly and verify that it resolves to the intended page.
3. Replace script-only navigation and inspect the rendered link again.

Use this for: Collections using pagination, Load more or infinite scroll.

Open a category or article archive and check how later items appear. If they require a click or scroll, provide linked paginated URLs that can load those items directly. Use real next-page links and distinct addresses such as /guides/?page=2, not a fragment-only change.

Open a later URL in a fresh tab before scrolling. It should display the intended items and link onward where needed. Google no longer uses rel="next" and rel="prev" tags; those tags do not replace working links.

1. Identify an item that is absent from the first collection view.
2. Find and open the later collection URL directly.
3. Add crawlable pagination when no reliable linked path exists, then follow it to the item.

Use this for: Sites using an XML sitemap.

Open the sitemap address from your CMS, robots.txt declaration or Search Console Sitemaps report. If it is a sitemap index, open the child file for the page type you are checking. Search for the exact preferred URL and open it.

Repair an inaccessible file, broken child reference or missing published page in the sitemap generator. Keep redirects, deleted addresses and intentionally excluded pages out of the discovery list. A successfully read sitemap does not prove its listed pages were crawled or indexed.

1. Open the sitemap and relevant child sitemap without signing in.
2. Find a recently published preferred URL and follow it.
3. Correct generator membership or file access and check the new output.

Use this for: Public pages that search crawlers should fetch.

Open /robots.txt on the same protocol and host as the affected page. Match the crawler’s applicable group and the full URL path against its Allow and Disallow rules. Reading a broad Disallow line alone is insufficient when a more specific matching rule changes the result.

If an important public path is accidentally blocked, have the technical owner narrow the rule. Preserve intended exclusions and test both kinds of URL. A robots block controls fetching; it does not reliably remove an address from search results.

1. Check the exact page host and its root robots.txt file.
2. Evaluate the matching crawler group and path rules.
3. Correct the accidental block, then test a wanted page and an intentionally blocked path.

Use this for: Pages deliberately available to the public.

Open the public URL in a private browser window. In Chrome Developer Tools → Network, reload and select the main document. Check Headers → Status Code and read the Response or displayed content. A page meant to be public should return its real content without login, challenge or error screens.

Give an unexpected 401, 403, 429 or 5xx response and its time to the site owner. The fix may belong to the route, application, host or access policy. Keep genuine private pages protected.

1. Load the exact URL in a signed-out or private session.
2. Inspect the document request, response status and returned content.
3. Repair the unintended access failure and reload under the same conditions.

Use this for: Internal links or old URLs that redirect.

Enable Preserve log in Chrome’s Network panel, then follow an internal link to the affected page. Check each document request and the final address. A loop or a chain ending at an unrelated page prevents the intended fetch even if the last response is 200.

Update internal links to the final preferred address. Have the developer simplify obsolete hops and use redirects that reflect the actual move. Keep a real not-found response for removed content with no suitable replacement.

1. Follow the actual internal link with Network and Preserve log enabled.
2. Read the redirect sequence and final page.
3. Correct the mapping and update the referring link to the intended destination.

Use this for: Sites with Search Console data, especially widespread or recurring fetch errors.

In Search Console → Settings → Crawl stats, open Host status and the response breakdown. Check whether robots.txt availability, DNS resolution or server connectivity failed around the affected requests. Open available example URLs and compare their times with hosting logs or incident records.

Ask the host or developer to repair the cause, such as a failed application process or repeated rate limiting. Use the site’s own trend and error pattern; there is no universal response-time cutoff that proves a crawl-budget failure.

1. Open Crawl stats and review Host status for the affected host.
2. Compare response errors and example times with hosting records.
3. Repair the confirmed availability cause and check fresh requests.

Use this for: Pages whose useful content or navigation depends on external resources or JavaScript.

Run a live URL test in Search Console and, when available, open View tested page. Compare its rendered HTML, screenshot and resource details with the intended page. Focus on a missing main section or navigation link, then identify the failed or blocked resource that supplies it.

Have the developer fix that resource’s URL, access rule or delivery error. A failed advertising request may be unrelated; do not treat every resource warning as missing main content.

1. Inspect the rendered page and resource details from a live test.
2. Connect each meaningful omission to a failed resource or rendering error.
3. Repair the dependency and repeat the rendered-page check.

Use this for: Single-page apps and sites with client-side routing.

Copy an important application URL into a fresh tab and reload it. It should load that specific page from a direct request, with a real path or query URL that links can expose. A view that exists only after clicking from the homepage can fail when a crawler requests its address.

Ask the developer to configure route handling and use addressable navigation rather than fragment-only content changes. Compare the directly loaded route with the in-app destination.

1. Copy the exact destination from the application navigation.
2. Open it in a new private tab and reload without visiting the homepage first.
3. Repair route handling and confirm both direct loading and incoming links.

Use this for: Sites with faceted filters, calendars or other expanding URL spaces.

For a small review, open a category and apply its filters one at a time, noting how the address changes. Try the same filters in a different order and look for redundant URLs or endlessly repeated parameters. For a larger review, group those patterns in your crawler’s URL list or server logs.

Decide which combinations deserve a public search landing page. Use the homepage-to-category link check above to confirm that products or articles still have ordinary category and pagination paths.

For unwanted crawl spaces, ask the developer to stop generating unnecessary crawlable combinations or apply a tested crawl-control plan. A canonical tag is not an immediate crawl block, and robots.txt does not reliably remove an already known URL from search.

1. Find repeating parameter or calendar patterns in discovered URLs.
2. Separate useful landing pages from redundant combinations and check the underlying item paths.
3. Apply the agreed control to the unwanted pattern and recrawl representative allowed and restricted URLs.

Use this for: Sites investigating crawler-specific access failures.

Ask the hosting or security owner for requests to the affected path, including time, response code and client IP. A user-agent that says Googlebot is only a claim. Verify the IP against the appropriate published Google crawler ranges or perform reverse DNS followed by forward DNS confirmation.

Compare verified requests with the firewall or CDN event at the same time. If a rule challenged or denied a legitimate request for public content, have the owner correct that specific rule and review fresh traffic. Changing your browser’s user-agent does not reproduce a verified Google request.

1. Obtain a relevant request and its client IP from server or edge logs.
2. Verify the claimed crawler using the published verification method.
3. Investigate confirmed blocks and check fresh verified requests after any targeted fix.

Use this for: Important URLs undergoing a crawlability review.

Inspect the exact page address in Search Console and choose Test live URL. Expand availability details and check Crawl allowed? and Page fetch. If access fails, use that named reason to revisit the robots, host or route check.

Keep the result’s scope narrow: a live fetch is a current diagnostic, not proof that ordinary Googlebot has already revisited the page. If Bing matters to this review, check the same URL in Bing Webmaster Tools separately; its Live Check and indexed information answer different questions.

1. Open URL Inspection for the exact public URL and start a live test.
2. Read crawl permission and fetch details, then investigate a named failure.
3. Repeat the live test after fixing access and keep its date separate from the indexed record.

Use this for: Repairs affecting shared navigation, routing or crawl rules.

After changing a shared menu, route, robots rule or resource policy, repeat the original failed path from the same starting page. Then test a neighboring public page and a deliberately restricted control URL touched by that rule. This checks whether a narrow repair accidentally broke another route or opened a path that should remain restricted.

Close the access fix only when the required behavior is visible on the published site. Keep a dated fetch result separate from any later search-index update; the findings log below has room for both.

1. Repeat the original discovery or fetch test against the published change.
2. Test another affected public URL and one deliberately restricted control.
3. Correct regressions and confirm all expected access results.