Technical SEO · Supporting checklist

Crawl Budget Audit Checklist

A crawl budget audit checklist is an advanced technical SEO workflow for finding whether unnecessary URL spaces or serving limits delay useful crawling. Use it mainly for large or rapidly changing inventories with a real symptom. A small, stable website usually needs basic crawlability and indexing checks first. Compare crawl demand, host capacity and observed requests, then test a focused change without chasing a universal crawl target.

Free to useNo account neededSaved in your browser
A bounded stream of crawler request tokens reaches grouped inventories of paper page tiles.
Review large-site URL groups against observed crawl demand and capacity.
Your working checklist

Work through the checklist

Click a square to mark a task as Pass. Click again to undo. Use the result menu for a problem, a blocker or a task that does not apply. You do not need to enter notes.

14 checks

Decide Whether Crawl Budget Needs Attention

Crawl budget work is an advanced diagnostic for a large or rapidly changing URL inventory with a real crawl problem. First decide whether the symptom is delayed fetching, an indexing decision or a rendering failure.

Check whether a crawl-budget audit fits the problem

Spend effort where crawl allocation could be limiting useful discovery or refresh.

Use this for: Sites considering advanced crawl-budget work.

Compare the size and publishing pace of your site with the actual symptom. Look for important new or changed pages whose fetching is repeatedly delayed, a rapidly expanding URL inventory, or host strain during crawling. A low number of requests alone is not a defect.

A stable 32-page site normally needs working links, a current sitemap and page-level indexing checks first. Targeted diagnostics can still help when a particular page fails. If you have no evidence of an inventory or capacity problem, mark the remaining advanced checks Not applicable and return to ordinary technical maintenance.

Example

Hypothetical example: A service site has 32 linked pages and its updates are fetched promptly. It does not need a target of hundreds of Googlebot requests per day.

  1. Compare page count and publishing pace with the reported problem.
  2. Check a few affected pages and a few healthy pages.
  3. Continue only where a crawl-allocation or host-capacity question remains.
More help and optional notesHigh · Website owner or SEO lead
PassThe reason to investigate is supported by a specific symptom, or the advanced audit is reasonably set aside.
FailThe audit assumes every site needs more crawling without establishing a relevant problem.

If it fails: Start with the affected page and determine which stage fails before proposing site-wide changes.

Retest: Revisit the decision if the inventory, publishing pace or observed crawl behavior changes.

Not applicable: This initial suitability decision applies to every crawl-budget audit.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Separate delayed crawling from indexing and rendering

Choose the diagnostic that matches what actually failed.

Use this for: Important pages reported as missing, stale or excluded from search.

In Search Console, inspect an affected URL and read its last crawl, page fetch and indexing details. Compare that timestamp with the publication or meaningful update time. If the page has already been fetched, an indexing exclusion is not proof that more crawl capacity would solve it.

Open the fetched or tested page details when available. Missing main content after JavaScript runs is a rendering question; an unwanted noindex or competing canonical is an indexability question. Treat Discovered - currently not indexed as a lead to investigate, not a confirmed diagnosis of an exhausted budget.

Example

Hypothetical example: A newly updated category was fetched yesterday but declares a different canonical. Investigate the canonical decision before buying more server capacity.

  1. Inspect an affected URL and its last crawl.
  2. Compare fetching, rendered content and indexing details with the intended outcome.
  3. Send the finding to the workflow that owns that stage.
More help and optional notesHigh · SEO lead and developer
PassThe next action addresses the observed stage, with unknown stages left unresolved.
FailA successful crawl or rendering error is mislabeled as a crawl-budget shortage.

If it fails: Diagnose the specific restriction, content or rendering issue before changing crawl controls.

Retest: Inspect the same URL after the targeted repair; keep live-test and indexed records separate.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Choose a hostname and a comparable observation window

Make request comparisons describe the same inventory and period.

Use this for: Any audit comparing request volume, freshness or serving behavior.

Choose the hostname, URL groups and bot type relevant to the problem. List the publication, migration and outage dates that could change demand. Use a period long enough to include the normal publishing and crawling pattern you are investigating; there is no universal minimum that proves a problem.

Check that the Search Console property and log export cover those hosts and dates. Label retention gaps and sampled data. Compare equivalent weekdays and release conditions where practical instead of treating a launch spike and an ordinary week as interchangeable.

Example

Hypothetical example: A news archive and a shop live on different hostnames. A combined traffic chart cannot show which host experienced a crawl slowdown after the shop release.

  1. Choose the host, bot category and affected URL group.
  2. Check the dates and completeness of available data.
  3. Mark releases or outages before comparing periods.
More help and optional notesHigh · SEO analyst
PassThe compared data has known host, bot, date and coverage boundaries.
FailDifferent hosts, partial periods or bot categories are mixed into one conclusion.

If it fails: Refilter the reports and label any remaining gaps before interpreting changes.

Retest: Repeat the comparison using the same scope after a correction.

Not applicable: No URLs in the stated scope.

Keep in mind: A bounded sample can reveal a pattern without establishing a site-wide rate.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Find Which URL Groups Receive Crawls

Crawl demand concerns what a crawler wants to fetch; crawl capacity concerns what the host can serve. Compare requests with the purpose and change rate of each URL group before calling them waste.

Read Crawl Stats as a request baseline

Find the dates and response groups worth investigating.

Use this for: Sites with access to the relevant Search Console Crawl Stats report.

In Search Console, open Settings → Crawl stats for an eligible root-level property. Review request volume, response time, host status and the response, file-type, purpose and bot breakdowns. Open an unusual day or group to see examples.

Requests are not distinct pages. Repeat fetches, resources and redirect hops can all contribute. The example URL list is incomplete, so absence from it does not mean a page was never requested. Use the report to choose a narrower investigation.

Example

Hypothetical example: A script deployment increases resource requests while HTML requests stay steady. The overall spike does not mean more product pages were discovered.

  1. Open the relevant host and time period.
  2. Compare HTML, resource, response and bot groups.
  3. Choose the anomalous date or group for a closer check.
More help and optional notesHigh · SEO analyst

Before you begin: Access to an eligible root-level Search Console property.

PassThe baseline identifies a specific pattern without equating total requests with crawled pages.
FailThe interpretation ignores request types or treats example URLs as a complete inventory.

If it fails: Use the matching breakdown and request-level data where available.

Retest: Reopen the same breakdown after the next relevant publishing or repair cycle.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Confirm suspect request groups in verified bot logs

Check whether the crawl pattern is present in reliable request data.

Use this for: A Crawl Stats or freshness pattern that needs request-level explanation.

Ask the site administrator for an existing access-log view covering the affected host and dates. Work within the approved local or internal environment. Filter the claimed bot, then verify its client IP using the operator’s published method before calculating crawler totals.

Confirm whether the view records CDN/edge requests, origin requests or both. A cached response may never reach the origin, and adding both layers can count a request twice. If the available data cannot establish the bot or coverage, keep that limit attached to the finding.

Example

Hypothetical example: The origin log shows few Googlebot rows because a CDN serves most category requests. The edge log changes the apparent crawl distribution.

  1. Select the host and dates in an existing log view.
  2. Verify the bot and identify the logging layer.
  3. Group the verified requests by the URL patterns you are investigating.
More help and optional notesHigh · SEO analyst and site administrator
PassThe analyzed requests have verified bot identity and a stated logging scope.
FailUnverified user-agent strings or mixed logging layers are presented as Googlebot totals.

If it fails: Correct bot verification and logging scope before using the totals to prioritize work.

Retest: Rerun the same filters on the corrected data.

Not applicable: No URLs in the stated scope.

Keep in mind: A browser request with a Googlebot user agent is not a verified Googlebot visit.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Find URL patterns that keep creating new combinations

Locate templates that generate many crawlable variants without a distinct search purpose.

Use this for: Faceted navigation, internal search, calendars or other expanding URL spaces.

In your crawler or verified-log summary, group requests by path and parameter names. Look for repeated sort/filter combinations, session identifiers, calendars extending indefinitely or changing tracking values. Count distinct requested URLs as well as requests for each pattern.

Open representative variants and use the interface that creates them. Decide which combinations serve a useful public search page and which only reorder or repeat the same material. A question mark in an address does not make the page wasteful.

Example

Hypothetical example: A furniture category can combine color, width and sort order. Useful oak-desk listings differ from dozens of sort-order permutations of the same products.

  1. Group exact requested URLs by generating template or parameter family.
  2. Open examples and identify how new variants are created.
  3. Separate useful destinations from redundant or endless combinations.
More help and optional notesHigh · SEO lead and developer
PassThe reviewed expansion patterns are correctly characterized with concrete examples, useful exceptions and any defects identified for repair.
FailThe review misclassifies useful variants or cannot explain how the suspect URL pattern is generated.

If it fails: Change the generating links or routing behavior for the unwanted pattern after testing wanted exceptions.

Retest: Repeat the interface actions and check that they no longer expose the unwanted new variants.

Not applicable: The site has no parameterized or otherwise expanding URL space.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Compare crawl frequency with real publishing needs

Find important page groups whose updates are not being fetched promptly enough for the business.

Use this for: A suspected discovery or refresh delay with publication records and usable logs.

Choose a small cohort of newly published or meaningfully updated pages. From the CMS, get their release times; from verified logs, find the first observed subsequent request. Compare those delays across equivalent templates and periods.

Use first observed request, not first-ever crawl, unless your data truly covers the earlier history. A stable information page may reasonably receive fewer refreshes than changing stock or event listings. Judge the pattern against the page’s update needs, not one desired crawl rate for the whole site.

Example

Hypothetical example: Daily-changing event pages wait longer for an observed fetch than the previous cohort, while a rarely edited terms page is fetched less often as expected.

  1. Select new or changed URLs with known publication times.
  2. Find their first observed verified fetches in the chosen window.
  3. Investigate repeated delays in important changing groups.
More help and optional notesHigh · SEO analyst and content team
PassThe reviewed important changed pages receive observed fetches within the practical freshness needs chosen for this cohort.
FailImportant changed pages repeatedly miss the site’s practical freshness needs and the cause remains untreated.

If it fails: Check discovery paths, URL expansion and host health for that group before choosing a repair.

Retest: Compare the next similar publishing cohort under the same observation method.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Give important new pages stable discovery paths

Remove avoidable discovery gaps for the affected page group.

Use this for: Wanted new or changing pages with weak discovery paths.

Open the relevant category or topic hub and follow its links to a newly published page. Check that it links to the preferred live URL through a real anchor with an href. In the sitemap output, look for that same preferred address and a truthful lastmod when your generator provides one.

Fix the template or publishing rule that omitted the link or sitemap entry. Do not add a link from every page or impose a universal click-depth target. The useful change is a reliable route from related content that visitors can understand.

Example

Hypothetical example: New workshop pages enter the CMS but the events hub lists only archived items. Repairing that listing gives each new workshop a relevant discovery path.

  1. Follow the normal hub-to-page navigation.
  2. Compare the target with the sitemap’s preferred address.
  3. Repair the listing or publishing rule and check another new page.
More help and optional notesHigh · SEO lead and developer
PassThe sampled pages have relevant crawlable links and accurate intended sitemap membership.
FailThe publishing process leaves wanted pages isolated or exposes obsolete variants.

If it fails: Update the hub/listing template and sitemap generator to use the intended public URLs.

Retest: Publish or inspect another representative page and repeat the same discovery route.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Check Host Capacity and Choose Focused Repairs

Host capacity problems need serving fixes; unnecessary URL spaces need inventory decisions. Use the observed bottleneck to choose a change, then check both the intended benefit and possible regressions.

Investigate availability failures on the affected dates

Determine whether the host was unable to serve crawler requests.

Use this for: A crawl slowdown accompanied by availability or capacity symptoms.

Compare Crawl Stats host-status warnings with the administrator’s incident timeline and verified request records. Look for DNS/connectivity failures, recurring 5xx responses or 429 rate-limit responses near the slowdown. A failed connection may leave no application access-log row.

Give the administrator the affected time, host and public URL group. Ask them to identify the failing component and restore normal service. Do not disable protection rules or remove authentication to make a chart greener.

Example

Hypothetical example: A product import exhausts the database each morning. Verified requests receive 503 responses during the same period; the import and serving capacity need attention.

  1. Match the crawl slowdown with host warnings and incident times.
  2. Identify the actual failing request layer.
  3. Have the administrator repair the serving problem and confirm recovery.
More help and optional notesHigh · SEO lead and developer
PassThe affected public routes serve normally after the identified availability problem is resolved.
FailRecurring host failures still interrupt requests to the affected public pages.

If it fails: Repair the confirmed application, hosting or delivery failure through the responsible administrator.

Retest: Check fresh requests and the relevant host report after recovery.

Not applicable: No host-health or request-failure symptom is present.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Investigate slow templates and repeated unchanged transfers

Find serving work that can be reduced without hiding useful content.

Use this for: Slow public templates or repeated large unchanged responses with a demonstrated serving cost.

In the existing log or monitoring view, compare response duration and bytes within the same template and cache state. Ask the developer to reproduce the slow query or resource, especially when slowdowns coincide with host warnings. A site-wide average can hide one expensive endpoint.

For unchanged public content, ask whether the application supports correct cache validation with ETag or Last-Modified and conditional 304 responses. Do not force 304 responses for changed pages. Required JavaScript and CSS must remain available for rendering.

Example

Hypothetical example: One filtered listing runs an expensive database query on every request. Caching the appropriate public result may help; hiding the product list from crawlers would change the page’s purpose.

  1. Compare durations and response bytes within a meaningful URL group.
  2. Reproduce the slow operation with the developer.
  3. Check caching or application improvements against changed and unchanged responses.
More help and optional notesHigh · SEO lead and developer
PassThe identified serving bottleneck is corrected and changed content is still served correctly.
FailA reproducible serving bottleneck remains or caching returns stale content after changes.

If it fails: Optimize the measured operation and use correct cache validators for appropriate public responses.

Retest: Retest both unchanged and newly edited pages, then compare equivalent request timings.

Not applicable: No relevant serving bottleneck or repeated-transfer issue is found.

Keep in mind: Faster responses do not guarantee increased crawling, indexing or rankings.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Choose controls from the URL’s intended purpose

Avoid using one directive for every duplicate or unwanted URL.

Use this for: URL families with a confirmed duplication, exclusion or unnecessary-crawling problem.

For each unwanted URL family, decide whether the page moved, duplicates a preferred page, must remain available but excluded from search, or should never be crawled. Fix the source of accidental new variants where possible before adding another rule.

Redirects, canonical signals, noindex and robots.txt have different jobs. A crawler must fetch a page to read its canonical or noindex; a crawl block can prevent that. Canonicals do not immediately stop crawling, and blocking a URL is not a reliable removal method for an already indexed page. Test wanted URLs against any proposed pattern.

Example

Hypothetical example: Sort-only duplicates and a public account confirmation need different decisions. The former repeats a category; the latter has a deliberate search exclusion.

  1. Choose the intended use of each suspect URL family.
  2. Select the appropriate specialist repair workflow.
  3. Test representative wanted URLs and unwanted variants before release.
More help and optional notesHigh · SEO lead and developer
PassThe proposed control matches the purpose and preserves crawler access needed to read other signals.
FailThe plan blocks wanted pages or prevents a required canonical/noindex signal from being fetched.

If it fails: Resolve the conflicting decisions and narrow the control to the intended pattern.

Retest: Check affected and exempt URLs after implementation, then observe fresh crawl behavior.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Fix sources of repeated obsolete requests

Reduce avoidable requests caused by current links to old routes.

Use this for: Repeated requests to redirected or removed URLs.

For frequently requested redirects or missing URLs, look for current internal links and sitemap entries that still point there. Follow representative redirects to the final destination. Repair active sources of obsolete URLs rather than treating every historic request as a new failure.

A permanent removal without a replacement can correctly return 404 or 410. A useful redirect can remain in place for old links. Update links to the final intended page and remove unnecessary intermediate hops; do not redirect unrelated missing pages to the homepage.

Example

Hypothetical example: The navigation links to /courses-old/, which redirects twice before reaching /courses/. Update the navigation and simplify the migration chain while preserving useful old redirects.

  1. Find frequently requested old URLs.
  2. Check current inlinks, sitemap entries and any redirect chain.
  3. Repair the source link or excess hop and keep appropriate removal behavior.
More help and optional notesHigh · SEO lead and developer
PassCurrent discovery paths use intended destinations and obsolete responses match their purpose.
FailCurrent templates keep generating broken destinations or unnecessary redirect hops.

If it fails: Update the emitting template or link and repair the specific redirect chain.

Retest: Follow the fixed link and confirm its response; review later logs for the same pattern.

Not applicable: No repeatedly requested obsolete route needs investigation.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Choose one bounded change with a recovery plan

Make the next change testable and reversible.

Use this for: A crawl-budget finding ready for implementation.

Choose a confirmed problem with a manageable URL group, such as one template that emits sort permutations. Agree with the developer what will change, which useful URLs must remain available and how the previous behavior can be restored if those pages break.

Choose a direct outcome such as fewer requests to that redundant pattern or fewer failures on the affected host. Pair it with a protection check for wanted pages. Avoid a universal daily crawl target or a promise that saved requests will move to another group.

Example

Hypothetical example: The first change removes tracking variants from one category template. The review checks the remaining links and category content before expanding the fix.

  1. Pick a confirmed problem and a limited affected group.
  2. Define the expected request change and wanted-page protection checks.
  3. Agree on implementation and recovery before making the change.
More help and optional notesHigh · SEO lead and developer
PassA specific change has a clear expected observation, protected examples and a recovery route.
FailThe plan changes unrelated systems or cannot show whether wanted pages were harmed.

If it fails: Reduce the scope and define a direct check before implementation.

Retest: Run the protected examples immediately after the release and restore the earlier behavior if they fail.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Compare the next relevant crawl cycle with the baseline

Determine whether the intended problem improved without shifting harm to wanted pages.

Use this for: A completed change with an established baseline.

After the change, rerun the same host, bot and URL-group comparison across a period that covers the site’s normal crawl pattern. Keep the release date, outages and publishing volume in view. Check request share, distinct URLs, failures and the freshness of the important cohort.

If unwanted requests fall but important pages are still delayed, investigate their discovery, content and indexing separately. If the logs are too incomplete or little crawling has occurred, call the result inconclusive and continue at a sensible review point. More requests alone is not success.

Example

Hypothetical example: Sort-variant requests fall after a template change, while new products remain discoverable and host errors stay low. That supports the local fix without proving a ranking benefit.

  1. Repeat the same scoped comparison after a relevant crawl cycle.
  2. Check the intended change and the wanted-page safeguards.
  3. Keep, adjust or undo the change based on the observed outcome.
More help and optional notesHigh · SEO analyst
PassThe expected local behavior improved and the reviewed wanted pages remain healthy.
FailThe original problem persists or the change harms wanted-page access or freshness.

If it fails: Adjust or restore the changed rule, then investigate the remaining bottleneck.

Retest: Repeat the same comparison after the correction; keep unobserved outcomes untested.

Not applicable: No crawl-budget change has been implemented yet.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Reset this checklist?

This removes saved statuses, evidence and result dates for the checks on this page. Other checklists keep their records. Export a CSV first if you need a copy.

A useful result, every time

Check a task. Choose a result.

This is a manual SEO workflow. It helps you decide what to investigate and keep a usable record. It does not crawl your website or verify your answers automatically.

  1. Choose your scope. Pick the relevant task group and representative pages you are authorized to inspect.
  2. Run the stated check. Follow the steps, then choose your result. Click the square for Pass, or use the result menu for another status. You do not need to type anything.
  3. Assign the next action. Fix a problem when you can, or ask the right person for help. Optional notes can remind you what to do next. Check the result again after a fix.

What your progress means

Review progress counts passes and failures against applicable checks. Failed work still counts as reviewed. Blocked and untested checks remain unfinished. This is not an SEO score.

Untested
You have not checked this yet.
Pass
You checked it and it works as described.
Fail
You checked it and found something to fix.
Blocked
You need access, information or someone’s help.
Not applicable
This task does not apply to your website.

Export CSV downloads the checks currently shown and any optional notes. Print expands the instructions for that view. Saved results stay in this browser; there is no account sync.

Apply the workflow

What Should You Do After the Crawl-Budget Checks?

Turn the observed bottleneck into a small change with a meaningful comparison and safeguards for wanted pages.

What Does a Useful Crawl-Budget Change Plan Contain?

A useful crawl-budget change plan names the URL pattern, the observed problem and a check that would show improvement. It keeps the original request scope visible so a later result can be compared fairly.

Use this hypothetical format in your own working file. The checklist’s CSV export saves checklist results and optional notes; it does not fetch your logs or generate a crawl-budget report.

Hypothetical crawl-budget findings and monitored changes.
FindingBounded changeCheck after releaseProtect
Sort permutations dominate one category’s verified requests.Stop that template emitting unnecessary sort variants.Compare requests and distinct variants in the same log scope.Useful filtered landing pages and product links.
5xx responses cluster during a daily import.Repair the measured serving bottleneck.Compare failures and response times during the next import.Normal public pages and fresh stock content.
New guide pages lack links from their topic hub.Fix the hub’s publishing rule.Follow its links and observe the next published cohort.Existing guide destinations and sitemap membership.

Which Crawl-Budget Measures Are Worth Comparing?

Compare measures that answer your original question: requests to a suspect pattern, distinct URLs fetched, response failures, serving time and observed delay after important updates. Request count and URL count answer different questions.

Crawl capacity is the host-related limit; crawl demand is the crawler’s interest in fetching the inventory. Neither is a fixed allowance you can read as a daily entitlement. Indexing and rendering remain separate outcomes, so keep them out of a claimed crawl-only success.

Which Checklist Helps With the Finding?

Use the detailed workflow for the problem you found. This keeps crawl-budget checks focused on allocation and capacity while the repair receives the right page-level testing.

A clear process, with honest limits.

Follow the steps, choose the result that matches what you found, and check again after changes. Notes are optional. A completed check does not guarantee indexing, traffic or rankings. How the checklists work.