Technical SEO · Supporting checklist

SEO Log File Analysis Checklist

An SEO log file analysis checklist is a request-analysis workflow for finding which verified crawlers reached your site, what they requested and which responses they received. Check bot identity, CDN or origin coverage, dates and sampling before interpreting the counts. Then turn recurring failures, unnecessary URL patterns and discovery gaps into specific fixes. Raw logs stay in your approved local or internal environment.

Free to useNo account neededSaved in your browser
Two abstract request-record strips from separate edge and origin blocks inspected with a magnifier.
Compare edge and origin request records and verify crawler identity before drawing conclusions.
Your working checklist

Work through the checklist

Click a square to mark a task as Pass. Click again to undo. Use the result menu for a problem, a blocker or a task that does not apply. You do not need to enter notes.

14 checks

Choose Logs That Can Answer the Question

SEO log file analysis examines recorded requests at a known point in your delivery system. Its conclusions depend on which hosts, dates, fields and requests that system actually retains.

Choose a crawl question and an observation window

Start with a question that request records can answer.

Use this for: Any SEO log analysis, including a targeted investigation on a small site.

Choose an issue such as repeated failures on category pages, unexpected parameter crawling or whether new articles were requested after publication. Name the hostname, affected URL group, bot and dates before asking the administrator for an existing access-log export.

Use a window that covers the relevant publishing or incident cycle and check its retention, missing intervals and sampling. A week that includes an outage may answer an incident question but not describe normal behavior. Request logs can show a fetch; they do not by themselves establish indexing or rendered content.

Example

Hypothetical example: A catalog team wants to know whether product pages received requests after Friday’s release. It chooses that cohort and the available following period, rather than analyzing every request ever retained.

  1. Name the question, host, URL group and bot.
  2. Choose dates that cover the relevant event or publishing cycle.
  3. Confirm retention and sampling before treating the result as complete.
More help and optional notesHigh · SEO analyst
PassThe log scope can answer the stated question and its coverage limits are understood.
FailThe report draws conclusions beyond the retained hosts, dates or request sample.

If it fails: Narrow the question or obtain an approved existing view with the missing coverage.

Retest: Recheck coverage whenever the export source, dates or filters change.

Not applicable: No request-level investigation is needed.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Identify CDN, edge and origin coverage

Understand where a request would be recorded before interpreting missing rows.

Use this for: Sites using a CDN, reverse proxy, load balancer or more than one log source.

Ask the administrator where each log comes from. A CDN or edge service may answer from cache or reject a request before it reaches the origin server. Origin-only logs therefore describe requests that reached that layer, not every visit to the public hostname.

Keep layer names with the data. When correlating edge and origin events, use a documented request ID if available; do not simply add their totals. For systems that create worker or proxy subrequests, separate the original client request from those internal events.

Example

Hypothetical example: An edge request receives a cached 200 response and has no matching origin row. That is a normal delivery path, not proof that the bot failed to fetch the page.

  1. Identify the layer and host covered by each file or dataset.
  2. Check how cache hits, denied requests and internal subrequests are represented.
  3. Choose one consistent request population for totals.
More help and optional notesHigh · SEO analyst and site administrator
PassThe report states the observed layer and avoids double-counting correlated requests.
FailOrigin logs are presented as complete public-request coverage or multiple layers are added together.

If it fails: Separate layers and correlate only through known request identifiers or documented behavior.

Retest: Trace a representative request through the corrected view and check the totals.

Not applicable: One verified logging point captures the entire stated scope without a separate delivery layer.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Keep raw logs local and use only needed fields

Prepare useful analysis without exposing visitors or credentials.

Use this for: Any access-log export or analysis summary.

Work in the organization’s approved local or internal analysis environment. Ask for timestamp with timezone, host, method, requested path and query, response status, user agent and the trustworthy original client IP needed for bot verification. Duration, bytes, cache state and request ID help only when their definitions are known.

Keep raw IPs in the restricted verification copy. Remove or redact passwords, access tokens, cookies, authorization headers, personal query values and unnecessary referrers before creating examples or sharing a summary. Use aggregates and sanitized paths in your report. Do not paste raw log rows into this checklist’s optional notes or an unapproved online analyzer.

Example

Hypothetical example: A confirmation request contains an email address in its query string. The analyst keeps the raw file restricted and shares only the route pattern and request count.

  1. Select only fields needed for the chosen question.
  2. Verify bots in the restricted local or internal copy.
  3. Share aggregates or redacted examples after removing sensitive values.
More help and optional notesHigh · SEO analyst and site administrator
PassThe analysis retains necessary fields privately and shared output contains no unnecessary sensitive raw data.
FailRaw logs or secret-bearing URLs are placed in public files, optional notes or unapproved services.

If it fails: Stop sharing the raw export and prepare a minimized, redacted summary in the approved environment.

Retest: Inspect every example row and shared output before it leaves the restricted workspace.

Not applicable: No log data is being accessed or summarized.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Check parsing, timezone and import completeness

Catch data errors before they turn into SEO findings.

Use this for: Logs imported into a local analyzer, spreadsheet or internal query system.

Open a few raw rows in the approved local viewer and match them to the imported columns. Check a path with parameters, a redirect, an error and a normal response. Confirm whether durations are seconds or milliseconds and whether timestamps are request time or export time.

Compare imported counts and earliest/latest timestamps with the export information. Check rejected rows, rotated files and repeated imports. Keep repeated real requests; remove only confirmed duplicate records of the same event. Align timezones before comparing with release or Search Console times.

Example

Hypothetical example: A parser reads a quoted user-agent string as several columns and treats response bytes as the status. Fixing the field mapping removes an apparent error spike.

  1. Match representative raw rows to parsed fields.
  2. Check timestamp timezone, duration units and rejected records.
  3. Resolve missing files or duplicate imports before calculating totals.
More help and optional notesHigh · SEO analyst
PassRepresentative rows parse correctly and the imported period/counts match the known export scope.
FailFields, units, dates or repeated imports distort the report.

If it fails: Correct the parser or field mapping and reimport a clean working copy.

Retest: Repeat the row sample and coverage checks on the corrected import.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Verify and Separate Crawler Requests

A user-agent string is a claim about the requester. Verify the source using the crawler operator’s method, then keep automatic search crawling separate from other bots and testing tools.

Verify claimed Googlebot requests using the source IP

Exclude spoofed identities without confusing unknowns with genuine crawlers.

Use this for: Any report claiming to measure Googlebot activity.

First filter requests whose user agent claims to be Googlebot. Use the trusted original client IP, not a CDN’s forwarding-server IP. Match it against Google’s current published common-crawler IP ranges, or ask the administrator to run a reverse DNS lookup followed by a forward lookup that resolves back to the same IP.

For DNS verification, check the exact permitted hostname ending for the crawler category; a lookalike domain containing the word googlebot is insufficient. Keep verified, spoofed and unresolved requests separate. Missing IPs or failed verification lookups mean unknown, not automatically malicious. This check is analysis, not an instruction to change firewall rules.

Example

Hypothetical example: Two requests use the same Googlebot user agent. Only one source IP passes the operator’s verification method, so only that request enters the verified crawl total.

  1. Filter claimed Googlebot requests and identify the trusted client-IP field.
  2. Use current published IP ranges or reverse-and-forward DNS verification.
  3. Retain separate verified, spoofed and unresolved categories.
More help and optional notesHigh · SEO analyst and site administrator
PassGooglebot totals include verified requests and clearly exclude or label unresolved identities.
FailA user-agent string, reverse lookup alone or a proxy IP is treated as sufficient verification.

If it fails: Correct the IP field and verification method, then recalculate the affected totals.

Retest: Reverify after field-mapping changes and refresh operator ranges for later analyses.

Not applicable: No Googlebot activity is being analyzed.

More detail (optional)

For other bots, use that operator’s own documented verification method. Do not assume Google’s hostnames or IP ranges verify another crawler. Historical IP ownership may need additional investigation when current ranges cannot resolve an older record.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Separate search crawlers, resources and user-triggered tests

Keep each total tied to the crawler behavior it describes.

Use this for: Log sets containing multiple bots, resource types or inspection requests.

After verification, split the relevant automatic search crawler from image crawlers, ads crawlers, user-triggered inspection tools and unrelated bots. Keep document requests separate from JavaScript, CSS and image requests when answering a page-crawl question.

A live inspection can create a test request; it does not show that automatic crawling has resumed. Name the crawler and request type beside every chart or count. A general Google-owned IP by itself does not identify which product made the request.

Example

Hypothetical example: A developer runs live tests after a release. Those requests are separated from the ordinary search crawler before the team compares product-page crawl frequency.

  1. Use the verified identity and user-agent category together.
  2. Separate document and resource request groups.
  3. Exclude manual test traffic from automatic-crawl conclusions.
More help and optional notesHigh · SEO analyst
PassEvery reported total has a consistent bot category and request type.
FailTests or unrelated crawler traffic are counted as ordinary page crawling.

If it fails: Split the population and relabel the report before comparing periods.

Retest: Rerun totals with the same category definitions in each period.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Count requests and distinct URLs by useful page group

Show where requests go without mistaking repeat fetches for wider coverage.

Use this for: A log analysis of crawl distribution.

In your local analyzer or internal query, group verified document requests by meaningful template or path family. For each group, count request rows and distinct exact URLs, then compare those with the intended page inventory. Keep the selected host, dates and bot filter fixed.

If you calculate a share, state the denominator, such as verified Googlebot document requests in this export. Do not divide an HTML-only numerator by a total that includes other bots or assets. A heavily requested page can explain a large request count without increasing the number of pages reached.

Example

Hypothetical example: A sample has 800 category requests but only 20 distinct category URLs. Those are two different observations, not 800 discovered pages.

  1. Group verified document requests by a useful URL family.
  2. Count request events and distinct URLs separately.
  3. Check the denominator before comparing shares.
More help and optional notesHigh · SEO analyst
PassRequest totals, distinct URL counts and any shares describe the same stated population.
FailRepeated requests are counted as distinct pages or incompatible denominators are compared.

If it fails: Recalculate from the correctly filtered event population.

Retest: Check a small group manually against the aggregated result.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Turn Request Patterns Into Specific Fixes

A useful log finding connects repeated requests to a page, template or serving problem you can inspect. Totals help prioritize the next investigation; they do not explain every cause on their own.

Find recurring response failures and their timing

Separate ongoing serving defects from deliberate redirects or removals.

Use this for: Verified requests showing errors, redirects or unusual response changes.

Group the verified requests by status, day and URL family, then open representative affected URLs now. Keep historical and current responses separate: a page returning 200 today may have failed during yesterday’s release.

Prioritize repeated 5xx or 429 responses on wanted pages, unexpected access failures and current links to missing destinations. A 301, 304 or a deliberate 404 is not automatically an error to eliminate. Logs that contain only status codes cannot establish a soft 404; inspect the page content or relevant Search Console finding.

Example

Hypothetical example: A template returns 502 for ten minutes during each catalog refresh, while old deleted products correctly return 404. The recurring template failure needs the first repair.

  1. Group responses by time and URL family.
  2. Inspect representative current pages and their purpose.
  3. Send recurring unintended failures to the responsible developer.
More help and optional notesHigh · SEO analyst and developer
PassUnintended recurring failures are resolved, and deliberate responses are correctly classified.
FailWanted pages continue to fail or the report treats every non-200 response as a defect.

If it fails: Repair the failing route or delivery component and correct current links where needed.

Retest: Check the next relevant request window and the exact affected URLs.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Compare slow responses within the same logging layer

Find slow groups without confusing response timing with browser rendering.

Use this for: Logs with a documented duration field and a serving-time question.

If duration is recorded, read the provider’s field definition and units. Compare equivalent URL templates, statuses and cache states; examine slow examples as well as the typical value. If the field is missing, leave response-time analysis Blocked rather than estimating it from timestamps.

An origin-duration field describes different work from an edge time-to-first-byte field. Neither is a measurement of JavaScript execution or the page’s Core Web Vitals. Give the developer reproducible paths and times to diagnose the relevant serving operation.

Example

Hypothetical example: Origin requests for one search endpoint take much longer on cache misses. Combining them with cached edge hits hides the expensive operation.

  1. Read the duration field’s definition and units.
  2. Compare like-for-like groups and inspect slow examples.
  3. Ask the developer to reproduce the affected serving operation.
More help and optional notesHigh · SEO analyst and developer
PassThe timing comparison is correctly scoped and any confirmed serving problem has been addressed.
FailSlow requests remain unexplained or different timing fields are treated as interchangeable.

If it fails: Separate the timing populations and repair the measured server-side bottleneck.

Retest: Repeat the same group comparison after the change.

Not applicable: Response duration is unrelated to the chosen investigation.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Join requested URLs to an independent page inventory

Find crawled variants and pages missing from normal navigation.

Use this for: An analysis comparing crawl behavior with the intended public inventory.

Compare the requested addresses with a current CMS inventory, sitemap or site crawl. Keep the original host, path, query and encoding in the request data. Create a separate grouping key only when you know which differences are intentionally equivalent; stripping every parameter can hide the issue you need to find.

Review requests that do not match the inventory. They may be old URLs, useful pages absent from the crawl, parameter variants or unwanted routes. A log-only page is not automatically an orphan: confirm the crawler’s scope and check relevant internal links before using that label.

Example

Hypothetical example: Requests for /catalog/?page=2 disappear when an analyst removes every query string. Preserving the original address reveals a real paginated page.

  1. Obtain an independent intended-page inventory.
  2. Match exact requested addresses and retain unmatched groups.
  3. Inspect examples before classifying them as variants, old URLs or missing links.
More help and optional notesHigh · SEO analyst
PassThe join preserves meaningful URL differences and unmatched groups have an investigated explanation.
FailOverbroad normalization or an incomplete crawl creates a misleading coverage conclusion.

If it fails: Restore the original addresses, refine the join and investigate the remaining differences.

Retest: Recheck a representative matched and unmatched URL from each affected family.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Trace repeated requests to unnecessary URL families

Turn a high-volume pattern into a fix at its source.

Use this for: Unexpected or redundant URL families appearing in the logs.

Sort verified request groups by repeated variants or unexpected paths, then open examples. Look for a template emitting session values, endless calendar dates or redundant sort combinations. Compare the content and links to determine whether each family has a legitimate search purpose.

Check where those addresses are still exposed, such as filters, internal links or a sitemap. Recommend repairing the generator or choosing a deliberate consolidation/crawl-control approach. Request frequency alone is not a reason to remove a useful page.

Example

Hypothetical example: A filter widget appends a new tracking token on each selection. The fix belongs in the widget’s URL generation, not in a list of thousands of individual redirects.

  1. Find repeated variant patterns in verified requests.
  2. Inspect the content and the interface that creates them.
  3. Repair the generating rule or send the family to the appropriate specialist workflow.
More help and optional notesHigh · SEO analyst and developer
PassThe targeted fix for confirmed unnecessary patterns is implemented and verified while useful destinations remain available.
FailA current generator continues to expose unwanted variants or useful pages are removed solely for being crawled often.

If it fails: Correct the generating template and review the intended use before applying broad controls.

Retest: Repeat the generating action, then compare later requests to the same pattern.

Not applicable: No unexpected or unnecessary URL family is found.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Investigate important URLs with no matching log row

Keep absence from the dataset separate from proof of no crawl.

Use this for: Wanted pages absent from the analyzed request population.

For an important unmatched URL, recheck the exact host, protocol information, encoded path, query, date range and bot filters. Check retention gaps, sampling and whether cached requests would appear only at the edge. Confirm that the URL existed during the observation window.

Then inspect its crawl/indexing record and discovery links. Report no verified request observed in this dataset unless stronger evidence supports more. A missing row does not prove that the page was never crawled, and a present row does not prove indexing.

Example

Hypothetical example: A product has no origin-log row but appears in the edge export. The original report should describe origin coverage, not claim that Google ignored the product.

  1. Check exact address matching and the page’s publication time.
  2. Check log scope, filters, sampling and delivery layer.
  3. Use page-level inspection and discovery checks for the remaining uncertainty.
More help and optional notesHigh · SEO analyst
PassThe absence is explained or accurately bounded, and any confirmed discovery defect has a next action.
FailNo matching row is presented as proof that no crawl occurred.

If it fails: Correct the scope or wording and investigate the page’s actual discovery path.

Retest: Check the same exact URL in the next suitable window and the appropriate page-level report.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Measure observed fetch delay after meaningful changes

Connect request timing to the publishing event you care about.

Use this for: Questions about discovery or refresh timing after a known content change.

For the selected URL cohort, pair the CMS publication or meaningful update time with the first observed verified document request afterward. Keep repeated requests and statuses available so a failed fetch is not mistaken for successful retrieval of the change.

Use the same timezone and definition across the cohort. If the observation window ends before a request appears, label that page not observed within the window; do not assign an invented delay. Without earlier complete logs, call the event first observed after this change rather than first-ever crawl.

Example

Hypothetical example: A guide is updated Monday at 09:00 and first receives a verified 200 document response Wednesday at 09:00. The observed interval is two days; its later indexing result needs a separate check.

  1. Select changed pages with reliable publication timestamps.
  2. Find the first observed subsequent verified document request and response.
  3. Compare equivalent cohorts while retaining unobserved cases.
More help and optional notesHigh · SEO analyst and content team
PassThe timing result uses verified requests, aligned timestamps and explicit unobserved cases.
FailThe calculation substitutes test requests, unknown dates or invented zeroes for missing observations.

If it fails: Correct the cohort and timestamp join before interpreting the delay.

Retest: Repeat the method for the next comparable publishing cohort.

Not applicable: No URLs in the stated scope.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Produce an actionable finding and rerun the same comparison

Finish with a repair that can be checked against the original question.

Use this for: A confirmed log finding that leads to a website repair.

Summarize each material finding with its scoped request count, sanitized example path, observed behavior and specific next action. Prioritize ongoing failures on wanted pages and broad generating defects over harmless historical requests.

After a repair, use the same bot verification, layer, dates-of-comparable-length and URL grouping. Account for changed publishing volume, outages and sampling. Check both the repaired group and representative wanted pages. If the data remains insufficient, keep the conclusion limited instead of declaring success.

Example

Hypothetical example: A report recommends fixing a category template that emits old paths, then checks whether new requests reach the final category and whether the old-path share declines in comparable data.

  1. Write the finding as an observed pattern plus a concrete repair.
  2. Check the repaired URLs immediately after implementation.
  3. Rerun the same scoped log comparison after relevant requests accumulate.
More help and optional notesHigh · SEO analyst and developer
PassThe reviewed fix behaves correctly and its request pattern improves under a comparable analysis.
FailThe defect persists, wanted pages break or the before/after comparison changes definitions.

If it fails: Correct the repair or comparison and repeat the affected checks.

Retest: Verify the next comparable request window after the correction.

Not applicable: The analysis found no change to implement.

Optional: the page or URL group, what you found and the next action. Keep confidential data out of shared exports.

No result recorded yet.

Changing this result updates your checkmark and progress immediately. Notes are optional.

Reset this checklist?

This removes saved statuses, evidence and result dates for the checks on this page. Other checklists keep their records. Export a CSV first if you need a copy.

A useful result, every time

Check a task. Choose a result.

This is a manual SEO workflow. It helps you decide what to investigate and keep a usable record. It does not crawl your website or verify your answers automatically.

  1. Choose your scope. Pick the relevant task group and representative pages you are authorized to inspect.
  2. Run the stated check. Follow the steps, then choose your result. Click the square for Pass, or use the result menu for another status. You do not need to type anything.
  3. Assign the next action. Fix a problem when you can, or ask the right person for help. Optional notes can remind you what to do next. Check the result again after a fix.

What your progress means

Review progress counts passes and failures against applicable checks. Failed work still counts as reviewed. Blocked and untested checks remain unfinished. This is not an SEO score.

Untested
You have not checked this yet.
Pass
You checked it and it works as described.
Fail
You checked it and found something to fix.
Blocked
You need access, information or someone’s help.
Not applicable
This task does not apply to your website.

Export CSV downloads the checks currently shown and any optional notes. Print expands the instructions for that view. Saved results stay in this browser; there is no account sync.

Apply the workflow

What Should You Keep After the Log File Analysis?

Keep a scoped summary of the useful findings and the repairs they support, with raw data restricted and sensitive examples redacted.

What Should an SEO Log Analysis Report Show?

An SEO log analysis report should show the observed request population, the finding and a repair that can be tested. Keep raw logs in the approved environment; give readers sanitized examples and enough scope to interpret the counts.

The examples below are hypothetical. Use the format in your own file. Export CSV on this page saves checklist results and optional notes, not a log analysis or an uploaded dataset.

Hypothetical log-analysis findings with honest boundaries.
Observed scopeFindingInterpretationNext action
Verified Googlebot document requests at the edge during the chosen week.Repeated 503 responses on /classes/.A recurring public serving failure needs diagnosis.Check the release/incident timeline and repair that route.
Verified origin requests only; CDN cache hits excluded.No row for /guides/soil/.No request observed at this layer.Check edge coverage and URL Inspection before concluding no crawl.
Verified document requests joined to current navigation.Many requests use old category paths still linked by the menu.The current template is exposing obsolete destinations.Update those links and compare later requests.

Which Log Fields Answer Which Question?

Choose log fields from the question you need to answer. Missing fields limit the analysis; they are not a reason to invent values or export unrelated personal data.

Minimum field groups for specific SEO log questions.
QuestionUseful fieldsImportant limit
Is the claimed bot genuine?Trusted original client IP, user agent, verification result.A proxy IP or user agent alone is insufficient.
Which addresses were requested?Host, path, query, method, timestamp and timezone.Preserve meaningful query and encoding differences.
What did that request receive?Client-facing response status and logging-layer label.An origin status may differ from the edge response.
Was serving slow?Documented duration field, unit, cache state and status.Response duration does not measure browser rendering.
Do two rows represent one delivered request?Provider request IDs and layer/subrequest relationship.Equal paths and timestamps alone do not prove duplication.

Which Checklist Should Follow the Log Analysis?

Choose the next checklist from the actual finding. Logs narrow the investigation; each repair still needs the page-level checks that establish its intended behavior.

A clear process, with honest limits.

Follow the steps, choose the result that matches what you found, and check again after changes. Notes are optional. A completed check does not guarantee indexing, traffic or rankings. How the checklists work.