Robots.txt Checklist

Use this for: Each new block or existing rule whose purpose is unclear.

For each path you plan to block, decide whether the goal is less crawling, exclusion from search, or private access. Use robots.txt for crawl control. For a public HTML page that must stay out of search, use a crawlable noindex directive. For private material, arrange authentication or access control. Neither robots.txt nor noindex keeps a URL secret.

1. Name the intended outcome for each affected path.
2. Choose crawl rules, a readable noindex, or access control for that outcome.
3. Check that the proposed mechanism can actually achieve the stated goal.

Use this for: Production hosts or protocols that serve content in the review.

List the public main host and any independent shop, blog or asset host. In a private browser window, open /robots.txt at the root of each relevant origin. A file at /blog/robots.txt cannot control a site rooted at /. Check the lowercase filename. If no crawl restrictions are needed, an intentionally absent file is not automatically an SEO defect.

1. List the origins that serve the pages or resources you need to review.
2. Open the exact root /robots.txt URL for each origin.
3. Confirm each active crawl policy is served from the origin it is meant to control.

Use this for: Every robots.txt endpoint in scope.

Open Chrome Developer Tools → Network, reload the robots.txt address, and select that request. Read Headers → Status Code, then Response. A deployed rule file should return 200 with its intended text. Google treats most 4xx responses, including 401, 403 and 404, as no robots restrictions; 429 and server/network failures need investigation. A 404 is acceptable only when absence is intentional, not when you depend on rules.

1. Reload /robots.txt with Network recording.
2. Check the status, final URL and response body.
3. Reload after any fix and confirm the expected public response.

Use this for: Sites that publish a robots.txt file.

Use your host’s robots editor or a plain-text editor. Put a User-agent line before its Allow and Disallow rules, write each directive on a separate line and save as UTF-8. Start paths with /. A # begins a comment. Google ignores unsupported fields such as noindex and crawl-delay; move indexing instructions to HTML or response headers. Keep large generated files within Google’s 500 KiB parsing limit.

1. Open the source that generates the live file.
2. Check encoding, separate lines, supported fields and path syntax.
3. Inspect the published text again to confirm no editor markup was added.

Use this for: Files with explicit crawler groups or more than one group.

Compare the crawler token with every User-agent group. For Google, the most specific matching group applies; repeated equally specific groups combine. The generic * group is not added to a specific Googlebot group. User-agent names are case-insensitive. Test Googlebot and any explicitly named crawler your site depends on; verify another engine’s own behavior before generalizing.

1. Identify the intended crawler token for each test.
2. Collect only the rules in its applicable group or combined equal-specificity groups.
3. Verify the effective rules against one intended allowed and one intended blocked URL.

Use this for: URLs matching both Allow and Disallow rules.

Within the effective group, find every path rule that matches the URL. Google selects the most specific matching path rule; an equally specific Allow/Disallow conflict resolves to Allow. Moving a rule to the bottom does not make it win. Use a narrow Allow exception only when the permitted child path really needs to be fetched.

1. Collect the matching rules for a sample URL.
2. Compare their specificity, including any exact conflict.
3. Verify both the intended exception and a neighboring blocked path.

Use this for: Every path restriction, especially short prefixes.

Copy paths from actual public URLs. Path matching is case-sensitive even though directive names and user-agent tokens are not. A prefix such as /sale also matches /sales-team/. Use /sale/ for that directory, and separately decide how the slashless /sale URL should behave. Check encoded or non-ASCII URLs with a compatible parser instead of guessing from how a browser displays them.

1. Copy the exact paths and capitalization from representative URLs.
2. Add slashless, neighboring-prefix and case variants that actually exist.
3. Confirm matching results agree with the policy for every variant.

Use this for: Rules containing * or $.

For supported Google path patterns, * matches zero or more characters and $ anchors the end of the URL. Test complete path-and-query examples. A pattern ending in .pdf$ does not match .pdf?download=1. Do not block every parameter URL simply because it has a question mark; the parameter may identify a useful page or a necessary resource.

1. Identify each wildcard or end-anchor rule.
2. Add matching, nonmatching and query-string variants to the test cases.
3. Check the expected outcomes with the intended crawler parser.

Use this for: Pages that depend on separate rendering resources.

Open an important page with Chrome Network recording and identify its stylesheet, script and meaningful image URLs. Test those exact URLs against robots.txt on their own hosts. A browser can load a resource that a compliant crawler is told not to fetch. If you block an asset directory, create a justified narrow exception or remove the unnecessary block.

1. Identify the resources needed for one important page template.
2. Check each resource against the correct origin’s effective rules.
3. After a fix, verify that the page and required resources are allowed together.

Use this for: Public URLs intentionally excluded through noindex.

For each public page or PDF that relies on noindex, test whether its effective robots rules allow a fetch. If the resource is blocked, the crawler cannot retrieve the new directive. Remove an accidental crawl block only after the correct noindex is in place. Keep confidential material protected by authentication; do not expose it just to make this test pass.

1. Identify public resources whose search exclusion depends on noindex.
2. Compare them with the applicable robots rules and verify the directive exists.
3. Retest the corrected resource’s crawl access and its noindex output.

Use this for: Sites using robots.txt for sitemap discovery.

Find the sitemap URL in the CMS or deployment output. Add a Sitemap: line with the complete HTTPS address, including the real host and filename. A sitemap index can be listed instead of every child sitemap. The line is independent of crawler groups, so it does not need to be repeated under each agent. Do not guess /sitemap.xml if the actual file has another address.

1. Find the published sitemap or sitemap-index address.
2. Compare it exactly with every Sitemap line.
3. Open each declared absolute address and confirm it points to the intended sitemap.

Use this for: Sitemaps named in the file.

Open each declared sitemap and check its final response in Network. It should serve the expected sitemap content without a login or error. Test the sitemap URL against robots rules, then sample important preferred URLs from it. A useful canonical page listed for discovery should not be accidentally disallowed. Resolve the policy mismatch at the rule or sitemap source.

1. Fetch the declared sitemap or index and inspect its content.
2. Check crawl access to the sitemap and a relevant sample of listed pages.
3. Retest the sitemap and sampled page rules after resolving any mismatch.

Use this for: Every proposed robots.txt change.

Copy the current file into a working text file and keep the original as a rollback copy. Build a small table of crawler, exact URL, expected allow/block and matching reason. Include the homepage, key template URLs, blocked examples, allowed exceptions and required assets. A developer can run the draft through Google’s open-source robots parser; a checker should explicitly support the intended engine’s matching behavior. Keep unrun tests Untested, or use Blocked when the needed tool or access is unavailable.

1. Keep the existing file and identify the proposed changed lines.
2. Write representative expected outcomes and run the draft through a suitable parser.
3. Resolve every unexplained difference before approving the candidate file.

Use this for: An authorized change with passing draft test cases.

Find whether the file is generated by the CMS, hosting configuration or deployment repository. Change that source rather than only a temporary copy. Keep the prior version available. Publish the reviewed change, refresh a stale site/CDN cache if your host requires it, and reload the public /robots.txt response. Never copy a staging-wide Disallow: / into an indexable production site.

1. Confirm the production source and rollback copy.
2. Publish the reviewed file through the normal site release process.
3. Fetch the public file and rerun the same URL test set against its served contents.

Use this for: Sites with an applicable Search Console property.

In Search Console, open the robots.txt report and select the file. Review its last fetch, errors and Versions against your release. After an urgent unblock or fetch repair, use the file’s menu → Request a recrawl if appropriate. Then inspect an affected page with URL Inspection → Test live URL. A current browser response, Google’s cached robots file and the indexed page record can show different moments.

1. Compare the report’s fetched version and time with the public file.
2. Request a robots recrawl only when a critical change warrants it.
3. Check affected page access with a live test and revisit the indexed record after recrawling.