Meta Robots Checklist

Use this for: HTML pages and non-HTML files in the review.

Choose representative URLs for public pages, intentional search exclusions and downloadable files. For each, decide whether it should be eligible for search and whether previews need limits. Add the responsible CMS/template or server owner to your existing tracker if useful. Keep private material behind access control; a noindex setting only concerns search results.

1. Select representative URL types and one real URL for each.
2. Choose the expected index and preview behavior.
3. Confirm the intended behavior with the person responsible for that content.

Use this for: HTML pages in scope.

Open Chrome Developer Tools → Network, reload the page and select the main document. In Response, search for name="robots", name="googlebot" and name="googlebot-news", allowing for different quote styles and capitalization. Read every matching content value. In your CMS or template, keep authored meta tags in the head. An absent robots tag is not automatically an error; default permission does not guarantee indexing.

1. Reload the page with Network recording and open its document Response.
2. Find all general and relevant crawler-specific robots meta tags.
3. Compare their combined meaning with the expected behavior.

Use this for: All sampled URLs, including PDFs.

In the Network panel, select the actual page or file request and open Headers → Response Headers. Read every X-Robots-Tag value, including crawler-specific ones. Follow the redirect to the final content response when present. Header rules may come from the application, web server or CDN. A meta tag cannot override an applicable noindex response header.

1. Open the final content request in Network.
2. Read all X-Robots-Tag response values and their crawler scope.
3. Compare the header and HTML restrictions with the intended result.

Use this for: Pages whose head or content is altered by JavaScript.

Compare the Network document response with the current head in DevTools → Elements. With Search Console access, use URL Inspection → Test live URL and inspect the tested page’s HTML. If the initial response contains noindex, do not rely on JavaScript to remove it: Google may skip rendering that page. Fix the initial response or CMS setting instead.

1. Inspect the initial document robots tags.
2. Compare them with the rendered head and available live-test HTML.
3. After any correction, retest both initial and rendered output.

Use this for: HTML pages whose indexing preference needs correction.

In WordPress with Yoast SEO, open the specific page, then Yoast SEO → Advanced. Use the setting that asks whether search engines may show the content in results: No requests noindex; Yes removes that page-level exclusion, subject to other rules. Save or update the page. For a code-managed page, set or remove <meta name="robots" content="noindex"> in the head template for that exact route. Check the public output after either method.

1. Open the specific page’s SEO setting or its head template.
2. Apply the intended page-level choice and save the scoped change.
3. Reload the public page and verify the HTML and response headers together.

Use this for: Pages with multiple tags, headers or crawler-specific directives.

Read general robots directives together with any applicable crawler-specific ones. Google combines negative rules and uses the more restrictive instruction when they conflict. Thus index cannot cancel noindex, and max-snippet:160 cannot cancel nosnippet. Find the plugin, template or response rule that adds the unintended restriction and change that source. Removing one copy is not enough if another remains.

1. List all relevant meta and header values for the same response.
2. Work out the combined restrictions for the intended crawler.
3. Remove only unintended rules, then inspect all directive locations again.

Use this for: Public pages or files intentionally using noindex.

Test the exact URL against the relevant robots.txt rules and confirm its public content response can be fetched. If robots.txt blocks it, Google cannot read its HTML or X-Robots-Tag instruction. For a public exclusion, retain noindex and correct the accidental crawl block. If the material is private, use authentication or remove public access rather than opening it for crawlers.

1. Confirm the intended noindex is in the served page or file response.
2. Check the URL’s crawl access and current response.
3. Retest both access and the directive after resolving an accidental block.

Use this for: PDFs or other non-HTML resources that should remain public but out of search.

Ask the developer to add X-Robots-Tag: noindex to the actual file response through the application, server or CDN configuration. Scope the rule to the intended file or directory before considering a sitewide extension rule. A noindex tag on the HTML download page does not automatically apply to the linked PDF. Keep the file crawlable so the header can be read.

1. Identify the exact file URL and the system serving its response.
2. Apply a narrowly scoped header rule in the normal configuration.
3. Fetch the file and an unaffected neighboring file to verify the intended header scope.

Use this for: Pages with a robots or X-Robots-Tag nofollow directive.

Check whether the intention really applies to every link on the page. If a normal public guide has an accidental page-level nofollow, remove it from the responsible CMS setting, template or header. If the intention concerns one paid or user-submitted link, handle that individual link instead. Do not treat nofollow as a privacy control or a guaranteed barrier to crawling a linked URL.

1. Identify where page-level nofollow is generated.
2. Decide whether all links genuinely need that instruction.
3. After correcting an accidental rule, inspect both the directive and ordinary navigation links.

Use this for: Paid placements, user-generated links or links you do not want to endorse.

In the editor’s link settings or anchor HTML, use rel="sponsored" for paid placements and rel="ugc" for user-generated links when appropriate. Use nofollow when those more specific relationships do not apply and you do not want association with that destination. Keep ordinary useful links free of unnecessary qualifiers. These attributes do not prevent the destination being found through other links or sitemaps.

1. Identify the relationship of each link needing qualification.
2. Apply the relevant rel value to that anchor in the editor or HTML.
3. Inspect the published anchor and confirm unrelated links keep their intended behavior.

Use this for: Pages that need explicit preview limits or have unexplained restrictions.

Inspect the current robots values before adding more. For Google, max-snippet sets a character limit; 0 acts like nosnippet and -1 removes that numeric limit. max-image-preview accepts none, standard or large. max-video-preview uses seconds; 0 allows at most a static image, while -1 sets no limit. A limit is permission, not a promise of a preview. Separate structured-data or licensing permissions may affect permitted uses. Google’s noarchive and nositelinkssearchbox rules are now ignored; do not rely on them for current Google behavior.

1. Identify any required preview restriction and its intended engine.
2. Set supported values in the responsible meta tag or header; remove accidental restrictive duplicates.
3. Inspect the served values and confirm their combined meaning matches the chosen policy.

Use this for: HTML pages where a particular text passage should not appear in Google snippets.

Wrap the intended text in a valid span, div or section with the data-nosnippet attribute. Keep the rest of the page available if a whole-page nosnippet is not wanted. The attribute is boolean: data-nosnippet="false" still applies, so remove the attribute to turn it off. Avoid adding or removing it from an existing node after rendering begins; include it when the element is created.

1. Identify the exact text that needs snippet exclusion.
2. Add data-nosnippet to a supported HTML wrapper with valid closing tags.
3. Inspect initial and rendered markup to confirm the exclusion covers only the intended passage.

Use this for: Launches, migrations or shared-template/configuration changes.

On a production site intended for search, check any environment-wide noindex setting alongside page settings and headers. In WordPress, review Settings → Reading → Discourage search engines from indexing this site. Clear an unintended production-wide exclusion, while retaining authentication or the chosen protections on staging. Do not remove every noindex indiscriminately; intentional utility-page exclusions still belong.

1. Compare production environment settings with the intended launch policy.
2. Correct any inherited global exclusion at its source.
3. Inspect a public page from each affected template and an intentionally excluded control URL.

Use this for: Any page whose directive was added, removed or narrowed.

Reload the exact public URL and inspect its final status, response headers, initial HTML and relevant rendered HTML again. In Search Console, compare Test live URL with the indexed result and last crawl date. A live test can show the corrected directive before the indexed record changes. Keep actual indexing or removal pending until fresh search-engine evidence supports it; do not keep changing a correct tag merely because processing is incomplete.

1. Repeat the same response and directive checks used before the edit.
2. Run the available live inspection and compare it with the indexed record.
3. After a fresh crawl, revisit the record and confirm the intended indexing state or investigate the remaining reason.