No error status codes: what the check tests and how to fix it

Counts crawled pages that return 4xx, 5xx or no response. Zero errors passes; any error is a partial scored by the share of pages that loaded.

Bot Accessibility · Weight 1 in the w2agent Score

How common is it?

93% of 1223 measured sites where this check applied pass.

Latest audit of each site under rubric v3.1. Sites where the check did not apply or could not be measured (for example, no llms.txt for the llms.txt checks) are left out. A partial result does not count as a pass.

What this check tests

The audit crawls a sample of your pages: the homepage and other pages found through your sitemap and links. Redirects are followed, so the status is the final one. A page counts as an error when that status is 400 or higher, or when there is no status at all (a timeout, a connection failure, or a body that could not be read).

  • Pass: no crawled page was an error. The score is 100.
  • Partial: at least one page was an error. The score is the percentage of crawled pages that were not errors.
  • There is no fail result for this check; the worst outcome is a partial that can be as low as 0.

The details list each failing URL with its status, or "timeout". Only crawled pages count. A broken link the crawler never reached is not seen, and a page that was blocked for the audit's user agent (a 403 from a bot filter) counts as an error even if browsers get a 200.

Why it matters to agents

An agent that follows a link and lands on an error page learns nothing, and a crawler may drop the URL. Google's documentation says it does not use the content of URLs that return 4xx, and stops using previously indexed ones over time. It says 5xx and 429 responses make its crawlers slow down, and that content returned with a 5xx is ignored. A page that returns 200 but shows an error message can be treated as a soft 404.

A few error pages in a large sample are normal, and the percentage reflects that. Errors on pages your own navigation or sitemap points to are the ones worth fixing, because they send every visitor, human or agent, to a dead end.

How to fix it

  1. Read the failing URLs in the audit details. Work out whether each one is a dead link you publish, or a page that fails only for automated requests.
  2. For a moved page, redirect it with a 301 or 308 to its new address and update internal links and the sitemap to the new URL. The same applies to http:// addresses, which should redirect to https://; see HTTPS.
  3. For a page that is gone, return a real 404 or 410. Remove it from your sitemap and links. Do not answer with 200 and a "not found" message.
  4. For 5xx errors, check your server logs around the audit time. Timeouts and intermittent 5xx often come from overloaded origins; see response time.
  5. For a 403 or 429 that only affects bots, review your firewall, rate limiting and bot-protection rules. Edge agent access tests for blocks on AI agent user agents specifically.
  6. Keep only live, canonical URLs in your XML sitemap and llms.txt.

Spot check with curl -s -L -o /dev/null -w '%{http_code}\n' https://example.com/some-page, which prints the final status.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →