Response time: what the check tests and how to fix it

Averages the audit's download time per crawled page. Under 1.5 s passes, 1.5 to 3 s is partial, 3 s or more fails. It is full download time, not first byte.

Bot Accessibility · Weight 1 in the w2agent Score

How common is it?

80% of 1223 measured sites where this check applied pass.

Latest audit of each site under rubric v3.1. Sites where the check did not apply or could not be measured (for example, no llms.txt for the llms.txt checks) are left out. A partial result does not count as a pass.

What this check tests

For every page the audit crawled, it records the time from sending the request to having read the HTML body, redirects included. That time also includes any retries and waits: a timeout is retried up to three more times, and a 429 or 503 is retried with a wait of 0.5 s, 1 s and 2 s. On a 429 only, the wait is the Retry-After value instead, when that value is greater than 0 and at most 30 seconds. A 429 or 503 that the crawler recognises as a bot challenge is not retried. Pages whose recorded time is 0 are left out. The check averages the rest and grades the average:

AverageResultScore
under 500 msPass100
500 to under 1500 msPass70
1500 to under 3000 msPartial35
3000 ms or moreFail0

If no page has a timing, the check is skipped. The result lists each page with its time.

Read the number with these limits in mind. It is one recorded time per crawled page (which can include retries, as above), taken from the audit's servers, not from your users' locations. It covers the whole HTML download, so a large page or a slow connection to the audit servers raises it even when your server's first byte is quick. A single slow page can pull an average of a few pages over a line.

Why it matters to agents

An agent fetching your page usually has a time limit, and a slow response may be abandoned. This audit's crawl gives each request attempt 10 seconds to start answering (the timer stops once the response headers arrive, so reading the body is not covered by it), so a server that does not answer within that time is retried and then counted as unreachable.

For search crawlers, Google's crawl-budget documentation says its crawl capacity limit rises when a site responds consistently with stable response times, and falls when the site slows down or returns server errors or rate-limiting signals. This page has not verified the same behavior for other agents' crawlers, so treat it as one documented example of slow servers being crawled less.

How to fix it

  1. Find out where the time goes. Run curl -s -o /dev/null -w 'ttfb %{time_starttransfer}s total %{time_total}s\n' https://example.com/ a few times. A high first-byte time points at server work; a gap between first byte and total points at page size or bandwidth.
  2. Cache what you can. Static or rarely changing pages should be served from a CDN or full-page cache with sensible Cache-Control headers. The MDN caching guide explains the header options.
  3. Cut server work on the slowest pages: database queries, uncached API calls, and plugins that run on every request.
  4. Reduce HTML weight: avoid inlining large JSON blobs or base64 images in the document, and enable compression (gzip or Brotli).
  5. Avoid redirect chains for the URLs you list in your sitemap and links. Each hop adds a round trip to the measured time.

Re-run the audit after a change; the next result replaces this one. Related checks: no error status codes and XML sitemap exists.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →