Reachable as Google-Agent: what the check tests and how to fix it

Weight 0.5. Fetches your page once with the Google-Agent string. A 2xx with a non-empty body passes. The only reach check that can run out of time.

Agent Protocols · Weight 0.5 in the w2agent Score

How common is it?

94% of 1208 measured sites where this check applied pass.

Latest audit of each site under rubric v3.1. Sites where the check did not apply or could not be measured (for example, no llms.txt for the llms.txt checks) are left out. A partial result does not count as a pass.

What this check tests

The audit requests the page you submitted once with the Google-Agent string, the third and last of its scored reach requests. Pass and fail follow the rule shared with ChatGPT-User and Perplexity-User: a 2xx status with a non-empty body passes at 100, any other status, an empty body, a timeout or a network error fails, and a missing record or an exhausted time budget is unmeasured and not scored. The ChatGPT-User page has the full table and explains how reach checks differ from AI agent access and edge agent access.

What is specific to this one:

  • The header is Google's desktop agent string: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36. Google also lists a mobile variant, which the audit does not send.
  • The three requests share a time budget, the smaller of three timeouts and 15 seconds, checked before each fetch. With the standard profile's 10-second timeout, two slow responses ahead of it can exhaust the budget, and then this check reports unmeasured rather than failed. Perplexity-User and ChatGPT-User cannot be skipped that way.
  • Google-Agent has no robots.txt token. It is not among the tokens AI agent access tests, so robots.txt has no part in this result at all.

Why it matters to agents

Google documents Google-Agent as used by agents hosted on Google infrastructure to navigate the web and perform actions on a user's request. It sits on Google's list of user-triggered fetchers, and Google says that, because a user requested the fetch, these fetchers generally ignore robots.txt rules. The list publishes the IP ranges Google-Agent uses, in a file called user-triggered-agents.json, and gives reverse DNS masks for verifying fetchers.

Google also says it is experimenting with Web Bot Auth, an IETF draft for signing requests, and that its implementation is experimental: only some Google-Agent requests are signed, and Google recommends continuing to rely on IP addresses, reverse DNS and user-agent strings. That is why this check does not look for signatures.

A pass here means your server does not reject the Google-Agent string from the audit's address. Because the real agent comes from Google's ranges, a firewall that verifies those ranges can treat the real agent differently. Nothing fetched for this page says what a Google-Agent visit does for your visibility in Search, and Google's own guidance on AI features says no extra technical requirements apply for appearing in AI Overviews or AI Mode. This check is about task agents reaching a page, not about ranking.

How to fix it

  1. Read the message. A status such as 403 or 429 means a rule is reacting to the string. A timeout points to slow responses.
  2. Check your CDN or WAF for bot rules that cover "AI agents" or all unrecognised automation. Many block by user-agent substring.
  3. If you want to allow Google-Agent, use Google's published ranges or reverse DNS together with the user-agent, which Google says to keep relying on, or a product that supports Web Bot Auth verification. Some services provide a built-in way to allow Google-Agent; use your provider's documentation for the exact steps.
  4. If the result is unmeasured, the earlier fetches in the queue were slow. Improve response time and re-run.
  5. If you intend to block it, the failing result is the intended outcome.

If you operate a bot yourself and want others to verify it, agent operator signing covers publishing a key directory. Background: AI crawler blocking mistakes.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →