Edge treatment of AI agent user agents: what the check tests and how to fix it

Tracked, not scored. Compares the homepage for the audit's own crawler with agent user-agent strings. A challenge, 403, 429 or other non-2xx for an agent fails.

Bot Accessibility · Tracked, not scored

How common is it?

93% of 1135 measured sites do not block agent user-agents at the edge (a pass; this check does not change the score).

Latest audit of each site under rubric v3.1. Sites where the check did not apply or could not be measured (for example, no llms.txt for the llms.txt checks) are left out. A partial result does not count as a pass.

What this check tests

This is a tracked check: it does not change your score. AI agent access reads what your robots.txt says. This check measures what your server or CDN actually does.

The audit fetches your homepage with its own declared crawler (the control), then fetches it again under user-agent strings of documented user-initiated agents: ChatGPT-User, Perplexity-User and Google-Agent first, followed by other agent tokens whose vendors publish a user-agent string. Up to two more key pages are fetched under the first three. Claude-User has no published user-agent string in the audit's list, so it is never probed and is listed as not probed.

Only a page where the control got a clean 2xx response counts as a comparison. Then, for each agent:

OutcomeResult
An anti-bot challenge, 403 or 429 for an agent user agentFail (blocked at the edge)
Any other non-2xx answer, except 402Fail (rejected)
402 Payment RequiredPass, noted as informational: a declared paywall is not a block
Every probed agent answered 2xx, and at least one page was answered by all agents probed on itPass (consistent)
No clean control, no agent response, or some probed agent never answered 2xx on a comparable pageUnmeasured (shown as an error, not a fail)

A block counts even if another agent timed out. A pass needs completeness, so "consistent" is not awarded on partial evidence.

Why it matters to agents

A robots.txt that welcomes an agent does nothing if the CDN's bot protection answers that agent with a challenge page it cannot solve or a 403. From the agent's side the page is simply unavailable, and users of the assistant see "could not read this site".

Read a pass with care. OpenAI and Google publish the IP ranges their agents use, and bot protection that verifies those can treat a request that only copies the user-agent string differently from the real agent. So a pass here shows your edge does not reject these user-agent strings from the audit's address. It does not prove the real agent is let through from the vendor's address. A fail is a stronger signal: your edge reacts to the user-agent string itself.

How to fix it

  1. Read the failing line in the details. It names the agent, the page and the status, and says if a challenge vendor was recognised.
  2. Open your CDN or WAF bot-management settings and look for rules or managed categories covering AI crawlers, "AI bots" or all automated traffic. Check security events for the audit's requests.
  3. Decide per agent. If you want user-initiated agents to read your pages, allow them (preferably by the vendor's published IP ranges plus user-agent, so the rule is not trivially spoofed) and leave your block for training crawlers in place if you want that.
  4. Check rate limiting. A 429 for an agent usually means a per-IP or per-user-agent limit set too low.
  5. Re-run the audit. If the result is unmeasured, the usual reason is that the control request itself was challenged or not 2xx; fix the status codes for the audit crawler first.

More on the topic: AI crawler blocking mistakes and robots.txt for AI bots. Agent operator signing covers the key directory a bot operator publishes for Web Bot Auth request signing. The scored reach checks for ChatGPT-User, Perplexity-User and Google-Agent make the same kind of request once per agent, but without a control and without challenge detection.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →