What this check tests
This is a tracked check: it does not change your score. AI agent access reads what your robots.txt says. This check measures what your server or CDN actually does.
The audit fetches your homepage with its own declared crawler (the control), then fetches it again under user-agent strings of documented user-initiated agents: ChatGPT-User, Perplexity-User and Google-Agent first, followed by other agent tokens whose vendors publish a user-agent string. Up to two more key pages are fetched under the first three. Claude-User has no published user-agent string in the audit's list, so it is never probed and is listed as not probed.
Only a page where the control got a clean 2xx response counts as a comparison. Then, for each agent:
| Outcome | Result |
|---|---|
| An anti-bot challenge, 403 or 429 for an agent user agent | Fail (blocked at the edge) |
| Any other non-2xx answer, except 402 | Fail (rejected) |
| 402 Payment Required | Pass, noted as informational: a declared paywall is not a block |
| Every probed agent answered 2xx, and at least one page was answered by all agents probed on it | Pass (consistent) |
| No clean control, no agent response, or some probed agent never answered 2xx on a comparable page | Unmeasured (shown as an error, not a fail) |
A block counts even if another agent timed out. A pass needs completeness, so "consistent" is not awarded on partial evidence.
Why it matters to agents
A robots.txt that welcomes an agent does nothing if the CDN's bot protection answers that agent with a challenge page it cannot solve or a 403. From the agent's side the page is simply unavailable, and users of the assistant see "could not read this site".
Read a pass with care. OpenAI and Google publish the IP ranges their agents use, and bot protection that verifies those can treat a request that only copies the user-agent string differently from the real agent. So a pass here shows your edge does not reject these user-agent strings from the audit's address. It does not prove the real agent is let through from the vendor's address. A fail is a stronger signal: your edge reacts to the user-agent string itself.
How to fix it
- Read the failing line in the details. It names the agent, the page and the status, and says if a challenge vendor was recognised.
- Open your CDN or WAF bot-management settings and look for rules or managed categories covering AI crawlers, "AI bots" or all automated traffic. Check security events for the audit's requests.
- Decide per agent. If you want user-initiated agents to read your pages, allow them (preferably by the vendor's published IP ranges plus user-agent, so the rule is not trivially spoofed) and leave your block for training crawlers in place if you want that.
- Check rate limiting. A 429 for an agent usually means a per-IP or per-user-agent limit set too low.
- Re-run the audit. If the result is unmeasured, the usual reason is that the control request itself was challenged or not 2xx; fix the status codes for the audit crawler first.
More on the topic: AI crawler blocking mistakes and robots.txt for AI bots. Agent operator signing covers the key directory a bot operator publishes for Web Bot Auth request signing. The scored reach checks for ChatGPT-User, Perplexity-User and Google-Agent make the same kind of request once per agent, but without a control and without challenge detection.