What this check tests
The audit reads your /robots.txt and, for each AI-related token in its list, asks whether that token may fetch the homepage path /. Tokens are grouped by what the vendor says they are for:
- User-initiated agents, which fetch a page because a person asked an assistant about it: ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, MistralAI-User and Meta-ExternalFetcher.
- AI search fetchers, which build search indexes: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot, Amzn-SearchBot, MistralAI-Index and Meta-WebIndexer.
- Training crawlers such as GPTBot, ClaudeBot and Google-Extended, shown in the details but not scored.
| Result | When |
|---|---|
| Fail (0) | Any user-initiated token is blocked from / |
| Partial (50) | No user-initiated token is blocked, but a search token is |
| Pass (100) | Neither group is blocked, or there is no robots.txt (a missing file allows everything) |
| Fail (0) | robots.txt answered with a 5xx status; RFC 9309 says a crawler must then assume complete disallow |
| Unmeasured (error) | The robots.txt fetch failed with a network error or timeout, or kept returning 429 |
Google-Agent has no robots.txt token, so it is not scored. Only the path / is tested; a rule that blocks a section of the site is not seen.
Why it matters to agents
The vendors describe these tokens differently. OpenAI says GPTBot and OAI-SearchBot are separate robots.txt tags, so you can allow search while opting out of training, and that ChatGPT-User visits come from user actions where robots.txt rules may not apply. Anthropic lists ClaudeBot (training), Claude-User and Claude-SearchBot as separate bots, describes what disabling each one does, and says disabling Claude-User can reduce your visibility for user-directed searches. Perplexity says Perplexity-User generally ignores robots.txt because a person asked for the fetch. Google says user-triggered fetchers generally ignore robots.txt.
So this check scores declared intent: a rule that tells user-initiated agents to stay out signals a policy that some vendors honor and others do not. Blocking training crawlers is a legitimate choice and does not lower this score. Blocking everything with a blanket rule, or a security plugin or CDN default that does so, is the usual cause of a fail. RFC 9309 itself notes robots.txt rules are not access authorization.
How to fix it
- Find the rule that applies. Check for
User-agent: *withDisallow: /, and for groups that name the tokens above. The audit result lists which tokens it found blocked. - Remove the block, or add a more specific group that allows the agents you want. A crawler uses the group matching its token, case-insensitively, so a named group overrides
*for that crawler:
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
The second group opts out of model training for those two crawlers and has no effect on this check.
- If you use a CDN or WAF, make sure it does not block the same agents at the network layer. Robots.txt allows do not help if the edge rejects the request; edge agent access looks for that.
- Re-run the audit, or fetch
/robots.txtyourself to confirm the final file.
Background: robots.txt for AI bots and AI crawler blocking mistakes. Related: robots.txt exists and declared AI usage preferences, which covers stating a training preference without blocking access. If you operate a crawler or agent yourself, agent operator signing covers publishing keys so other sites can verify your requests. Whether the server answers the user-initiated agents is a separate question, covered by the reach checks for ChatGPT-User, Perplexity-User and Google-Agent.