AI agent access in robots.txt: what the check tests and how to fix it

Fails if robots.txt blocks the homepage for a user-initiated AI agent such as ChatGPT-User; partial if an AI search bot is blocked. Training bots are unscored.

Bot Accessibility · Weight 2 in the w2agent Score

How common is it?

97% of 1219 measured sites where this check applied pass.

Latest audit of each site under rubric v3.1. Sites where the check did not apply or could not be measured (for example, no llms.txt for the llms.txt checks) are left out. A partial result does not count as a pass.

What this check tests

The audit reads your /robots.txt and, for each AI-related token in its list, asks whether that token may fetch the homepage path /. Tokens are grouped by what the vendor says they are for:

  • User-initiated agents, which fetch a page because a person asked an assistant about it: ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, MistralAI-User and Meta-ExternalFetcher.
  • AI search fetchers, which build search indexes: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot, Amzn-SearchBot, MistralAI-Index and Meta-WebIndexer.
  • Training crawlers such as GPTBot, ClaudeBot and Google-Extended, shown in the details but not scored.
ResultWhen
Fail (0)Any user-initiated token is blocked from /
Partial (50)No user-initiated token is blocked, but a search token is
Pass (100)Neither group is blocked, or there is no robots.txt (a missing file allows everything)
Fail (0)robots.txt answered with a 5xx status; RFC 9309 says a crawler must then assume complete disallow
Unmeasured (error)The robots.txt fetch failed with a network error or timeout, or kept returning 429

Google-Agent has no robots.txt token, so it is not scored. Only the path / is tested; a rule that blocks a section of the site is not seen.

Why it matters to agents

The vendors describe these tokens differently. OpenAI says GPTBot and OAI-SearchBot are separate robots.txt tags, so you can allow search while opting out of training, and that ChatGPT-User visits come from user actions where robots.txt rules may not apply. Anthropic lists ClaudeBot (training), Claude-User and Claude-SearchBot as separate bots, describes what disabling each one does, and says disabling Claude-User can reduce your visibility for user-directed searches. Perplexity says Perplexity-User generally ignores robots.txt because a person asked for the fetch. Google says user-triggered fetchers generally ignore robots.txt.

So this check scores declared intent: a rule that tells user-initiated agents to stay out signals a policy that some vendors honor and others do not. Blocking training crawlers is a legitimate choice and does not lower this score. Blocking everything with a blanket rule, or a security plugin or CDN default that does so, is the usual cause of a fail. RFC 9309 itself notes robots.txt rules are not access authorization.

How to fix it

  1. Find the rule that applies. Check for User-agent: * with Disallow: /, and for groups that name the tokens above. The audit result lists which tokens it found blocked.
  2. Remove the block, or add a more specific group that allows the agents you want. A crawler uses the group matching its token, case-insensitively, so a named group overrides * for that crawler:
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

The second group opts out of model training for those two crawlers and has no effect on this check.

  1. If you use a CDN or WAF, make sure it does not block the same agents at the network layer. Robots.txt allows do not help if the edge rejects the request; edge agent access looks for that.
  2. Re-run the audit, or fetch /robots.txt yourself to confirm the final file.

Background: robots.txt for AI bots and AI crawler blocking mistakes. Related: robots.txt exists and declared AI usage preferences, which covers stating a training preference without blocking access. If you operate a crawler or agent yourself, agent operator signing covers publishing keys so other sites can verify your requests. Whether the server answers the user-initiated agents is a separate question, covered by the reach checks for ChatGPT-User, Perplexity-User and Google-Agent.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →