robots.txt exists: what the check tests and how to fix it

Passes when /robots.txt answers 200 with a text file, not HTML. A missing file does not block crawlers under RFC 9309, but you lose a place to state rules.

Bot Accessibility · Weight 1 in the w2agent Score

How common is it?

86% of 1223 measured sites where this check applied pass.

Latest audit of each site under rubric v3.1. Sites where the check did not apply or could not be measured (for example, no llms.txt for the llms.txt checks) are left out. A partial result does not count as a pass.

What this check tests

The audit requests /robots.txt on your host. It passes when the answer is a 200 whose body is a text file. A 200 is rejected as missing only when its media type is text/html and the body is empty or starts with < (after optional whitespace and a byte-order mark), as a custom 404 page or a single-page app shell served with a 200 usually does. A body that looks like HTML but is served under another media type, such as text/plain, is accepted. Any other status, a timeout or a network error is a fail.

This check only reports whether the file is there. Whether its rules block AI agents is a separate question, answered by AI agent access. That check behaves differently when the file is absent, so a site with no robots.txt can pass that one and fail this one.

Why it matters to agents

RFC 9309 says a crawler that gets a 4xx answer for robots.txt may treat the whole site as open, so a missing file does not lock anyone out. The file is still the standard place to tell automated clients what they may fetch and where your sitemap is. Google's introduction describes it as telling crawlers which URLs they may access, mainly to avoid overloading your site, and notes it is not a mechanism for keeping a page out of search results.

Two cautions from the RFC are worth knowing. The rules are requests that well-behaved crawlers honor, not access control. And a server error (5xx) or unreachable robots.txt means a crawler must assume everything is disallowed, which is worse than having no file.

How to fix it

Serve a UTF-8 plain-text file at the top level of the host, named robots.txt in lowercase, with a 200 status and Content-Type: text/plain:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
  • Create one per host. Rules at example.com/robots.txt do not cover docs.example.com.
  • The Sitemap: line takes a full URL and is how the sitemaps.org protocol suggests announcing a sitemap; see XML sitemap exists.
  • To limit specific AI crawlers, add groups for their user-agent tokens instead of changing *. AI agent access explains which ones the audit looks at, and robots.txt for AI bots gives background.
  • To say what AI systems may do with your content, such as training, without blocking them, add a Content-Signal line. Declared AI usage preferences lists the lines the audit reads.
  • Do not let a catch-all route answer /robots.txt with your app HTML. Put the file in your static folder or add an explicit route.
  • After deploying, request /robots.txt and confirm the status and content type.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →