What this check tests
The audit requests /robots.txt on your host. It passes when the answer is a 200 whose body is a text file. A 200 is rejected as missing only when its media type is text/html and the body is empty or starts with < (after optional whitespace and a byte-order mark), as a custom 404 page or a single-page app shell served with a 200 usually does. A body that looks like HTML but is served under another media type, such as text/plain, is accepted. Any other status, a timeout or a network error is a fail.
This check only reports whether the file is there. Whether its rules block AI agents is a separate question, answered by AI agent access. That check behaves differently when the file is absent, so a site with no robots.txt can pass that one and fail this one.
Why it matters to agents
RFC 9309 says a crawler that gets a 4xx answer for robots.txt may treat the whole site as open, so a missing file does not lock anyone out. The file is still the standard place to tell automated clients what they may fetch and where your sitemap is. Google's introduction describes it as telling crawlers which URLs they may access, mainly to avoid overloading your site, and notes it is not a mechanism for keeping a page out of search results.
Two cautions from the RFC are worth knowing. The rules are requests that well-behaved crawlers honor, not access control. And a server error (5xx) or unreachable robots.txt means a crawler must assume everything is disallowed, which is worse than having no file.
How to fix it
Serve a UTF-8 plain-text file at the top level of the host, named robots.txt in lowercase, with a 200 status and Content-Type: text/plain:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
- Create one per host. Rules at
example.com/robots.txtdo not coverdocs.example.com. - The
Sitemap:line takes a full URL and is how the sitemaps.org protocol suggests announcing a sitemap; see XML sitemap exists. - To limit specific AI crawlers, add groups for their user-agent tokens instead of changing
*. AI agent access explains which ones the audit looks at, and robots.txt for AI bots gives background. - To say what AI systems may do with your content, such as training, without blocking them, add a
Content-Signalline. Declared AI usage preferences lists the lines the audit reads. - Do not let a catch-all route answer
/robots.txtwith your app HTML. Put the file in your static folder or add an explicit route. - After deploying, request
/robots.txtand confirm the status and content type.