What this check tests
The audit picks up to two pages: your homepage, and one docs page it crawled (a docs. subdomain, or a path starting with /docs or /documentation). For each page it makes two requests:
- The same URL with the header
Accept: text/markdown. One redirect is followed. - A markdown URL: the page path with
.mdadded (/index.mdfor the homepage), query string removed.
A request counts as a hit only if all of these hold:
- the final status is 2xx;
- the
Content-Typeistext/markdownortext/x-markdown(text/plaindoes not count); - the body is not empty and does not start like an HTML document;
- the body is not a copy of the page the crawler already fetched.
For .md hits there is one more test. The audit requests a random, nonexistent .md path in the same directory. If that also returns markdown, the site answers every .md URL and the hit is discarded as a catch-all.
| Result | When |
|---|---|
| Pass (100) | A .md URL hit, or an Accept: text/markdown hit whose response (or its redirect) has Vary: Accept |
| Partial (50) | An Accept: text/markdown hit but no Vary: Accept |
| Fail (0) | No hit |
The Accept: text/markdown and .md requests share a 10 second budget. The extra request that checks whether a .md hit is only a catch-all echo has its own 5 second limit.
Why it matters to agents
Markdown is a smaller, cleaner input than a full HTML page. The llms.txt proposal recommends offering a markdown version of pages at the same URL with .md added. Serving markdown for a request that asks for it by media type is ordinary HTTP content negotiation: the client lists acceptable types in Accept and the server picks a representation (text/markdown is the registered type, RFC 7763).
Vary: Accept matters because the same URL now has two representations. MDN explains that Vary tells caches which request headers affected the response so they key their stored copies on them. Without it, a CDN can cache the markdown and hand it to a browser, or the reverse. That is why a hit without Vary is only a partial.
Which agents ask for markdown today varies by tool. Support is not universal.
How to fix it
Choose one or both.
A .md URL. Generate or render a markdown twin for each page and serve it at the page path plus .md, with Content-Type: text/markdown; charset=utf-8. This works on static hosts and CDNs, with no header logic. Return a real 404 for .md paths that do not exist, so the catch-all test passes.
Header-based negotiation. Respond to Accept: text/markdown on the normal URL:
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Send Vary: Accept on the HTML response as well as the markdown one, so caches treat both correctly.
Do not label an HTML page as markdown, and do not return the same body for every request. Link the markdown from the page with a markdown alternate link and list the pages in llms.txt.