Markdown content negotiation: what the check tests and how to fix it

Passes when the homepage or a docs page returns real markdown at a .md URL, or for Accept: text/markdown with Vary: Accept. Without Vary it is partial.

Bot Accessibility · Weight 0.5 in the w2agent Score

How common is it?

21% of 1223 measured sites where this check applied pass.

Latest audit of each site under rubric v3.1. Sites where the check did not apply or could not be measured (for example, no llms.txt for the llms.txt checks) are left out. A partial result does not count as a pass.

What this check tests

The audit picks up to two pages: your homepage, and one docs page it crawled (a docs. subdomain, or a path starting with /docs or /documentation). For each page it makes two requests:

  1. The same URL with the header Accept: text/markdown. One redirect is followed.
  2. A markdown URL: the page path with .md added (/index.md for the homepage), query string removed.

A request counts as a hit only if all of these hold:

  • the final status is 2xx;
  • the Content-Type is text/markdown or text/x-markdown (text/plain does not count);
  • the body is not empty and does not start like an HTML document;
  • the body is not a copy of the page the crawler already fetched.

For .md hits there is one more test. The audit requests a random, nonexistent .md path in the same directory. If that also returns markdown, the site answers every .md URL and the hit is discarded as a catch-all.

ResultWhen
Pass (100)A .md URL hit, or an Accept: text/markdown hit whose response (or its redirect) has Vary: Accept
Partial (50)An Accept: text/markdown hit but no Vary: Accept
Fail (0)No hit

The Accept: text/markdown and .md requests share a 10 second budget. The extra request that checks whether a .md hit is only a catch-all echo has its own 5 second limit.

Why it matters to agents

Markdown is a smaller, cleaner input than a full HTML page. The llms.txt proposal recommends offering a markdown version of pages at the same URL with .md added. Serving markdown for a request that asks for it by media type is ordinary HTTP content negotiation: the client lists acceptable types in Accept and the server picks a representation (text/markdown is the registered type, RFC 7763).

Vary: Accept matters because the same URL now has two representations. MDN explains that Vary tells caches which request headers affected the response so they key their stored copies on them. Without it, a CDN can cache the markdown and hand it to a browser, or the reverse. That is why a hit without Vary is only a partial.

Which agents ask for markdown today varies by tool. Support is not universal.

How to fix it

Choose one or both.

A .md URL. Generate or render a markdown twin for each page and serve it at the page path plus .md, with Content-Type: text/markdown; charset=utf-8. This works on static hosts and CDNs, with no header logic. Return a real 404 for .md paths that do not exist, so the catch-all test passes.

Header-based negotiation. Respond to Accept: text/markdown on the normal URL:

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

Send Vary: Accept on the HTML response as well as the markdown one, so caches treat both correctly.

Do not label an HTML page as markdown, and do not return the same body for every request. Link the markdown from the page with a markdown alternate link and list the pages in llms.txt.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →