llms.txt links resolve: what the check tests and how to fix it

Fetches up to the first 10 URLs in your llms.txt and counts those answering 2xx or 3xx. All must resolve to pass; a fetch the crawler fails counts as broken.

Agent Discovery · Weight 0.5 in the w2agent Score

How common is it?

65% of 868 measured sites where this check applied pass.

Latest audit of each site under rubric v3.1. Sites where the check did not apply or could not be measured (for example, no llms.txt for the llms.txt checks) are left out. A partial result does not count as a pass.

What this check tests

If no llms.txt was found, the check is skipped. Otherwise it pulls every http:// or https:// URL out of the file text with a pattern, keeps the first 10, and requests each one with a 5 second timeout, following redirects. A URL is "resolved" when the final answer is a 2xx or 3xx status.

  • The score is the share of checked URLs that resolved. All resolved is a pass, some is a partial, none is a fail.
  • A file with no URLs at all is a fail.
  • A timeout, a network error, or a 4xx or 5xx answer counts as broken. That includes a link your server blocks for the audit crawler (a 403 from a bot filter, for example), even if browsers load it fine.
  • The pattern stops at whitespace and at ), >, ], , and : after the scheme, so a URL containing a comma or a port number can be cut short. Any URL in the file counts, not only the ones in list items.
  • Only the first 10 matches are checked. A broken link on line 40 of a long file will not show up here.

Why it matters to agents

The file is a list of places an agent is told to go next. A link that returns 404 sends a coding assistant to an error page, and a link behind a bot challenge sends it nowhere. The link list is the point of the file, so stale entries cost more than a missing blockquote.

How to fix it

  1. Open the details of a failing result in your audit report. It lists each broken URL.
  2. For a 404, update the link to the page's new address or remove the entry. For a redirect chain that ends in an error, link straight to the final working page.
  3. For a 403 or 429 that only affects automated requests, check your CDN or firewall rules for the path. See edge agent access for how the audit looks for that.
  4. Keep the first 10 links the most important ones, and keep them absolute.
## Docs

- [Quickstart](https://example.com/docs/quickstart): Create a project
- [API reference](https://example.com/docs/api): Endpoints and auth

A quick local check, run from a shell, prints the final status for each URL:

grep -oE 'https?://[^ )>]+' llms.txt | while read -r u; do
  printf '%s %s\n' "$(curl -s -L -o /dev/null -w '%{http_code}' "$u")" "$u"
done

Related: llms.txt exists, llms.txt follows the format, and the llms.txt validator, which checks a live URL's links.

Test your own file with the llms.txt validator.

Platform guides

Step-by-step instructions for specific platforms:

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →