Declared AI usage preferences: what the check tests and how to fix it

Tracked, not scored. Passes if you declare AI usage preferences via Content-Signal, Content-Usage, License or TDMRep. It records the declaration only.

Bot Accessibility · Tracked, not scored

How common is it?

13% of 1223 measured sites publish a passing result (observed adoption; this check does not change the score).

Latest audit of each site under rubric v3.1. A site with nothing published counts as not passing. A partial result does not count as a pass.

What this check tests

This is a tracked check: it does not change your score. The audit looks for an AI usage preference in up to five places:

SourceWhere the audit looks
Content-SignalA Content-Signal: line in robots.txt, with comma-separated name=yes or name=no pairs
Content-Usage (robots.txt)A Content-Usage: rule in robots.txt, optionally with a path pattern
LicenseA License: line in robots.txt whose value is an absolute URI
Content-Usage (HTTP)A Content-Usage header on the homepage response
TDMRep/.well-known/tdmrep.json, a JSON array of rules with tdm-reservation values

If at least one is present the check passes. If none is, it is skipped (there is no fail). The result also notes whether a training opt-out was declared, whether training is permitted, or whether the training preference is limited to certain agents or paths. A training preference is read from ai-train in Content-Signal, train-ai in Content-Usage, and tdm-reservation in TDMRep. It also notes when the robots.txt carries a CDN-managed Content-Signal block (Cloudflare's managed robots.txt) instead of one you wrote.

These are statements of preference. The check does not test whether any crawler reads or follows them.

Why it matters to agents

Robots.txt says whether a crawler may fetch a URL. It does not say what the content may be used for once fetched. Several vocabularies fill that gap:

  • Cloudflare's Content Signals policy defines three signals, search, ai-input and ai-train, each set to yes or no in robots.txt. Leaving one out means you express no preference on it.
  • The IETF AI Preferences (aipref) working group has drafts for attaching preferences to content with a Content-Usage HTTP header and a Content-Usage rule in robots.txt, and a vocabulary of categories that includes train-ai, ai-use and search. Both are Internet-Drafts on the verification date, so syntax may change.
  • TDMRep is a W3C Community Group report for text-and-data-mining reservations. RSL 1.0 defines a License: directive in robots.txt that points to a license document.

Declaring a preference is a signal, not enforcement. Treat it as one layer next to robots.txt rules and network-level controls.

How to fix it

Pick the vocabulary that matches your intent and add it. A robots.txt using Content-Signal, in the form the policy documents:

User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

The same intent in the draft Content-Usage syntax, as it appears in the draft's examples:

User-agent: *
Allow: /
Content-Usage: train-ai=n

Or as an HTTP response header on your homepage: Content-Usage: train-ai=n.

For TDMRep, serve /.well-known/tdmrep.json with a rule reserving rights for the whole site:

[{ "location": "/", "tdm-reservation": 1 }]
  • Decide first. Cloudflare's managed default is search=yes, ai-train=no, but yes is just as valid for training if that is your intent.
  • A training opt-out is a statement about use, not an access rule. Whether agents may fetch your pages is set by your robots.txt groups; see AI agent access.
  • If you use Cloudflare's managed robots.txt, a default block may already be there; review what it says before relying on it.
  • After deploying, fetch robots.txt and the homepage headers yourself to confirm the lines and header are served as written.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →