What this check tests
This is a tracked check: it does not change your score. The audit looks for an AI usage preference in up to five places:
| Source | Where the audit looks |
|---|---|
| Content-Signal | A Content-Signal: line in robots.txt, with comma-separated name=yes or name=no pairs |
| Content-Usage (robots.txt) | A Content-Usage: rule in robots.txt, optionally with a path pattern |
| License | A License: line in robots.txt whose value is an absolute URI |
| Content-Usage (HTTP) | A Content-Usage header on the homepage response |
| TDMRep | /.well-known/tdmrep.json, a JSON array of rules with tdm-reservation values |
If at least one is present the check passes. If none is, it is skipped (there is no fail). The result also notes whether a training opt-out was declared, whether training is permitted, or whether the training preference is limited to certain agents or paths. A training preference is read from ai-train in Content-Signal, train-ai in Content-Usage, and tdm-reservation in TDMRep. It also notes when the robots.txt carries a CDN-managed Content-Signal block (Cloudflare's managed robots.txt) instead of one you wrote.
These are statements of preference. The check does not test whether any crawler reads or follows them.
Why it matters to agents
Robots.txt says whether a crawler may fetch a URL. It does not say what the content may be used for once fetched. Several vocabularies fill that gap:
- Cloudflare's Content Signals policy defines three signals,
search,ai-inputandai-train, each set to yes or no in robots.txt. Leaving one out means you express no preference on it. - The IETF AI Preferences (aipref) working group has drafts for attaching preferences to content with a
Content-UsageHTTP header and aContent-Usagerule in robots.txt, and a vocabulary of categories that includestrain-ai,ai-useandsearch. Both are Internet-Drafts on the verification date, so syntax may change. - TDMRep is a W3C Community Group report for text-and-data-mining reservations. RSL 1.0 defines a
License:directive in robots.txt that points to a license document.
Declaring a preference is a signal, not enforcement. Treat it as one layer next to robots.txt rules and network-level controls.
How to fix it
Pick the vocabulary that matches your intent and add it. A robots.txt using Content-Signal, in the form the policy documents:
User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /
The same intent in the draft Content-Usage syntax, as it appears in the draft's examples:
User-agent: *
Allow: /
Content-Usage: train-ai=n
Or as an HTTP response header on your homepage: Content-Usage: train-ai=n.
For TDMRep, serve /.well-known/tdmrep.json with a rule reserving rights for the whole site:
[{ "location": "/", "tdm-reservation": 1 }]
- Decide first. Cloudflare's managed default is
search=yes, ai-train=no, butyesis just as valid for training if that is your intent. - A training opt-out is a statement about use, not an access rule. Whether agents may fetch your pages is set by your robots.txt groups; see AI agent access.
- If you use Cloudflare's managed robots.txt, a default block may already be there; review what it says before relying on it.
- After deploying, fetch robots.txt and the homepage headers yourself to confirm the lines and header are served as written.