Agent operator signing directory: what the check tests and how to fix it

Tracked, not scored. For sites that run a signing bot: looks for an http-message-signatures-directory with an Ed25519 key and a signature_agent on your origin.

Agent Auth · Tracked, not scored

How common is it?

0% of 1223 measured sites publish a passing result (observed adoption; this check does not change the score).

Latest audit of each site under rubric v3.1. A site with nothing published counts as not passing. A partial result does not count as a pass.

What this check tests

This is a tracked check: it does not change your score. It is about the operator of a bot, not about whether agents can use your site, so it only means something if your organization runs a crawler or agent that signs its requests.

The audit requests /.well-known/http-message-signatures-directory. The crawl accepts the response only if the status is 200, the media type is exactly application/http-message-signatures-directory+json, and the body differs from what your server returns for a random path under /.well-known/ (so a catch-all page is not mistaken for a file). Then:

  • Skip: nothing accepted. Most sites get this, and it is not a finding.
  • Fail: the body is not a JSON object with a keys array containing an object with kty OKP, crv Ed25519 and a string x.
  • Fail: it has such a key, but no signature_agent member, or one whose origin differs from the audited origin.
  • Pass: both hold.

The audit does not verify any signature, the signature on the directory response, the key validity dates, or that your bot's traffic actually uses these keys.

One requirement here is the audit's own. The IETF Web Bot Auth drafts checked for this page, including the current working group draft (draft-ietf-webbotauth-httpsig-protocol-00) and the earlier directory drafts 00 to 05, define the directory as a plain JSON Web Key Set with a keys array. The origin of the bot is carried in the Signature-Agent request header, not in a signature_agent JSON member, and none of the drafts defines such a member. A directory that follows the draft exactly will therefore fail this check on the second condition.

Why it matters to agents

Web Bot Auth lets an automated client prove who it is. The client signs requests with HTTP Message Signatures (RFC 9421) and adds a Signature-Agent header naming where its public keys are published. A site that wants to tell a real crawler from an impersonator fetches the directory and verifies the signature. The draft registers the well-known path and the media type for that directory, and requires the directory to be served over HTTPS.

Cloudflare documents the same approach for its verified bots: you publish the directory, sign each request, and register the bot with Cloudflare. In its documentation the directory is a standard JSON Web Key Set, and Cloudflare reads only kty, crv and x of an Ed25519 key.

This matters to the site that receives the traffic and decides whether to trust a bot. It tells you nothing about how well a given website serves agents, which is why the check is tracked rather than scored. Web Bot Auth is still an Internet-Draft, so details can change.

How to fix it

Only if you operate a signing agent:

  1. Generate an Ed25519 key pair and keep the private key out of the web root.
  2. Publish the public key as a JWKS at https://bot.example.com/.well-known/http-message-signatures-directory, with Content-Type: application/http-message-signatures-directory+json:
{
  "keys": [{ "kty": "OKP", "crv": "Ed25519", "x": "<base64url public key>" }],
  "signature_agent": "https://bot.example.com"
}
  1. Sign your requests per the draft and RFC 9421, and send Signature-Agent with the same origin.

The signature_agent member is there only because this audit expects it. Cloudflare's documentation says it ignores key parameters other than kty, crv and x; none of the sources checked for this page says whether a verifier tolerates an extra top-level member, so check your verifier before adding it. Serve the file as a real file, not through a catch-all route.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →