UTF-8 charset: what the check tests and how to fix it

Passes when 80% of crawled pages declare UTF-8 in a meta tag or the Content-Type header. Low weight; the fix is one meta tag or header.

Technical · Weight 0.5 in the w2agent Score

How common is it?

95% of 1223 measured sites where this check applied pass.

Latest audit of each site under rubric v3.1. Sites where the check did not apply or could not be measured (for example, no llms.txt for the llms.txt checks) are left out. A partial result does not count as a pass.

What this check tests

For each crawled page (the homepage plus a sample of same-site pages) the audit counts the page as good if either of these is true:

  • The HTML has <meta charset="..."> and its value, lower-cased, is exactly utf-8. If no meta charset exists, a <meta http-equiv="Content-Type" content="..."> counts when its content contains the lower-case text utf-8.
  • The Content-Type response header, lower-cased, contains utf-8, for example text/html; charset=utf-8.

The score is the share of good pages, rounded.

  • Pass: 80 or more.
  • Fail: anything under 80. There is no partial result.

Some details change results. The value utf8 without the hyphen does not match. The http-equiv form is compared in a case-sensitive way, so charset=UTF-8 written there does not count on its own, although the same value in the Content-Type header or in <meta charset="UTF-8"> does. A page with <meta charset="iso-8859-1"> is judged only on its header, since the first meta charset found is used and the http-equiv tag is then ignored. The check does not decode the page or look for garbled text, and it does not test whether the declared encoding matches the bytes the server actually sends.

Why it matters to agents

A character encoding declaration is the mechanism that tells a reader how a document's bytes were encoded as characters. The HTML Standard requires that a charset value be an ASCII case-insensitive match for utf-8, and says the encoding declaration must be inside the first 1024 bytes of the document. If a client decodes a page with a different encoding from the one used to write it, accented letters, quotation marks and non-Latin text come out wrong, and the text an agent extracts is damaged.

This page cites no source on how any particular AI crawler handles encodings. The check has weight 0.5, the lowest tier, and the fix is small. If it passes, there is nothing further to do. A failure points at old templates, static files uploaded without headers, or servers that send a legacy default encoding.

How to fix it

  1. Put <meta charset="utf-8"> as the first element inside <head>, before any <title> or long script. The HTML Standard requires the declaration to be within the first 1024 bytes, so a large block of inline CSS or scripts above it can push it out of range.
  2. Save the files as UTF-8. The tag only describes the encoding. It does not convert the file.
  3. Set the server header too: Content-Type: text/html; charset=utf-8. Most web servers and frameworks do this by default; for static hosting check the host's settings for default charset or MIME types. If the meta tag and header disagree, fix the one that is wrong.
  4. Use only one declaration style. The HTML Standard says a document must not contain both a meta charset and an http-equiv encoding declaration, and not more than one meta-based declaration.
  5. Check with curl -sI https://example.com/ | grep -i content-type and by viewing the page source.

If a page returns something other than HTML, such as JSON or plain text, set its charset on its own response header. For other <head> hygiene, see Mobile viewport. Text files that agents read, such as llms.txt and robots.txt, should also be served as UTF-8.

Examples from measured sites

Up to 10 of the highest-scoring sites in each group, from recent audits. Failing includes partial results.

Passing

Failing or partial

Sources

Verified against these sources on

Does your site pass this check?

Check your own site →