What this check tests
For each crawled page (the homepage plus a sample of same-site pages) the audit counts the page as good if either of these is true:
- The HTML has
<meta charset="...">and its value, lower-cased, is exactlyutf-8. If nometa charsetexists, a<meta http-equiv="Content-Type" content="...">counts when its content contains the lower-case textutf-8. - The
Content-Typeresponse header, lower-cased, containsutf-8, for exampletext/html; charset=utf-8.
The score is the share of good pages, rounded.
- Pass: 80 or more.
- Fail: anything under 80. There is no partial result.
Some details change results. The value utf8 without the hyphen does not match. The http-equiv form is compared in a case-sensitive way, so charset=UTF-8 written there does not count on its own, although the same value in the Content-Type header or in <meta charset="UTF-8"> does. A page with <meta charset="iso-8859-1"> is judged only on its header, since the first meta charset found is used and the http-equiv tag is then ignored. The check does not decode the page or look for garbled text, and it does not test whether the declared encoding matches the bytes the server actually sends.
Why it matters to agents
A character encoding declaration is the mechanism that tells a reader how a document's bytes were encoded as characters. The HTML Standard requires that a charset value be an ASCII case-insensitive match for utf-8, and says the encoding declaration must be inside the first 1024 bytes of the document. If a client decodes a page with a different encoding from the one used to write it, accented letters, quotation marks and non-Latin text come out wrong, and the text an agent extracts is damaged.
This page cites no source on how any particular AI crawler handles encodings. The check has weight 0.5, the lowest tier, and the fix is small. If it passes, there is nothing further to do. A failure points at old templates, static files uploaded without headers, or servers that send a legacy default encoding.
How to fix it
- Put
<meta charset="utf-8">as the first element inside<head>, before any<title>or long script. The HTML Standard requires the declaration to be within the first 1024 bytes, so a large block of inline CSS or scripts above it can push it out of range. - Save the files as UTF-8. The tag only describes the encoding. It does not convert the file.
- Set the server header too:
Content-Type: text/html; charset=utf-8. Most web servers and frameworks do this by default; for static hosting check the host's settings for default charset or MIME types. If the meta tag and header disagree, fix the one that is wrong. - Use only one declaration style. The HTML Standard says a document must not contain both a
meta charsetand anhttp-equivencoding declaration, and not more than onemeta-based declaration. - Check with
curl -sI https://example.com/ | grep -i content-typeand by viewing the page source.
If a page returns something other than HTML, such as JSON or plain text, set its charset on its own response header. For other <head> hygiene, see Mobile viewport. Text files that agents read, such as llms.txt and robots.txt, should also be served as UTF-8.