The headline number
Of the 322 sites we scored, 21% earned a D or F — failing grades that mean an AI agent would struggle to extract meaningful value from the site. The average score is 54/100.
The five failure modes
These aren't random failures. Across 136 sites in 9 categories, we see the same five patterns. They stack — a site that fails at discovery also tends to fail at protocols.
No agent discovery layer
Sites don't declare what an agent can do with them. No llms.txt, no agent-card.json, no structured capability signal. Agents can't find the front door.
Protocol adoption is near-zero
MCP server adoption sits at 0% across all 136 sites audited. llms.txt reaches only 32%. The plumbing that enables agent action simply doesn't exist yet.
Auth walls agents can't negotiate
Sites that require login to access their core value — SaaS dashboards, banking portals, enterprise tools — score near 0 on Auth & Access. Most have no OAuth surface, no API key provisioning, no agent-compatible session flow.
Structured data is incomplete or absent
Agents rely on structured data to understand content without rendering it. Most enterprise sites have partial Schema.org coverage — enough for Google, not enough for an agent trying to extract pricing, availability, or contact info.
Bot accessibility is blocked or inconsistent
robots.txt policies often block legitimate AI crawlers alongside scrapers. Rate limits hit agent traffic hard. Sites built to serve millions of humans haven't been tuned for the request patterns of AI agents.
Failure rates by category
Some industries fail harder than others. agent-auth scores the lowest on average (30/100); technical performs best at 89/100 — still a C grade.
| Industry | Avg score | D/F rate |
|---|---|---|
| Local Business | 39 | 100% |
| Media & Content | 40 | 80% |
| Social | 41 | 75% |
| Blog | 43 | 67% |
| Finance | 43 | 57% |
| Government | 43 | 67% |
| E-Commerce | 43 | 100% |
| Other | 44 | 48% |
| Documentation | 48 | 27% |
| E-Commerce | 51 | 29% |
| AI & Tech | 52 | 20% |
| SaaS | 55 | 14% |
| AI & ML | 55 | 25% |
| Developer Tools | 57 | 14% |
| Data & Analytics | 58 | 10% |
| Productivity | 59 | 0% |
| Sales & Marketing | 60 | 0% |
| CRM | 61 | 0% |
| Infrastructure & DevOps | 62 | 5% |
| Identity & Auth | 62 | 0% |
| Agent Tools | 64 | 5% |
| Payments & Fintech | 64 | 0% |
| Communication & Support | 66 | 0% |
Where scores collapse
Across all sites, two subcategories pull scores down most consistently: agent discovery and agent protocols. Technical infrastructure — serving fast, stable responses — is actually the strongest area.
This isn't a web quality problem
The sites in our dataset are not bad websites. Many are household names with millions of daily users, fast CDNs, and polished UX. They fail agents not because they're broken — but because they were never built for agents.
The gap between "good website" and "agent-ready website" is real and measurable. It lives in the absence of discovery layers, protocol adoption, and auth surfaces that agents can negotiate without human intervention.
The good news: these are mostly additive improvements, not rewrites. An llms.txt takes minutes. An agent card takes an afternoon. The structural work is auth — but even there, publishing OAuth endpoints and API key flows is incremental.
What to do about it
Audit your site
Get your current score across all 6 categories. Takes under 30 seconds.
Fix discovery first
Publish llms.txt and agent-card.json. These are the cheapest, highest-signal improvements available.
Check your robots.txt
Review which AI crawlers you're blocking. If your site provides value, letting agents in is usually a win.
Score your site now
Get your free w2agent score and generate the files your site needs.
Get Your Score