Bot Visit Data: Which AI Agents Are Crawling the Web in 2026

We track every AI agent that visits sites in our network. Here's what the data shows.

90-day bot visit summary

Across all sites in our network, we logged 560 bot visits in the past 90 days from 7 distinct AI agents. The landscape is highly concentrated: Amazonbot accounts for 56% of all recorded visits.

560
Total visits (90d)
7
Distinct agents
56%
Amazonbot share

Top AI agents by visit volume

GPTBot and ClaudeBot lead by a wide margin. Newer entrants — GrokBot, Applebot-Extended — are ramping fast. Bot traffic that was essentially zero in 2024 is now a measurable fraction of server load on large sites.

#AgentVisits (90d)ShareFirst seen
1Amazonbot31556%2026-07-24
2GPTBot22340%2026-07-21
3PetalBot153%2026-07-17
4ChatGPT-User31%2026-09-15
5OAI-SearchBot20%2026-07-23
6SemrushBot10%2026-07-26
7ClaudeBot10%2026-07-20

Market share breakdown

OpenAI and Anthropic together represent the dominant share of AI agent traffic. The long tail is growing but fragmented — over a dozen agents each account for under 2% of visits.

Amazonbot
56%
GPTBot
40%
PetalBot
3%
ChatGPT-User
1%
OAI-SearchBot
0%
SemrushBot
0%
ClaudeBot
0%

What this means for your site

AI bot traffic is no longer negligible. Top sites in our network see AI crawlers matching 5–15% of search crawler volume.
robots.txt decisions made in 2024 need revisiting. Many sites opted out of AI training data collection — but GPTBot is now also used for real-time retrieval, not just training.
Sites that serve llms.txt get more structured visits and fewer repeated full-crawls. Discovery layers reduce bot overhead while improving agent accuracy.
Auth walls are a hard stop. Bots that hit a login screen give up. If your core value is behind auth, you're invisible to agents — by design.

Bot tracking data is collected from sites in the w2agent audit network. Agents are identified by User-Agent string against a curated list of known AI crawlers. Data covers the 90 days prior to publication (February–May 2026). For per-site bot analytics, audit your domain and visit the Bot Visits tab in your audit report.

Score your site now

Get your free w2agent score and generate the files your site needs.

Get Your Score