What is MCP?
The Model Context Protocol (MCP) is an open standard that lets AI agents interact with external tools and data sources. Instead of scraping your HTML, an AI agent connects to your MCP server and calls structured tools — like searching your site or reading a specific page.
Think of it as an API designed specifically for AI. Claude Desktop, Cursor, Windsurf, and other AI clients already support MCP natively.
How w2agent Agents Work
1. Crawl
w2agent crawls your site, extracts content, and converts pages to clean markdown.
2. Index
Pages are indexed for full-text search with titles, URLs, and structured content.
3. Serve
An agent endpoint goes live instantly — AI agents can connect and query your content.
Agentize in one command
npx w2agent agentize https://your-site.comHosted vs Self-Host
By default, w2agent hosts your agent server for free on w2agent.dev. If you need full control, you can self-host on Vercel or Cloudflare Workers.
| Hosted | Self-Host (Vercel) | Self-Host (CF) | |
|---|---|---|---|
| Setup | One click or CLI | --self-host vercel | --self-host cf |
| Infra | w2agent.dev (free) | Your Vercel account | Your CF account |
| Updates | Webhooks / CLI | Redeploy | Redeploy |
| Custom domain | slug.w2agent.dev | Your domain | Your domain |
Auto-Detected Tools
Every agent server w2agent deploys exposes these tools automatically:
list-pagesReturns all indexed pages with titles and URLs.get-pageFetches the full markdown content of a specific page.search-siteFull-text search across all indexed pages.search-faqSearches FAQ-style content extracted from your site.Connecting AI Clients
Once deployed, add your agent to any compatible AI client:
Claude Desktop — claude_desktop_config.json
{
"mcpServers": {
"my-site": {
"url": "https://my-site.agent.w2agent.dev"
}
}
}Cursor — .cursor/mcp.json
{
"mcpServers": {
"my-site": {
"url": "https://my-site.agent.w2agent.dev"
}
}
}Keeping Content Fresh
Your agent server reflects the content at deploy time. To keep it current:
Webhooks
w2agent provides a webhook URL at deploy time. Wire it to your CMS (WordPress save_post, Shopify webhooks) to re-index on publish.
CLI update
Run npx w2agent agent update to re-crawl and refresh the index on demand.
Platform plugins
The generated WordPress plugin and Next.js API route call the webhook automatically when content changes.
Get Started
Run an audit to get your w2agent score, then agentize your site in one click — or one command.
Security and Authentication
Agent servers expose your site content over HTTP — but that doesn't mean the content is more public than your website. The server only serves what your site already makes publicly available; it crawls your public-facing pages and indexes them.
For self-hosted deployments, you can add API key authentication to the MCP endpoint so only authorized agents can connect. w2agent's hosted agents use rate limiting to prevent abuse: each endpoint is limited by IP and API key, and unusual traffic patterns (bulk downloads, unusually high search rates) trigger automatic throttling.
Pair your agent server with an agent-card.json that declares the interface URL — this lets orchestrators discover and connect to your agent without manual configuration.
Response Times and Caching
Coding agents are latency-sensitive: a slow MCP tool call breaks the flow of a coding session. w2agent agent servers cache all indexed content in-memory, so list-pages and search-site respond in under 50ms for most sites. The get-page tool serves pre-converted markdown, not a live fetch — so it's fast regardless of your origin's speed.
Agent servers complement llms.txt for different use cases: llms.txt is a static file for passive AI discovery; agent servers provide active, queryable access for agents that need to search or retrieve specific content. The SKILL.md format is how coding agents discover and configure these connections.
Related Articles
- agent-card.json — Declare your agent server's endpoints so AI agents can discover them.
- SKILL.md — The format coding agents use to auto-configure connections to your server.
- What is llms.txt? — Passive content discovery that complements an active agent server.
Score your site now
Get your free w2agent score and generate the files your site needs.
Get Your Score