Astro + AI Readiness
Astro's static-first architecture is ideal for AI crawlers, but you still need llms.txt and proper configuration. Here's how.
Step-by-step guides
- How to add llms.txt to an Astro site (llms.txt)
Why Astro Is Great for AI (With a Few Gaps)
Astro's static HTML output is exactly what AI crawlers want — fully rendered pages with no JavaScript dependency. But most Astro sites still miss key files AI agents look for.
No llms.txt
AI models need a structured index of your site. Astro doesn't generate llms.txt — your content is invisible to LLMs.
Missing agent-card.json
If your site runs an A2A agent, .well-known/agent-card.json lets other agents discover it — not included by default.
robots.txt may block AI
Default or third-party robots.txt configs sometimes block GPTBot, ClaudeBot, and other AI crawlers.
No structured data
Static Astro pages often lack JSON-LD markup that helps AI understand your content semantically.
What w2agent Generates
Static files first (Astro's strength), with optional endpoints:
public/llms.txtStatic file — zero overhead, works on any host. This is the primary serving method.
public/.well-known/agent-card.jsonStatic A2A agent card, served directly from public/ — generated only when you provide your A2A endpoint (--a2a-endpoint).
w2agent-integration.mjsAstro integration plugin that regenerates files at astro:build:done.
src/pages/llms.txt.tsOptional endpoint that builds llms.txt from your content. In a static site it runs at build time; it runs per request only if you opt out of prerendering.
Example: Astro build integration
// astro.config.mjs
import w2agent from "./w2agent-integration.mjs";
export default defineConfig({
integrations: [w2agent()],
});
// Regenerates llms.txt at build time via astro:build:done hookAstro-Specific Checks
| Check | What It Tests |
|---|---|
| Sitemap integration | Checks if @astrojs/sitemap is installed so AI crawlers can discover all your pages. |
| robots.txt AI bot access | Verifies robots.txt exists and doesn't block AI crawlers like GPTBot or ClaudeBot. |
| Rendering mode detection | Identifies static vs SSR mode — static is ideal for AI crawlers, SSR needs proper cache headers. |
Step-by-Step Integration
CLI coming soon — run the web audit
Generate files
Run the web audit, open the Manual Fixes tab, click Generate, and download the files for your platform.
Score your Astro site now
Get your free w2agent score and generate the files your site needs.
Get Your ScoreCLI coming soon — run the web audit