Published August 15, 2026 · 10 min read
What is an agent-ready website?
An agent-ready website is one that AI agents can reliably discover, query, and act on — not just one that renders in a browser. Most websites are not.
TL;DR
- • Most websites fail AI agents: JS rendering gaps, single-URL fetching limits, and zero citations.
- • An agent-ready website has 7 core properties: crawler access, SSR content, llms.txt, sitemap, answer-first content, structured data, and an MCP/ask interface.
- • You can grade your site in 30 seconds:
npx @agentreadyweb/mcp grade yourdomain.com - • AgentReady makes any website agent-ready by hosting the interface layer — no code required.
Why most websites fail AI agents
When an AI agent needs to answer a question about a product, a competitor, or a technical topic, it typically falls back to web_fetch — a direct HTTP request to the URL. This approach has three systemic failure modes:
1. JavaScript rendering
Most modern SaaS websites (Notion, Figma, Webflow, Linear, Loom) use React, Next.js, or Vue. Their HTML shells contain almost no content — everything renders client-side via JavaScript. AI crawlers do not execute JavaScript. They fetch an empty shell and report no content.
Benchmark finding: web_fetch achieves 56% fact coverage on JS-heavy sites vs ask_site at 76% — a 20pp gap.
2. Multi-page answers
A question like "What does the Pro plan include and how does it differ from Team?" requires reading the pricing page, the feature comparison, and possibly the changelog. web_fetch fetches one URL. The answer is never on one page.
3. No citations
web_fetch returns raw text with no attribution layer. The model synthesizes an answer and the user has no way to verify where it came from, whether it is current, or which page to read for more detail. ask_site returns source URLs with every answer — 60/60 vs 0/60 in the benchmark.
The 7 properties of an agent-ready website
These properties are graded A (critical), B (important), or A+ (best-in-class). An agent-ready website achieves at minimum all A and B grades.
Crawlable by AI bots
robots.txt explicitly allows major AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot. 68% of enterprise SaaS sites inadvertently block at least one.
npx @agentreadyweb/mcp grade yourdomain.comServer-side rendered content
All critical content is in the initial HTML response — not client-rendered via JavaScript. AI crawlers do not execute JS. React/Next.js/Vue sites that rely on client hydration return an empty shell.
curl https://yourdomain.com | grep "your key content"llms.txt present
A machine-readable summary of your site at /llms.txt. Describes your product, key pages, API endpoints, and pricing. Signals AI-readiness and gives crawlers a structured entry point.
curl https://yourdomain.com/llms.txtSitemap with accurate dates
XML sitemap submitted to Google Search Console and Bing Webmaster Tools, with real lastmod dates (not today's date for static pages). Freshness signals affect AI citation preference.
curl https://yourdomain.com/sitemap.xmlAnswer-first content structure
44% of ChatGPT citations come from the first third of content. Each page leads with a direct answer to its primary question. Paragraphs are 40–60 words. H2/H3 hierarchy mirrors likely user queries.
Read your page: does the first paragraph answer the title question?Structured data (JSON-LD)
Organization, SoftwareApplication, FAQPage, BreadcrumbList, and Article schemas where relevant. Helps AI systems understand entity relationships, not just page text.
Google Rich Results Test → test your URLHosted MCP server + /ask endpoint
A structured interface layer that agents can call directly instead of parsing HTML. Returns cited answers, handles JS rendering, synthesizes across multiple pages, and exposes a capability manifest.
agentready.it.com → paste your URLHow to grade your website
The AgentReady CLI checks all grading criteria in under 30 seconds:
npx @agentreadyweb/mcp grade yourdomain.comThe command checks: robots.txt AI crawler access, llms.txt presence and quality, sitemap submission status, JavaScript rendering behavior (by comparing HTML response to rendered DOM), and whether the site is already indexed in AgentReady's directory. It returns a letter grade from A to F and exits non-zero below a B — making it usable as a CI gate.
For a detailed visual report, paste your URL at agentready.it.com/grade.
How to make your website agent-ready
Step 1: Fix crawler access (10 minutes)
Check your robots.txt. Add explicit allow rules for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended. 68% of enterprise SaaS sites block at least one. If you use a WAF (Cloudflare, AWS WAF), also whitelist AI crawler user-agents there.
Step 2: Add llms.txt (30 minutes)
Create /llms.txt at your domain root. Include: a one-paragraph product description, key pages (pricing, docs, API reference), and your MCP or /ask endpoint if you have one. AgentReady auto-generates a spec-compliant llms.txt for any indexed site.
Step 3: Verify SSR content (1 hour)
Run curl https://yourdomain.com and confirm your key content (product description, pricing, features) is in the raw HTML response. If not, your site is JS-rendered. Options: enable SSR in your framework (Next.js App Router, Nuxt, SvelteKit), add an AgentReady MCP server, or use a rendering service.
Step 4: Get an MCP server (60 seconds)
Paste your URL at agentready.it.com. AgentReady crawls the site (handling JS rendering), generates llms.txt, and hosts a live MCP server at /api/sites/[id]/mcp. Agents can now query your site with cited answers across all pages.
Frequently asked questions
Is my Docusaurus / GitBook / VitePress site agent-ready?
My site is Next.js — is it agent-ready?
Does having a sitemap make my site agent-ready?
What is the difference between agent-ready and SEO-ready?
Grade your website
Get a letter grade in 30 seconds. Free, no account required.