July 2, 2026 · 6 min read
There's a gap between what developers expect AI agents to do and what they actually can do. Developers assume that Claude, GPT-4, or any capable LLM can "just go look at" a website. The reality is messier. Here's how website discovery actually works for AI agents today — and what it means for your documentation.
AI agents have three ways to know about a website:
Training data — if the model was trained on content from your site, it may know it. But training data has a cutoff, knowledge degrades for niche content, and specifics (exact API parameters, current pricing, recent changelog) are unreliable.
Tools — the agent can call a tool at runtime. This might be a web search tool (returns a list of URLs and snippets), a browser tool (fetches and parses a specific page), or a domain-specific tool like an MCP server. Tools are accurate because they return current content.
Context window — you paste content directly into the conversation. Works but doesn't scale — you can only paste so much, and you have to know what to paste.
Say a developer asks Claude "how do I paginate results with the Stripe API?". Here's what Claude actually does, depending on what tools it has:
No tools: Claude answers from training data. If Stripe's pagination API changed since the training cutoff, Claude gives the old answer. If it's an obscure endpoint, Claude guesses. Either way, there's no citation and no guarantee of accuracy.
Web search tool: Claude searches "Stripe API pagination" and gets back a results list with snippets. It picks the most relevant result, reads the snippet, and generates an answer. Better than pure training data, but it's still reading a snippet — not the full docs page, and not multiple pages.
Browser/fetch tool: Claude fetches https://stripe.com/docs/api/pagination and gets raw HTML. It strips the HTML and reads the text. Good if it fetches the right URL. Limited because it's one page — if the full answer spans multiple docs pages, it misses them.
MCP + indexed content: Claude calls ask_site("stripe.com", "how do I paginate results?"). AgentReady runs a semantic search over pre-indexed chunks from multiple Stripe docs pages and returns the most relevant content with citations. Most accurate and most efficient.
The browser/fetch approach sounds good on paper — just fetch the page. The problems:
JavaScript rendering — most modern sites require JavaScript execution to render content. An HTTP fetch returns the HTML shell. If you inspect the raw HTML of a React or Next.js SPA, you'll see a near-empty document. The content is only there after JavaScript runs, which requires a real browser. Most agent tools don't run JavaScript.
One page at a time — fetching one URL gives you one page. Your docs have 50 pages. The agent can't discover the others without following links, which requires multiple round-trips and takes time. In practice, agents access one or two pages and stop.
Context window limits — even if an agent could fetch all 50 pages, it can't fit them all in context. It needs to know which pages are relevant before fetching them — which is a retrieval problem, which is what RAG solves.
The practical answers are:
1. Make sure your site renders real HTML server-side, not just via JavaScript. Test by disabling JS and loading your page.
2. Publish an llms.txt file so models trained in the future have a good summary of your site to work with.
3. Index your docs via an MCP service like AgentReady so agents can do semantic search over all your content, not just fetch individual pages.
The third option is the most impactful because it works at runtime — when a user is actively asking Claude about your product right now, not when the next training cycle runs.
The web was built for browsers. Search engines adapted it for human-initiated keyword queries. AI agents need something different: structured, queryable, semantically indexed content that they can retrieve relevant chunks from in response to natural-language questions. That's what agent-readiness means — and it's a different problem from SEO, even if the surface overlap (crawlability, content quality) looks similar.