Why web_fetch Fails AI Agents — and What to Do Instead

AI agents use web_fetch as their default way of reading the web: send an HTTP request, get HTML back, extract the text, answer the question. It works fine in tutorials. In practice, it fails silently on a surprisingly large slice of the modern web — and when it fails, agents don't say "I don't know." They hallucinate.

We ran a benchmark across 60 test cases — JS-heavy SaaS pricing pages, static documentation, multi-page cloud pricing, exact-fact API reference lookups, and sites that block HTTP entirely — comparing web_fetch against AgentReady's ask_site. The judge scores every expected fact individually and quotes every wrong claim it finds, so the results are auditable down to the fact level.

Facts covered is a statistical tie — and that tie is the interesting part. The two approaches get there very differently: web_fetch racks up facts on pages it can read while confidently fabricating on pages it can't. Its 6 high-hallucination answers and 73 wrong claims are concentrated exactly where an agent has no way to know the answer is wrong. An agent that tells a user the wrong Stripe fee, the wrong Zoom meeting limit, or a nonexistent Retool plan is worse than one that says it doesn't know.

Three ways web_fetch fails

1. JavaScript-rendered SPAs return empty HTML. The majority of modern SaaS products — pricing pages especially — render their content client-side with React, Vue, or Angular. A plain HTTP request returns a nearly empty <div id="root"></div>. The agent sees a loading shell and has to guess what's on the page.

2. Bot detection blocks the request entirely. Companies like Webflow actively block HTTP scrapers. The request returns a connection reset or a CAPTCHA wall. The agent gets 0 bytes — but may still hallucinate an answer from its training data.

3. The answer spans multiple pages. "What does the free plan include?" often requires reading a pricing page, a feature comparison table, and a limits FAQ. web_fetch fetches one URL. If the answer isn't on that page, it guesses.

4. Geo-localized content misleads silently. A new failure mode surfaced in the v2 run: Stripe's pricing page served India-market rates to our benchmark runner's IP, and web_fetch confidently reported "2% for Mastercard and Visa cards issued in India" as Stripe's standard pricing. The answer was technically on the page — and wrong for almost anyone asking.

Case study: Webflow

This is the starkest example. Webflow blocks all automated HTTP requests. When an agent calls web_fetch("https://webflow.com/pricing"), the connection resets. Zero bytes. The agent either says it can't access the page, or — more dangerously — hallucinates Webflow pricing from its training data (which is almost certainly stale).

AgentReady's indexed version of the same question returns a complete, cited answer: the free Starter plan, page limits, CMS restrictions, subdomain publishing. All five expected facts covered. The source URL is cited so the user can verify.

The difference: AgentReady indexed Webflow's pricing page using Jina Reader, a cloud service that fetches URLs in a real browser before returning the content. The result is cached and available to any agent query instantly — no per-query crawl latency, no bot detection.

Case study: Zoom

Zoom's pricing page returns 1,363 characters of HTML — a loading shell with no pricing data. web_fetch scores 0/4 facts and is correctly marked not useful. AgentReady scores 3/4: it correctly returns the 40-minute free tier limit and participant count with a citation, and admits uncertainty on the one fact it couldn't find.

How the crawl pipeline actually works

AgentReady's production crawler doesn't just do a raw HTTP fetch. It tries four approaches in order, stopping as soon as it gets enough content.

Layer 1: llms.txt

If the site publishes a machine-readable content summary at /llms.txt or /llms-full.txt — the emerging open standard for AI-readable site indexes — AgentReady uses it directly. Site owners have already curated the content they want agents to see. Zero rendering overhead, highest quality.

Layer 2: HTTP + Readability

A plain HTTP fetch with cheerio-based text extraction: strip navigation, scripts, footers, and sidebars; pull the text from <main> or <article>. This handles all server-rendered sites and is what web_fetch essentially does. Fast, no latency.

Layer 3: __NEXT_DATA__ extraction

This is the layer most people don't know about. When a page built with Next.js renders client-side, the HTML looks empty — but Next.js always embeds the page's initial data as a JSON blob in a <script id="__NEXT_DATA__"> tag so the first render can happen without an API roundtrip.

AgentReady parses this JSON, walks the tree skipping non-content keys like className, style, and src, and extracts the actual page data — pricing tiers, feature lists, copy — without ever running JavaScript. This is why Linear, Stripe, and Vercel index successfully despite being React SPAs. Roughly 40–50% of modern SaaS products use Next.js, making this a high-leverage layer.

Layer 4: Jina Reader

When all HTML-based approaches fail — because the site uses a non-Next.js SPA or actively blocks HTTP requests — AgentReady falls back to Jina Reader. Jina is a cloud service that fetches the URL in a real headless browser, waits for JavaScript to finish executing, and returns clean markdown of the rendered page.

This is how Webflow and Coda get indexed. The tradeoff is latency — Jina adds 5–15 seconds per page — so it only runs when the first three layers return insufficient content. Once indexed, queries against the cached content are instant.

Why indexed RAG hallucinates less

The hallucination gap between web_fetch and indexed RAG is structural, not accidental.

web_fetch passes raw page text to the LLM and says "answer from this." When the content is a loading shell or a partial scrape, the model fills gaps from training data. It doesn't know the page was incomplete — it just sees less text and works with what it has.

Indexed RAG works differently. The LLM only receives chunks that matched a semantic similarity query against a vector database. If no relevant chunk exists, the answer explicitly says so — the model has nothing to hallucinate from. When it does answer, it cites the source URL, making any remaining errors detectable and correctable.

In our v2 benchmark: web_fetch produced 6 high-hallucination answers and 73 judge-quoted wrong claims (typically wrong prices or limits — it fabricated a complete Retool pricing table from partial marketing text). AgentReady produced 1 and 21. The one remaining case is instructive: on MongoDB's pricing page, retrieval surfaced the paid Flex tier's "5GB storage" chunk for a question about the free tier (512MB). Retrieval grounds answers in real page content, but adjacent-tier confusion within a page is still possible — section-aware chunking is next on the roadmap.

Worth noting: multi-page synthesis — the category AgentReady lost in the July 8 v1 run because of a 10-page crawl cap — reversed after crawl-pipeline fixes shipped that week. In v2 AgentReady wins it 78% vs 71%.

Where web_fetch still wins

The v2 benchmark added a category designed to test web_fetch's home turf: 12 exact-fact API reference lookups (Kubernetes restartPolicy values, PostgreSQL JOIN types, Dockerfile instructions) where the agent knows exactly which page holds the answer and that page is server-rendered.

web_fetch won it: 91% vs 77%. When the exact page is known, fetchable, and fits in context, reading the whole page beats retrieving chunks — RAG retrieval occasionally surfaces an adjacent-but-wrong section. It also edged static SSR docs, 86% vs 81%. We report this because it's true, and because it sharpens the real conclusion: the two tools are complements, not substitutes.

The right mental model

Use web_fetch when:

• You know the exact URL and the page is server-rendered — API reference lookups are its best case (91% in our benchmark)
• The content changes frequently and you need it fresh right now
• The site is simple HTML and you just need one specific fact

Use indexed RAG (AgentReady) when:

• The site is a React SPA, Next.js app, or anything with client-side rendering
• The site blocks HTTP scrapers (most SaaS companies do)
• The question might span multiple pages
• You need citations — the agent should tell you which page it read
• Hallucination risk matters — wrong pricing or wrong API behaviour is worse than no answer

Reproduce the benchmark

The full test suite and runner live in the AgentReady repository. 60 test cases across 5 categories, GPT-4o as an independent judge scoring each expected fact individually, both approaches hitting live production endpoints.

# Run all 60 cases
npx tsx eval/run.ts

# Run a single category or case
npx tsx eval/run.ts --category js_spa
npx tsx eval/run.ts --only stripe_fees

Every run writes raw JSON plus a markdown report to eval/results/ — per-case fact tables, judge-quoted wrong claims, citation counts, and latency percentiles.