Why web_fetch Fails AI Agents — and What to Do Instead

AI agents use web_fetch as their default way of reading the web: send an HTTP request, get HTML back, extract the text, answer the question. It works fine in tutorials. In practice, it fails silently on a surprisingly large slice of the modern web — and when it fails, agents don't say "I don't know." They hallucinate.

We ran a benchmark across 42 test cases — JS-heavy SaaS pricing pages, static documentation, multi-page cloud pricing, and sites that block HTTP entirely — comparing web_fetch against AgentReady's ask_site. The results were cleaner than expected.

The hallucination number is the one that matters most. Agents don't just fail to answer — they confidently answer wrong. An agent that tells a user the wrong Stripe fee, the wrong Zoom meeting limit, or a nonexistent Vercel plan feature is worse than one that says it doesn't know.

Three ways web_fetch fails

1. JavaScript-rendered SPAs return empty HTML. The majority of modern SaaS products — pricing pages especially — render their content client-side with React, Vue, or Angular. A plain HTTP request returns a nearly empty <div id="root"></div>. The agent sees a loading shell and has to guess what's on the page.

2. Bot detection blocks the request entirely. Companies like Webflow actively block HTTP scrapers. The request returns a connection reset or a CAPTCHA wall. The agent gets 0 bytes — but may still hallucinate an answer from its training data.

3. The answer spans multiple pages. "What does the free plan include?" often requires reading a pricing page, a feature comparison table, and a limits FAQ. web_fetch fetches one URL. If the answer isn't on that page, it guesses.

Case study: Webflow

This is the starkest example. Webflow blocks all automated HTTP requests. When an agent calls web_fetch("https://webflow.com/pricing"), the connection resets. Zero bytes. The agent either says it can't access the page, or — more dangerously — hallucinates Webflow pricing from its training data (which is almost certainly stale).

AgentReady's indexed version of the same question returns a complete, cited answer: the free Starter plan, page limits, CMS restrictions, subdomain publishing. All five expected facts covered. The source URL is cited so the user can verify.

The difference: AgentReady indexed Webflow's pricing page using Jina Reader, a cloud service that fetches URLs in a real browser before returning the content. The result is cached and available to any agent query instantly — no per-query crawl latency, no bot detection.

Case study: Zoom

Zoom's pricing page returns 1,363 characters of HTML — a loading shell with no pricing data. web_fetch scores 0/4 facts and is correctly marked not useful. AgentReady scores 3/4: it correctly returns the 40-minute free tier limit and participant count with a citation, and admits uncertainty on the one fact it couldn't find.

How the crawl pipeline actually works

AgentReady's production crawler doesn't just do a raw HTTP fetch. It tries four approaches in order, stopping as soon as it gets enough content.

Layer 1: llms.txt

If the site publishes a machine-readable content summary at /llms.txt or /llms-full.txt — the emerging open standard for AI-readable site indexes — AgentReady uses it directly. Site owners have already curated the content they want agents to see. Zero rendering overhead, highest quality.

Layer 2: HTTP + Readability

A plain HTTP fetch with cheerio-based text extraction: strip navigation, scripts, footers, and sidebars; pull the text from <main> or <article>. This handles all server-rendered sites and is what web_fetch essentially does. Fast, no latency.

Layer 3: __NEXT_DATA__ extraction

This is the layer most people don't know about. When a page built with Next.js renders client-side, the HTML looks empty — but Next.js always embeds the page's initial data as a JSON blob in a <script id="__NEXT_DATA__"> tag so the first render can happen without an API roundtrip.

AgentReady parses this JSON, walks the tree skipping non-content keys like className, style, and src, and extracts the actual page data — pricing tiers, feature lists, copy — without ever running JavaScript. This is why Linear, Stripe, and Vercel index successfully despite being React SPAs. Roughly 40–50% of modern SaaS products use Next.js, making this a high-leverage layer.

Layer 4: Jina Reader

When all HTML-based approaches fail — because the site uses a non-Next.js SPA or actively blocks HTTP requests — AgentReady falls back to Jina Reader. Jina is a cloud service that fetches the URL in a real headless browser, waits for JavaScript to finish executing, and returns clean markdown of the rendered page.

This is how Webflow and Coda get indexed. The tradeoff is latency — Jina adds 5–15 seconds per page — so it only runs when the first three layers return insufficient content. Once indexed, queries against the cached content are instant.

Why indexed RAG hallucinates less

The hallucination gap between web_fetch and indexed RAG is structural, not accidental.

web_fetch passes raw page text to the LLM and says "answer from this." When the content is a loading shell or a partial scrape, the model fills gaps from training data. It doesn't know the page was incomplete — it just sees less text and works with what it has.

Indexed RAG works differently. The LLM only receives chunks that matched a semantic similarity query against a vector database. If no relevant chunk exists, the answer explicitly says so — the model has nothing to hallucinate from. When it does answer, it cites the source URL, making any remaining errors detectable and correctable.

In our benchmark: web_fetch produced 6 high-hallucination answers (confident wrong facts, typically wrong prices or limits). AgentReady produced 3. The three remaining cases were all in multi-page synthesis, where the production auto-index capped at 10 pages and missed the relevant page entirely.

Where web_fetch still wins

One category in our benchmark went the other way: multi-page synthesis, where web_fetch scored 71% vs AgentReady's 62%.

The reason is a current production limitation: AgentReady's auto-index crawls up to 10 pages per domain. For large sites like Cloudflare, Netlify, or Datadog, the specific pricing page may not have been in the first 10 pages crawled. web_fetch fetches the exact pricing URL the benchmark specifies, so it always hits the right page.

This is a solvable problem — AgentReady's deep-index tool already prioritises pricing and docs paths — but it represents a real trade-off in the current production setup. For agents that know exactly which URL to fetch and the page is server-rendered, web_fetch is perfectly fine.

The right mental model

Use web_fetch when:

• You know the exact URL and the page is server-rendered
• The content changes frequently and you need it fresh right now
• The site is simple HTML and you just need one specific fact

Use indexed RAG (AgentReady) when:

• The site is a React SPA, Next.js app, or anything with client-side rendering
• The site blocks HTTP scrapers (most SaaS companies do)
• The question might span multiple pages
• You need citations — the agent should tell you which page it read
• Hallucination risk matters — wrong pricing or wrong API behaviour is worse than no answer

Reproduce the benchmark

The full test suite and runner are open in the AgentReady repository. 42 test cases across 4 categories, GPT-4o as an independent judge, both approaches hitting live production endpoints.

# Run all 42 cases
npx tsx eval/run.ts

# Run a single category
npx tsx eval/run.ts --category js_spa

Results are saved as JSON to eval/results/ and include per-case scores, latency, content length, and hallucination ratings.