llms.txt generator: create a spec-compliant llms.txt in 60 seconds

llms.txt is a plain-text file you place at the root of your website — like robots.txt, but for AI agents. It tells language models and agent runtimes what your site covers and which pages are worth reading. This page explains what it is, how to generate one automatically, and what you need beyond the file itself to make your site fully queryable.

What is llms.txt?

The llmstxt.org spec defines a markdown file that lives at yourdomain.com/llms.txt. It contains a short description of your site followed by grouped sections of links — each link with a human-written description of what that page covers.

The idea is straightforward: instead of asking an AI agent to crawl and parse your entire site to understand what you do, you give it a clean, curated index upfront. The agent reads llms.txt, knows which pages exist and what they contain, and can decide which ones to fetch based on the user's question.

It complements robots.txt rather than replacing it.robots.txt is a blocklist — pages crawlers should skip.llms.txt is a table of contents — pages AI agents should prioritize.

What a spec-compliant llms.txt looks like

A minimal compliant file:

# YourProduct

> A one-sentence description of what your product does.

## Documentation

- [Getting started](https://yourdomain.com/docs/getting-started): Install and run your first query in under 5 minutes.
- [API reference](https://yourdomain.com/docs/api): Full endpoint documentation with request and response examples.
- [Authentication](https://yourdomain.com/docs/auth): API keys, OAuth flows, and token scoping.

## Blog

- [Changelog](https://yourdomain.com/blog/changelog): Recent product updates and new features.

The file must be UTF-8, hosted at the exact path /llms.txt, and follow the markdown heading/link structure the spec defines. Writing it manually is feasible for a small site; for anything with more than 20 pages, generation is faster and more consistent.

How to generate llms.txt automatically

AgentReady generates a spec-compliant llms.txt from any public URL:

1. Paste your URL in the box above (or on the homepage)
2. AgentReady crawls your site via sitemap or link-following — no setup required
3. AI extracts clean content from each page and groups it by topic
4. You get a downloadable llms.txt plus hosted /ask and /mcp endpoints

The whole process takes about 60 seconds for a typical docs site. No account required for the first site. No developer, no configuration files, no deployment pipeline.

llms.txt alone isn't enough

A llms.txt file is a discovery mechanism. It tells an AI agent your content exists and where to find it. But most agent runtimes — Claude Desktop, Cursor, Windsurf — don't automatically fetch and read every page listed in your llms.txt. They need a queryable endpoint to get grounded answers from your content at runtime.

That's why AgentReady also hosts a live /ask RAG endpoint and an /mcp server alongside your llms.txt. When an agent calls ask_site with a question, it gets a cited answer drawn from your indexed content — not a hallucinated summary from training data.

Who should add llms.txt to their site

Any site where AI agents are likely to need accurate, up-to-date information:

• Developer tools with documentation — when developers ask Claude about your API, you want it to answer from your actual docs, not stale training data
• SaaS products with help centers — AI customer support agents need reliable answers from your knowledge base
• Research or content sites — making your content discoverable to AI-assisted research workflows
• E-commerce or service sites — so AI shopping agents can accurately represent your products and policies

The earlier you add llms.txt, the more crawl history you accumulate before agent-driven traffic becomes a meaningful channel. The keyword llms.txt generator is still uncrowded — there's no dominant incumbent, and the sites that act early will define how the category is indexed.