How to Create an llms.txt File for Your Website

llms.txt is a plain-text file that tells AI agents what your website is, what it contains, and where to find the most important content. Think of it as robots.txt — but instead of telling crawlers what to avoid, it tells AI agents exactly what to read.

The spec is simple, adoption is growing fast, and adding it to your site takes about 15 minutes. Here's everything you need to know.

Why llms.txt exists

When an AI agent tries to understand your website, it faces a problem: your site is built for human eyes. Navigation menus, hero images, cookie banners, and JavaScript-rendered content are all noise to an agent trying to answer a question about your product.

llms.txt cuts through that. It's a structured, human-readable, machine-parseable summary of your site that agents can fetch in a single request and immediately understand your content hierarchy. No crawling needed, no JS rendering, no HTML parsing.

It was proposed by Jeremy Howard in 2024 and has since been adopted by Anthropic, Vercel, Supabase, Cloudflare, and hundreds of other developer-facing products.

The format

An llms.txt file is Markdown. The spec defines a small set of conventions:

# Your Product Name

> One-sentence description of what your product does and who it's for.

Optional longer paragraph with more context — product category,
key differentiators, use cases.

## Section Name

- [Page Title](https://yoursite.com/page): One-line description of what this page covers.
- [Another Page](https://yoursite.com/other): What an agent will find here.

## Another Section

- [API Reference](https://yoursite.com/api): Full API documentation.
- [Pricing](https://yoursite.com/pricing): Plans, limits, and costs.

## Optional

- [Changelog](https://yoursite.com/changelog): Recent updates.
- [Blog](https://yoursite.com/blog): Articles and guides.

The structure is:

• H1 — your product name
• Blockquote — a one-sentence description (required)
• Optional paragraph — more context
• H2 sections — content categories
• Bullet links — [Title](url): description for each page

A real example

Here's what a documentation site's llms.txt might look like:

# Acme Payments

> Acme is a payment processing API for developers — accept cards,
> bank transfers, and crypto in minutes with a single integration.

Acme handles the compliance, fraud detection, and global coverage.
Developers integrate once and support 135+ currencies out of the box.

## Getting Started

- [Quickstart](https://docs.acme.com/quickstart): Install the SDK
  and charge your first card in under 5 minutes.
- [Authentication](https://docs.acme.com/auth): API keys, test mode,
  and environment setup.
- [Webhooks](https://docs.acme.com/webhooks): Listen to payment events
  in real time.

## Core API

- [Charges](https://docs.acme.com/api/charges): Create, capture, and
  refund payments.
- [Customers](https://docs.acme.com/api/customers): Save and manage
  customer payment methods.
- [Subscriptions](https://docs.acme.com/api/subscriptions): Recurring
  billing and plan management.

## Pricing

- [Plans](https://acme.com/pricing): 2.9% + 30¢ per transaction on
  standard. Volume discounts available.
- [Fee calculator](https://acme.com/pricing/calculator): Estimate
  your monthly costs.

## Optional

- [Changelog](https://docs.acme.com/changelog): API updates and
  breaking changes.
- [Status](https://status.acme.com): Uptime and incident history.

llms-full.txt: the extended version

The spec also defines llms-full.txt, which is the same format but includes the full content of each linked page inline, rather than just links. This lets an agent get everything in a single request without following links.

llms-full.txt can get large — Vercel's is several hundred KB — so it's optional. Start with llms.txt and add the full version later if you want to optimise for agents that prefer a single-request approach.

Where to host it

Serve it at the root of your domain: https://yoursite.com/llms.txt. Content type should be text/plain.

In Next.js, the easiest approach is a static file in public/llms.txt. For a docs site that changes frequently, generate it dynamically:

// app/llms.txt/route.ts — Next.js dynamic route
export async function GET() {
  const content = `# Your Product

> What your product does.

## Docs

- [Getting Started](https://yoursite.com/docs): ...
`
  return new Response(content, {
    headers: { 'Content-Type': 'text/plain' },
  })
}

What to include

Do include:

• Your most important documentation pages — quickstart, API reference, authentication
• Pricing and plan information — agents frequently need this
• Changelog — helps agents know what's recent
• The pages you most want agents to cite when answering questions about your product

Don't include:

• Every page on your site — agents don't need your cookie policy or 404 page
• Authenticated or private pages
• Pages that change so frequently the link would be stale within hours

Who reads llms.txt

Right now, the primary readers are AI indexing services — platforms that crawl the web to build knowledge bases agents can query. AgentReady, for example, checks for llms.txt as the first step in its indexing pipeline. When it finds one, it uses it directly as a high-quality content source rather than crawling and parsing HTML.

As agents become more capable of autonomous web browsing, llms.txt will likely become as standard as robots.txt — a basic expectation for any professional website. Adding it now is low-effort and high-upside.

Check if your site is agent-ready

Once you've added your llms.txt, you can verify it's working and check your site's overall agent-readiness score at AgentReady's grade tool — it checks for llms.txt, robots.txt, sitemap, readable HTML, and whether the site is already indexed for agent queries.