Auto-Generated llms.txt: Why Your Documentation Needs It in 2026

Your documentation has a new primary reader, and it isn't a human.

It's Cursor pulling context for a developer building on your API. It's Claude Code trying to understand how your SDK works. It's Copilot suggesting the correct function signature based on your latest release notes. In 2026, AI agents read more documentation than humans do, and they read it differently.

They don't care about your carefully crafted sidebar navigation. They can't render your interactive code tabs. They can't click through a visual product tour. What they need is a clean, structured, machine-readable manifest of what your product does and where to learn more.

That manifest is called llms.txt, and if your docs don't have one, you're invisible to the tools your users actually use to code.

This post covers why llms.txt matters, what a good one looks like, and how Dokly auto-generates one for every documentation site—no plugins, no config, no enterprise tier required.

The Problem: AI Agents Can't Read Your Docs#

Modern documentation sites are optimized for humans. They're built with fancy React components, client-side rendering, lazy-loaded images, and collapsible sections that make content look polished in a browser.

AI agents see almost none of that.

When Cursor or Claude Code tries to understand your product, it typically does one of three things:

1. Crawls your sitemap and tries to parse hundreds of HTML pages, most of which are wrapper markup and navigation noise
2. Gives up and relies on whatever stale information was in its training data
3. Hits your docs through a search index and pulls back fragments with no structural context

None of these produce good results. The agent ends up with partial information, outdated examples, or worse—it hallucinates API endpoints that don't exist.

The consequences show up in your support inbox:

• Developers filing bugs that are actually "the AI gave me wrong code"
• Users complaining that your SDK "doesn't work with Cursor" when it works fine
• Declining adoption among AI-native developers who evaluate tools by asking their coding agent first

Your docs might be beautiful. They might be comprehensive. But if they aren't machine-readable, the next generation of developers is getting a broken version of your product.

What Is llms.txt, Exactly?#

llms.txt is a proposed standard—pioneered by Jeremy Howard in 2024 and adopted broadly through 2025 and 2026—that gives AI agents a single, predictable entry point for understanding a website.

You place a file at yourdomain.com/llms.txt. It uses a simple Markdown structure:

# Your Product Name
 
> A one-sentence description of what your product does.
 
A short paragraph with more context—who it's for, the core use case, any
key concepts an agent needs to understand before diving into the docs.
 
## Core Concepts
 
- [Authentication](https://docs.yoursite.com/auth): How to authenticate API requests
- [Data Models](https://docs.yoursite.com/models): The primary resource types
 
## Guides
 
- [Quick Start](https://docs.yoursite.com/quickstart): Get up and running in 5 minutes
- [Building Your First Integration](https://docs.yoursite.com/first-integration)
 
## API Reference
 
- [REST API](https://docs.yoursite.com/api/rest)
- [Webhooks](https://docs.yoursite.com/api/webhooks)

That's it. No XML, no JSON-LD, no schema definitions. Just Markdown headings, a one-line summary, and curated links with descriptions.

The format is deliberately simple because the goal is universal—any AI agent, regardless of sophistication, should be able to parse it and build a mental model of your product in under a second.

The Difference Between llms.txt and sitemap.xml#

A common mistake is treating llms.txt like sitemap.xml. They serve completely different purposes.

• sitemap.xml is a dump of every URL on your site. It exists to help search crawlers find every page. It has no editorial judgment and no narrative.
• llms.txt is a curated map of the pages that matter for understanding your product. It's written for a reader (the AI) that has a limited context window and needs to decide, in seconds, where to look next.

Think of sitemap.xml as a phone book and llms.txt as the table of contents in a textbook. One lists everything; the other explains what's important.

Why "llms-full.txt" Exists#

Some sites also publish llms-full.txt—a single file containing the full text of all documentation concatenated together.

Anthropic, Stripe, and Vercel all do this. The reasoning: if an agent has a large enough context window, it's cheaper for the agent (and faster for your users) to pull one big file than to make dozens of HTTP requests crawling your docs.

A well-equipped documentation platform generates both:

• llms.txt for agents that want to navigate intelligently
• llms-full.txt for agents that want everything at once

The Hidden Cost of Doing This Manually#

You can absolutely write llms.txt by hand. It's just Markdown. But consider what that actually means in practice.

Every time you add a new doc page, you need to remember to update llms.txt. Every time you restructure your information architecture, you need to regenerate the links. Every time someone on your team ships a new guide, there's a new chance to forget.

Within six months, your hand-maintained llms.txt is out of date. It links to pages that have been renamed. It's missing the three new tutorials you shipped last sprint. The descriptions reference a product version that no longer exists.

For a solo developer or a small team, this is exactly the kind of maintenance work that gets dropped first. And the moment it drops, your AI visibility decays.

There's also the full-text version. Generating llms-full.txt manually means concatenating every markdown file in your docs, handling code blocks correctly, preserving front-matter metadata, and re-running the whole process every time anything changes.

Nobody does this consistently. That's why most sites either don't have llms.txt at all or have a broken, stale version that's worse than nothing.

How Dokly Solves It#

Dokly auto-generates llms.txt and llms-full.txt for every documentation site, on every publish. You write your docs normally in the visual editor. When you hit publish, both files regenerate automatically.

Here's what happens behind the scenes:

1. Structure Detection#

Dokly reads your documentation hierarchy—your groups, sections, and pages—and uses them to build the outline of your llms.txt. Your top-level groups become ## headings. Your pages become bullet-pointed links.

You don't manually curate this. If you organize your docs well for humans, the AI-friendly version is organized well too.

2. Description Generation#

For each linked page, Dokly pulls the description from your front-matter description field, or—if you haven't set one—automatically generates a short summary from the first paragraph of the page.

This means the descriptions an AI agent sees are the same descriptions a human sees in search previews. You maintain one source of truth.

3. Full-Text Compilation#

For llms-full.txt, Dokly concatenates the markdown source of every published page, in the order they appear in your navigation. Code blocks are preserved. Front-matter is stripped. The output is a single, clean text file ready to be dropped into any AI's context window.

4. Automatic Republishing#

Every time you publish a change—whether it's a new page, a renamed section, or a typo fix—both files regenerate. There's no build step to trigger, no cron job to maintain. The files are always in sync with what's actually on your docs site.

5. Standards Compliance#

The generated files follow the llms.txt specification exactly. Dokly updates the generator as the spec evolves, so you're not stuck on an outdated format.

What a Dokly-Generated llms.txt Looks Like#

Here's an example from a real Dokly site, showing a SaaS API's auto-generated llms.txt:

# Flowpipe API
 
> Payment orchestration API for subscription businesses.
 
Flowpipe is a REST API that lets you accept payments, manage subscriptions,
and handle billing across multiple payment processors from a single
integration. This documentation covers authentication, core resources, and
common workflows.
 
## Getting Started
 
- [Quick Start](https://docs.flowpipe.dev/quickstart): Make your first API call in under 5 minutes
- [Authentication](https://docs.flowpipe.dev/auth): Using API keys and OAuth2
- [Sandbox Environment](https://docs.flowpipe.dev/sandbox): Test without real transactions
 
## Core Resources
 
- [Customers](https://docs.flowpipe.dev/customers): Create and manage customer records
- [Subscriptions](https://docs.flowpipe.dev/subscriptions): Recurring billing and plan changes
- [Invoices](https://docs.flowpipe.dev/invoices): Generate and send invoices
- [Webhooks](https://docs.flowpipe.dev/webhooks): Real-time event notifications
 
## Guides
 
- [Migrating from Stripe](https://docs.flowpipe.dev/guides/stripe-migration)
- [Handling Failed Payments](https://docs.flowpipe.dev/guides/failed-payments)
- [Building a Usage-Based Plan](https://docs.flowpipe.dev/guides/usage-based)
 
## API Reference
 
- [REST API](https://docs.flowpipe.dev/api): Complete endpoint reference
- [Error Codes](https://docs.flowpipe.dev/errors): All error responses explained
- [Rate Limits](https://docs.flowpipe.dev/rate-limits): Request limits by plan

Notice what's happening here. An AI agent hitting this file gets, in under 50 lines:

• The product's purpose
• The primary use case
• A map of where to learn about each major concept
• Descriptions that explain when to follow each link

That's the entire job of llms.txt. And it was generated automatically, with no manual curation.

What This Means for Your Product#

If you sell a developer tool, an API, or any product that developers integrate into their code, your AI visibility is now a competitive feature.

Consider the user journey in 2026:

1. A developer asks their AI agent "what's the best way to handle subscription billing in my Next.js app?"
2. The agent has been trained on data from months or years ago. To give current advice, it needs to pull live documentation.
3. It fetches llms.txt from every major player in the space—Stripe, Lemon Squeezy, Paddle, your product.
4. It uses those files to decide which product to recommend and which docs to read in detail.

If your llms.txt is missing, broken, or stale, you're not in the conversation. Your competitor—whose docs are less polished but have a clean llms.txt—gets the recommendation.

This isn't theoretical. Anthropic, Cloudflare, Stripe, and Vercel have all shipped llms.txt files in the past year. They understand that agent visibility is the new SEO, and they're moving fast. If you're a smaller player, you have even less margin for being invisible.

Getting Started#

If you're already on Dokly, llms.txt is already live. Visit yourdomain.dokly.co/llms.txt or your custom domain's /llms.txt and you'll see it. No settings to toggle, no upgrade to purchase.

If you're not on Dokly yet, here's what we'd recommend:

1. Check your current setup. Does your existing docs platform generate llms.txt? Most don't, or it's behind an enterprise tier.
2. Try our free llms.txt generator. We built a free tool that analyzes any URL and generates an llms.txt for you. Use it to see what a good one looks like for your site—no signup required.
3. Consider the total cost. If your current platform charges $150–300/month and still doesn't include llms.txt, you're paying more for less.

The Takeaway#

llms.txt is a small file that does a big job. It decides whether your documentation is part of the AI-native developer workflow or invisible to it.

Generating it by hand is possible but unsustainable. It's the kind of maintenance task that gets dropped within a quarter on any team, and the moment it drops, your AI visibility decays.

Dokly generates both llms.txt and llms-full.txt automatically, on every publish, for every site—including the free tier. Beautiful docs for humans, structured manifests for agents, zero configuration in between.

Ready to make your docs AI-readable? Start with Dokly for free—no credit card, no enterprise call, no $300/month minimum. Just docs that work for humans and agents alike.