llms.txt

What is llms.txt?

llms.txt is a static file placed at the root of a website that provides AI models with a structured, human-readable description of the site and its most important content. Think of it as a hybrid of three existing conventions: robots.txt (a file at the site root, read by automated systems), sitemap.xml (a structured list of URLs), and a website's "about" or documentation index page (a human-readable summary).

The file was introduced by Jeremy Howard, co-founder of Answer.AI and fast.ai, in a blog post and accompanying specification at llmstxt.org in September 2024. The motivating problem was straightforward: LLM context windows are limited, and when AI systems scrape websites to gather information, they often ingest noise - navigation menus, footer links, analytics tags, unrelated pages. A curated, Markdown-formatted list of the site's most important URLs with short descriptions would give AI systems a clean summary to work from.

The spec is open and vendor-neutral. No single company controls it, and the convention has been adopted voluntarily by documentation sites, SaaS products, and content-heavy publishers throughout 2024 and 2025. Adoption remains uneven: LLMrefs data suggests only about 10% of websites have published an llms.txt file.

The llms.txt format

The file is Markdown. The spec defines a minimal structure that is intentionally simple - easy for both humans to write and LLMs to parse.

• H1 - the name of the site or project. Required. First line of the file.
• Blockquote - a short summary of what the site is about. Optional. Sits directly below the H1.
• Intro paragraphs - zero or more paragraphs providing context, caveats, or pointers. Optional.
• H2 sections - groups of related URLs. Each H2 is a category name (e.g., "Docs", "Blog posts", "Tools"). Below each H2 is a bullet list of linked URLs, each with a short description.
• Optional H2 - a specially-named section flagging URLs that are lower-priority and can be skipped if the model has limited context.

A minimal valid file looks roughly like this structure: site name, one-line summary in a blockquote, then 2-5 H2 sections grouping the most important URLs. The full spec, including edge cases and examples, lives at llmstxt.org.

Why llms.txt matters

Adoption is still early enough that llms.txt is a differentiator. Several factors make publishing one worthwhile even while formal AI-platform endorsement remains ambiguous.

It is a crawlable signal

Any AI system that can fetch arbitrary URLs can retrieve /llms.txt without knowing in advance what the file contains. As AI crawlers and retrieval systems evolve, a file that costs nothing to publish may end up being parsed by tools that don't yet exist. The downside risk is zero; the upside is optionality.

It forces editorial clarity

Writing an llms.txt file requires deciding what the site is about in one or two sentences and which URLs matter most. That exercise surfaces internal disagreements about positioning and content hierarchy - making the file a useful artifact for marketing and product teams even before AI systems read it.

It overlaps with emerging best practices

LLMrefs reports that content formatted for LLM extraction is 3x more likely to be cited than standard web copy. llms.txt is one expression of that formatting discipline: structured, cleanly described, explicitly prioritized content.

How to create an llms.txt file

There are three paths.

Use a generator

AI-Advisors provides a free llms.txt Generator that crawls a site and produces a compliant file in under two minutes. The tool handles the Markdown structure, pulls page titles and descriptions automatically, and groups URLs into reasonable sections. Download the result and upload to the site root.

Write it manually

For small sites or sites with a clear content taxonomy, hand-writing the file is fine. Start with the H1 (site name), add a blockquote summary, then group URLs under H2 sections. Keep descriptions short - one line each. Reference examples at llmstxt.org.

Generate it from existing metadata

Static site generators and headless CMSs can often render an llms.txt file from existing content metadata - page titles, descriptions, and category data. For Next.js sites, a dynamic route or build step that reads from the same metadata used in sitemap generation is the cleanest approach.

Once the file is in place, publish it at the exact path /llms.txt at the root of the domain. AI systems expect that location - putting it anywhere else defeats the convention.

llms.txt vs robots.txt vs sitemap.xml

All three files live at the root of a site. They serve different purposes and should be published together.

A well-prepared AI-visibility stack publishes all three at the site root and keeps them aligned: robots.txt permits the crawlers you want, sitemap.xml lists every URL you want indexed, and llms.txt highlights the few URLs you want prioritized.

Common misconceptions

llms.txt replaces sitemap.xml

It doesn't. sitemap.xml is machine-optimized URL inventory for search engines. llms.txt is a curated human-readable summary for LLMs. They serve different systems and solve different problems. Publish both.

llms.txt is officially endorsed by major AI platforms

As of early 2026, major AI companies have not formally confirmed that llms.txt is parsed as part of their retrieval pipelines. Adoption is community-driven. That does not mean it is useless - it is discoverable by any crawler and is increasingly read by AEO and AI-visibility tooling. But claims that ChatGPT or Perplexity "definitely reads llms.txt" should be treated cautiously until the platforms say so publicly.

More is better

The whole point of llms.txt is curation. A file that lists every URL on a site is just an inferior sitemap. The file should highlight the 5 to 30 URLs that best explain what the site is and what it is authoritative about - not every blog post or landing page.

Frequently asked questions