What Is llms.txt, and What Does It Do?
Updated 2026-07-22
What llms.txt is
llms.txt is a proposed plain-text file, placed at a site's root, that gives an AI system a short, curated map of the site's most important pages along with brief descriptions of what each one covers. The idea, put forward as an open proposal at llmstxt.org, addresses a specific problem: language models have a limited context window, and a full sitemap or an entire crawled site is often too large or too noisy to hand to a model directly. llms.txt aims to solve that by offering a compact, editorially chosen index instead, similar in spirit to how a robots.txt file offers crawlers a compact set of instructions rather than requiring them to infer intent from the whole site.
How llms.txt differs from robots.txt
robots.txt is a set of crawl directives: it tells automated agents which paths they may or may not fetch, using a syntax built around allow and disallow rules. llms.txt serves a different purpose entirely. Rather than gating access, it curates content, offering a short list of links with plain-language summaries meant to orient a model quickly. The two files can coexist without conflict, since one manages permission and the other manages discovery and context. A site can maintain strict robots.txt rules while still publishing an llms.txt file that highlights the pages it wants an AI system to prioritize among whatever it is permitted to access.
How llms.txt differs from a sitemap
An XML sitemap is built for search engine crawlers and optimized for completeness: it lists every indexable URL on a site along with metadata like last-modified dates, with no editorial narrowing. llms.txt takes the opposite approach, favoring a short, deliberately incomplete list of the pages an editor considers most representative or most useful for an AI system to read closely. The format is markdown rather than XML, written to be readable by both a model and a human glancing at the raw file. Where a sitemap answers what exists, llms.txt tries to answer what matters most, a distinction that reflects the different constraints of a crawler indexing many pages versus a model working within a fixed context budget.
Why the convention exists
As language models took on more retrieval and browsing behavior, site owners had no standard way to hand a curated summary of their content directly to a model, the way they long have for search crawlers through robots.txt and sitemaps. llms.txt fills that specific gap. It is not a guarantee that any given assistant reads or honors the file, since support is inconsistent and the convention itself is still young and unofficially standardized. Its value is closer to a hygiene practice: a clearly maintained llms.txt costs little to produce and gives any AI system that does check for it a faster, cleaner path to a site's most useful pages than crawling and inferring structure from scratch.
What to include in an llms.txt file
A well-formed llms.txt typically opens with a short line identifying the site and its purpose, followed by grouped links to the pages most useful for an AI system to reference, each with a one-line description of what the page covers. Effective entries favor the same self-contained, clearly answered content that performs well in AI answers generally, since a link is only as useful to a model as the page behind it. Pages already structured around a single grounding query, in the sense described on the grounding query page, tend to make natural llms.txt entries, since their titles and summaries map cleanly onto a one-line description.
Frequently asked questions
Does llms.txt guarantee an AI engine will read it?
No. It is a voluntary convention, not a protocol any assistant is required to support. Adoption varies by engine and continues to evolve.
Is llms.txt a replacement for a sitemap?
No. A sitemap lists URLs for crawling and indexing purposes, while llms.txt is meant as a curated, human-readable summary aimed at a language model's context window rather than a crawler's queue.
Where should llms.txt live on a site?
At the domain root, following the naming convention used by robots.txt, so an automated system checking for it in the expected location can find it without extra configuration.