What is llms.txt?

The question

What is llms.txt?

The short answer

llms.txt is a proposed Markdown file at the root of a website, usually /llms.txt, that gives AI agents a short summary of the site and a curated list of links to its most useful pages. It is a community proposal, not a web standard. Google says Search ignores it, so it won't help or hurt your Google rankings, but some agents and tools do read it.

Source: The /llms.txt file, v2, Jeremy Howard, Optimizing your website for generative AI features on Google Search, Google Search Central, Overview of OpenAI Crawlers, OpenAI, Introducing Search Generative AI performance reports in Search Console, Google Search Central blog

A vendor tells a business that AI can't "see" its website without an llms.txt file. The fix is a premium package.

The file is a short Markdown list. The package is the expensive part.

What is llms.txt?

llms.txt is a plain-text file written in Markdown that sits at a known address on your site and tells AI agents what the site is and where the most useful pages are.

The idea comes from a proposal by Jeremy Howard, first published in September 2024 and updated to a v2 in August 2026. It describes the file as a way "to provide information to help agents use a website": brief background, guidance, and links to detailed pages.

The reasoning is practical. Web pages are built for people, wrapped in navigation and scripts. An agent with limited room for text does better with a short, clean summary in one place.

What does an llms.txt file look like?

The proposal sets a simple order. Only the first item is required:

1. An H1 with the site or project name.
2. A blockquote with a short summary.
3. Optional paragraphs or lists with more detail (no headings).
4. Optional H2 sections, each a list of links in the form [name](url): notes.

A section called "Optional" is, by convention, for links an agent can skip.

Here is the top of our own file, trimmed:

# RafalAI

> RafalAI is a public educational reference for practical AI at work: workflows, source material, agents, MCP tools, machine-readable sites, search visibility, templates, evidence, review, and useful artifacts.

## Core Pages

- [Workflows](https://rafalai.com/workflows/): How to write one repeating job: the material, the tools, the check, and the artifact.
- [AI visibility](https://rafalai.com/ai-visibility/): How to record where AI answers mention you, cite you, or skip you.

That is the whole format. If you can write a bulleted list, you can write one.

Does Google use llms.txt?

No. Google's guide to generative AI features in Search lists llms.txt under things you can ignore. It says you don't need "new machine readable files, AI text files, markup, or Markdown" to appear in Google Search, "as Google Search itself doesn't use them."

It also says it is "completely fine" to keep one for other services, and that doing so "will neither harm nor help" your visibility in Google Search.

So for AI Overviews and AI Mode, the file does nothing. If you want to know how you show up there, Search Console's generative AI performance report shows impressions in those features by page.

Who does read llms.txt?

Agents, mostly at the moment a user asks them something. The proposal says it is used "on demand, when an agent needs information about a topic while assisting a user," more than for training. It notes that OpenAI, Anthropic and Google's Gemini publish llms.txt files for their own developer docs, and that Chrome's Lighthouse checks for one in its agentic browsing audits.

That is not the same as a search engine promising to use it for ranking. OpenAI's crawler documentation, for example, tells sites to allow OAI-SearchBot in robots.txt to appear in ChatGPT search. It doesn't mention llms.txt as a requirement.

Our read: useful for agents and coding tools, unproven for AI search citations.

How is llms.txt different from robots.txt and sitemap.xml?

The proposal itself draws the line: robots.txt is about access, and a sitemap lists everything, often more than an agent can read at once. llms.txt is a short, curated tour. It does not replace either of the other two.

How do you create and check an llms.txt file?

1. Write the H1 and one-sentence summary. Say what the site is in words a stranger understands.
2. List your 5 to 15 most useful pages, each with one line on what it answers.
3. Use full URLs and check that every link returns a page, not a redirect or a 404.
4. Put it at /llms.txt and make sure it is served as plain text.
5. Check robots.txt agrees with it. A file pointing agents to pages that robots.txt blocks sends mixed signals.
6. Test it. The proposal's own advice: ask an agent questions about your site, giving it only your llms.txt as a starting point.
7. Update it when you publish. A stale list is worse than none, because it sends agents to old pages.

There is no official validator. Checking the order above, the link format and the status of every URL covers what a validator would check.

FAQ

Is llms.txt an official web standard?

No. It is a community proposal hosted at llmstxt.org and open for input. No standards body has adopted it.

Will llms.txt help Google rankings?

No. Google says Search ignores it, and that it won't help or hurt your visibility there.

Does ChatGPT use llms.txt?

OpenAI's crawler docs say to allow OAI-SearchBot in robots.txt to appear in ChatGPT search, and don't list llms.txt as a requirement. Agents may still read it when a user asks about a site.

What is llms-full.txt?

A common companion file with the full text of key pages in one place. It's a convention some sites follow, not part of the core proposal's required format.

Is there an llms.txt validator?

Not an official one. Check the H1, the summary, the link-list format and that every URL loads.

Should a small business bother?

If you keep it current, sure. It is a short list. Just don't pay a premium for it, and don't expect it to change Google.