Technical AI search · Implementation guide

llms.txt for AI search: useful map, not magic switch.

It can give AI applications a concise guide to your best public resources. It does not grant access, force crawling, guarantee a citation, or replace content and technical SEO.

Plastorium Research · August 23, 2026 · 10 min read

Short answer

llms.txt is a proposed Markdown convention for a curated, machine-readable guide to a website. It is optional. Add it only if you have clear public resources to curate and can keep the file accurate.

Its value is plausible as a low-cost orientation layer for tools or agents that choose to use it. Its value is not proven as a universal AI-search ranking signal. Build it after—not instead of—indexable pages, factual accuracy, crawl access, canonical URLs, a sitemap, useful structured data, and real third-party evidence.

What llms.txt is designed to be

The proposed format places a Markdown file at /llms.txt. It starts with the name of the site, gives brief context, then organizes links to useful resources with plain-language descriptions. Think of it as a hand-curated reading list for a language-model application with limited context—not as an instruction file.

Good use“Here is what this organization does, and these are its authoritative public pages, docs, policies, and guides.”
Good useGive an agent a smaller, intelligible route through a large documentation or content library.
Not its jobTell a crawler where it may fetch, train on, index, or avoid content.
Not its jobConvince an answer engine to recommend, quote, or cite a weak page.

The file is especially sensible for public documentation, product references, research libraries, and sites whose navigation is hard to interpret quickly. For a small brochure site with five clear pages, value may be marginal.

What llms.txt does not do

Overclaiming is the main risk. A file can only help when a particular system decides to fetch and use it; there is no universal adoption requirement. Treat these claims as false:

  • “It guarantees a ChatGPT, Gemini, Perplexity, or Google AI Overview citation.”
  • “It improves Google rankings or makes a page eligible for AI Overviews.”
  • “It replaces robots.txt, a sitemap, canonical tags, or schema.”
  • “It prevents AI training or blocks a bot.”
  • “A list of keywords or all URLs will make a site easier to recommend.”
Priority rule: if a page is thin, inaccurate, blocked, non-canonical, private, or unsupported by real evidence, fix that condition first. A better index of a weak site is still a weak source.

llms.txt vs. robots.txt, sitemap, and structured data

File / markupPrimary purposeWhat it does not do
robots.txtExpress crawl-path preferences to compliant bots.It does not curate meaning, guarantee indexing, or protect a public URL from access.
sitemap.xmlHelp search engines discover canonical, indexable URLs.It does not establish content quality, authority, or answer relevance.
Structured dataDescribe eligible entities and attributes in a machine-readable vocabulary.It does not create facts, force rich results, or function as “AI schema.”
llms.txtCurate and explain priority public resources for systems that choose to use it.It does not control access, indexing, training, rankings, or citations.

These layers can coexist. They solve different problems. In particular, do not place a pretend sitemap directive in robots.txt for an llms.txt file: it is not an XML sitemap, and that does not create platform support.

Should your site add llms.txt?

Use this decision test before assigning work.

Reasonable nowYou have stable public pages with clear ownership: docs, policies, research, products, tutorials, FAQs, or authoritative case studies.
Reasonable nowYou can select a small set of canonical URLs, write accurate descriptions, and review the file whenever those pages change.
Defer itYour sitemap, canonicalization, rendering, access, or core content quality is broken—or the site lacks real public resources worth curating.
Defer itThe only business case is “rank in AI answers quickly.” No evidence supports treating it as a shortcut.

For Plastorium, the file is a compact guide to the public methodology, service pages, reports, and articles. It points a reader toward information that already exists; it does not claim that the file itself drives recommendations.

A safe, useful llms.txt format

Keep it short enough to remain curated. Start with the project name, a factual summary, and grouped links. Use absolute canonical URLs. Link only to public pages you would be comfortable having an AI system read and summarize.

# Example Company

> Example Company provides [plain-language description]. This file lists its most useful public resources.

## Start here
- [Product overview](https://example.com/product): What the product does and who it is for.
- [Documentation](https://example.com/docs): Setup, reference, and limitations.

## Trust and policy
- [Security](https://example.com/security): Current public security information.
- [Privacy](https://example.com/privacy): Data handling and privacy requests.

## Optional
- [Archive](https://example.com/archive): Older reference material.

Include

  • Canonical public URLs and concise, verifiable descriptions.
  • Pages with a clear reader purpose: product, documentation, support, methodology, policy, or current research.
  • A small optional section for lower-priority resources, when useful.

Exclude

  • Private reports, user URLs, admin pages, login screens, API endpoints, staging domains, tracking parameters, and expired campaigns.
  • Keyword lists, sales claims that the linked page cannot support, or instructions trying to manipulate a model.
  • Every page in a site just because it exists. That is what an XML sitemap is for.

Implementation and maintenance checklist

  1. Inventory source pages. Pick authoritative, public, canonical pages. Confirm HTTP success, crawl access where intended, and matching page titles/descriptions.
  2. Write factual annotations. Each link should answer “why should a reader open this?” without promising outcome or ranking impact.
  3. Publish at the intended path. Use /llms.txt as plain text/Markdown and confirm it loads without authentication or a redirect chain.
  4. Keep existing technical basics healthy. Preserve your XML sitemap, robots rules, canonicals, page rendering, and appropriate structured data.
  5. Version the file with material site changes. Add new canonical resources; remove stale, redirected, private, or superseded URLs.
  6. Measure outcomes honestly. Track visibility separately through a stable prompt panel and cited-source evidence. Do not attribute a change to the file from one answer.

For broader site readiness, see A Human Sees Your Website. AI Sees Missing Data. and AI Visibility Audit Checklist.

Make the site understandable to AI—without mistaking markup for proof

The goal is to make your business, offers, evidence, and priority pages clear enough for AI systems to identify and verify them. Plastorium checks the evidence gaps behind misses; it does not promise a markup file will produce a recommendation.

Check my AI visibility

Frequently asked questions

What is llms.txt?

It is a proposed Markdown file that gives language-model applications a concise guide to a site's important public resources. It is optional and system adoption is not universal.

Does llms.txt improve Google AI Overview visibility?

Do not use it as a Google ranking or AI Overview tactic. It is not a replacement for useful indexable content, crawlability, site quality, or evidence. A system must choose to use the file.

Can llms.txt block an AI crawler?

No. It is not a crawl control. Use robots rules to communicate crawl preferences to compliant bots, while understanding that public content may still be accessed or reproduced through other means.

Do I need llms-full.txt too?

Only if you have a well-maintained public corpus for which a full, consolidated text resource is useful. It has the same adoption uncertainty and must never expose private, outdated, or licensed material.

Related reading