As AI assistants read more of the web, website owners want a way to guide them. llms.txt is one proposal: a simple file that summarises your site for AI tools. It is quick to create, but expectations should be realistic.
Where llms.txt came from
The format was proposed in 2024 by Jeremy Howard of Answer.AI and is documented at llmstxt.org. The idea is that AI tools struggle with complex HTML pages full of navigation, scripts and ads, so a concise Markdown file pointing to clean, important content could help them understand a site quickly.
What an llms.txt file contains
The proposal suggests a Markdown file with:
- An H1 with the site or company name.
- A short summary in a blockquote.
- Optional paragraphs with more context.
- H2 sections listing important links, each with a short description.
- An optional section for secondary links.
llms.txt example
# Example Agency
> Example Agency is a B2B marketing agency offering SEO, paid advertising and website development for SaaS companies in the United States.
## Services
- SEO services (example.com/services/seo): technical SEO, content and link building for SaaS.
- Paid advertising (example.com/services/ppc): Google and LinkedIn campaigns for demo generation.
## Guides
- SaaS SEO guide (example.com/blog/saas-seo): how software companies rank and convert organic traffic.
In the real file, each item uses standard Markdown link syntax: the page name in square brackets immediately followed by the full URL in parentheses. Keep descriptions factual and short, and link only to pages you want AI tools to rely on.
llms.txt vs robots.txt for AI bots
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Tells crawlers which URLs they may or may not crawl | Summarises and points to important content |
| Status | Long-established, widely supported standard | Proposal, optional, inconsistent support |
| Controls access? | Yes, for compliant crawlers | No |
| Location | /robots.txt | /llms.txt |
If you want to control AI crawler access, use robots.txt rules for specific user agents such as OAI-SearchBot, GPTBot, PerplexityBot or Google-Extended. llms.txt cannot block or allow anything. Google's robots.txt introduction explains how crawl rules work.
Does llms.txt help SEO?
There is no evidence that llms.txt improves rankings in Google or other search engines. Google has indicated it does not use the file for Search, and major AI companies have not committed to treating it as a ranking or citation signal. Some AI tools and developer platforms do read it, particularly for documentation.
So why add one? It is low effort, can help tools that do read it, and the process of writing it forces you to identify your most important pages and describe them clearly. That clarity is valuable for your generative engine optimization work regardless.
How to create llms.txt
- List the pages that best explain your business: homepage, main services, about, pricing or engagement models, key guides.
- Write a two-sentence summary of what your company does, for whom and where.
- Group links under clear headings with one-line descriptions.
- Save as plain text Markdown at the root of your domain as /llms.txt.
- Serve it with a text content type and make sure it is publicly accessible.
- Update it when you add or retire important pages.
Some CMS platforms can generate llms.txt automatically from published pages. If yours does, review the output for accuracy and remove pages you would not want summarised.
Is there an llms-full.txt?
The proposal also mentions optional expanded versions that include full content in Markdown, mainly useful for documentation sites. Most marketing websites do not need one.
AI crawler access: what actually matters
For visibility in AI search, the more important decisions are which crawlers you allow in robots.txt and whether security tools block them. Review:
- Search and answer crawlers such as Googlebot, Bingbot, OAI-SearchBot and PerplexityBot.
- Training crawlers such as GPTBot, Google-Extended and CCBot, based on your policy about model training.
- Firewall and bot-protection settings that may block legitimate crawlers.
Our guides to ChatGPT search visibility and Perplexity SEO explain these crawlers in detail.
A sensible llms.txt policy for businesses
If you decide to publish llms.txt, treat it like any other public summary of your company:
- Assign an owner. Usually the person responsible for SEO or the website.
- Review quarterly. Add new services and important guides, and remove retired pages.
- Match your other sources. The description should match your About page, structured data and LinkedIn profile.
- Avoid sensitive links. Do not list admin areas, private documents or internal tools.
- Measure nothing directly. Because tools rarely report reading the file, judge your AI visibility work through referrals, mentions and prompt testing instead.
This keeps the file accurate without turning it into a project that distracts from higher-impact work.
Should your business add llms.txt?
If you have a clear set of important pages and can maintain the file, adding llms.txt is reasonable and harmless. Just do not expect it to change rankings or guarantee AI citations. Prioritise the fundamentals first: accessible pages, answer-first content and a consistent brand presence.
Need help deciding which AI crawlers to allow and how to present your site to AI tools? Our AI search visibility service covers crawler policy, content and measurement together.
