llms.txt is a plain-text file, placed at the root of a website (/llms.txt), that gives large language models a short, curated summary of the site and a list of links to its most useful pages, written in Markdown. It was proposed by Jeremy Howard of Answer.AI in September 2024. It is a proposal, not a web standard, and as of November 2026 no major AI company has said its crawlers use other sites' llms.txt files to decide what to show or cite.
That last sentence is the one most explanations leave out, so most of this article is about it: what the file is for, what the companies that run search and AI systems have actually said, what server logs show, and what I found when I checked the same 34 Nepali websites I have been auditing since August.
What does an llms.txt file contain?
The specification lives at llmstxt.org. It was revised on 10 August 2026 into a "v2" that its author says reflects two years of adoption, so older tutorials describe an earlier version. Under v2:
- An H1 with the name of the site or project. This is the only required part.
- An optional blockquote with a one-paragraph summary.
- Optional free text with no headings.
- H2 sections, each a list of Markdown links in the form
[name](url): notes. - An optional section headed Optional, for links an AI agent can skip when it is short of space.
A minimal file looks like this:
# Kathmandu Trek Co.
> Licensed trekking agency in Thamel, Kathmandu. Guided treks in the
> Everest, Annapurna and Langtang regions since 2012.
## Treks
- [Everest Base Camp, 14 days](https://example.com.np/ebc.html): itinerary, permits, price
- [Annapurna Circuit](https://example.com.np/annapurna.html): route and season notes
## Optional
- [Company history](https://example.com.np/about.html)The proposal also suggests offering a clean Markdown copy of each page at the same address with .md appended, and v2 adds a way to announce both in the page's HTML (rel="alternate" type="text/markdown" and rel="describedby").
One thing the specification does not define is llms-full.txt, a single file containing a site's entire documentation. That convention came from Mintlify, a documentation-hosting company, in late 2024. It is widely copied, but it is not part of llms.txt.
Is llms.txt the same as robots.txt?
No, and confusing them is the most expensive mistake in this area.
| robots.txt | llms.txt | sitemap.xml | |
|---|---|---|---|
| Purpose | Says which crawlers may fetch which URLs | Summarises the site and points to key pages | Lists URLs for search engines to discover |
| Controls access? | Yes, for crawlers that honour it | No | No |
| Standard | RFC 9309 (2022) | Proposal only | sitemaps.org protocol |
| Documented use by OpenAI, Anthropic, Google, Perplexity | Yes, all four | No public confirmation | Yes, for search |
OpenAI's crawler documentation says opt-outs for ChatGPT search and model training are controlled through robots.txt, using separate tokens for each. The llmstxt.org page itself says robots.txt governs access and llms.txt only adds context for content that is already allowed. If you want to change whether ChatGPT, Claude or Perplexity can see your site, robots.txt is the file that matters. I explained the tokens and what each controls in my study of AI crawler rules on Nepali websites.
Does Google use llms.txt?
Google has said no, repeatedly, and in writing.
- In June 2025 John Mueller of Google wrote that, as far as he knew, no AI system currently used llms.txt (reported by Search Engine Roundtable, 17 June 2025).
- In July 2025 Gary Illyes of Google said at a Search Central event that Google did not support llms.txt and was not planning to, as reported by attendees. In August he compared it publicly to the old keywords meta tag.
- Google's own guide to generative AI features in Search now says site owners do not need new machine-readable files, AI text files or Markdown to appear in Google Search, and that such files neither help nor harm because Search ignores them (Google Search Central).
The signals are not perfectly clean. Google briefly published an llms.txt for its own developer documentation in December 2025, then removed it, and in 2026 an experimental Lighthouse audit category for "agentic browsing" began checking whether an llms.txt file is well formed. Neither is a statement that Google Search reads other people's files. The written guidance is unambiguous for Search and AI Overviews.
Do ChatGPT, Claude or Perplexity read it?
Nobody has said so. OpenAI and Anthropic both publish llms.txt files for their own developer documentation, which is useful if you are pointing a coding assistant at their docs. That is a different thing from their crawlers reading yours, and I could find no public statement from OpenAI, Anthropic or Perplexity that they use third-party llms.txt files for search, ranking or citation.
What exists instead is log evidence, all of it from vendors or individual practitioners rather than independent research:
| Who | Sample | Finding |
|---|---|---|
| Ahrefs (June 2026) | 137,210 domains using its analytics, May 2026 | 28% had a file, which Ahrefs calls an upper bound; 97% of valid files received zero requests |
| SE Ranking | About 300,000 domains | 10.13% had a file; having one showed no relationship with how often a domain was cited by AI |
| OtterlyAI (Feb 2026) | One site, 90 days | 84 of more than 62,100 AI bot visits went to /llms.txt |
| Dan Petrovic (Aug 2026) | One site, 20 days | 4 requests for llms.txt against 1,150 for robots.txt |
Each study has its own method, and they should not be compared directly. Together they say the same thing: AI crawlers request robots.txt constantly and llms.txt rarely, and nobody has shown that publishing one changes whether you get cited.
How many Nepali websites have an llms.txt?
On 5 November 2026 I requested /llms.txt and /llms-full.txt from the same 34 Nepali websites I audited in August: trekking companies, hotels, news sites, banks and payment companies, universities and SEO agencies. A file counted only if it returned 200 as plain text or Markdown rather than an HTML error page dressed up as a success.
- 7of 34 sites serve a genuine llms.txt
- 0serve llms-full.txt
- 4of the 7 belong to SEO consultants or agencies
- 1was generated by a WordPress SEO plugin, at 94 KB
The other three are a trekking company, a heritage hotel group and an online marketplace. Seven of 34 is 21%, between SE Ranking's 10% and Ahrefs' upper bound of 28%, but a sample this small and this hand-picked says more about who adopts the file than about the country.
Who adopts it is the interesting part. Four of the six SEO consultancies in the sample have one. Among the 28 sites that are not SEO businesses, three do. That repeats the pattern I found in September with AI crawler rules in robots.txt: the people who sell search services are the ones experimenting, which is reasonable, and the businesses they serve mostly have not started.
The finding I did not expect
I also re-read each site's robots.txt on the same day. Six of the 34 now name at least one AI crawler. Four sites have both files. But three of the seven sites with an llms.txt say nothing about any AI crawler in robots.txt. They have published the file no AI company has confirmed reading, and left blank the one that OpenAI, Anthropic, Google and Perplexity all document as the control.
That is not a criticism of the file. It is a question of order. A plugin that writes llms.txt in one click is easy to switch on; deciding whether OAI-SearchBot and Google-Extended may read your site takes thought. The thinking is the part that has an effect.
The aggregate results are published as JSON, with the method. Individual sites are not named.
Should you create an llms.txt?
It depends on what your site is.
If you publish documentation, an API reference or a software library: yes, probably. Developers increasingly paste documentation into coding assistants or point agents at it, and a clean index with Markdown copies of each page makes that easier. This is the use the proposal was written for, and the reason companies such as Anthropic and OpenAI publish their own.
If you run a business website in Nepal, such as a hotel, clinic, school or trekking agency: it will not hurt, it costs ten minutes, and you should not expect anything from it. Spend the time first on the things that are documented to matter:
- Decide your AI crawler policy in robots.txt, explicitly.
- Make sure the pages you want cited are indexable and return real content without JavaScript.
- Give each important page one clear heading, a direct answer near the top and facts with dates. That helps people, Google and every AI system at once.
- Add accurate structured data for what you are. This is the markup that has an effect.
If someone is selling llms.txt to you as a way to get into ChatGPT or AI Overviews, ask them for the evidence. On the record so far, there is none.
For transparency: this site does not have an llms.txt. That is not a statement against the format. Every page here is static HTML that loads without JavaScript, the sitemap is complete, and robots.txt states an explicit policy for the search and training crawlers of OpenAI, Anthropic, Perplexity and Google, so the documented paths are already covered. If an AI company documents that it reads llms.txt, I will add one and say so here.
Questions people ask
Is llms.txt a ranking factor?
Not for Google, which says Search ignores it. No other search or AI company has said it is a factor either.
Where does the file go?
At the root of the domain, so it loads at https://yourdomain/llms.txt. It must be served as plain text, not as an HTML page.
Can llms.txt block AI from using my content?
No. It contains no access rules. Blocking is done in robots.txt, with the specific crawler names each company documents, and at the server or CDN level for crawlers that ignore robots.txt.
Do I need llms-full.txt?
Only if you have documentation that someone would want to load into an AI assistant in one go. It is a documentation-industry convention, not part of the llms.txt proposal.
My SEO plugin generated one automatically. Should I delete it?
There is no reason to, as long as it lists the right pages. Open it and check: one plugin-generated file in my sample listed 387 links, which defeats the point of a curated summary.
Sources
- Jeremy Howard, The /llms.txt file, llmstxt.org, first published 3 September 2024, revised 10 August 2026.
- Google Search Central, Optimizing your website for generative AI features on Google Search, accessed 5 November 2026.
- OpenAI, Overview of OpenAI crawlers, accessed 5 November 2026.
- Search Engine Roundtable, Google: no AI system currently uses llms.txt, 17 June 2025; Illyes compares llms.txt to the keywords meta tag, August 2025.
- Ahrefs, llms.txt study, 15 June 2026 (vendor research).
- SE Ranking, llms.txt and AI citations (vendor research).
- OtterlyAI, The llms.txt experiment, updated 5 February 2026 (vendor research).
- Dan Petrovic, Crawler census, 15 August 2026.
- IETF, RFC 9309: Robots Exclusion Protocol, September 2022.
Method: on 5 November 2026, one request each for /llms.txt, /llms-full.txt and /robots.txt on 34 Nepali websites, from a server in India. A file counted as llms.txt only if it returned HTTP 200 with a text or Markdown body.
Featured photo: mhx from London, United Kingdom, Wikimedia Commons, CC BY-SA 2.0. Cropped to 16:9.
.jpg){kind=link}