There is a line in your robots.txt file that now decides whether ChatGPT, Claude, Perplexity and Google's AI Overviews are allowed to read your website. On most Nepali business websites, that line does not exist — which is a decision too, just not one anybody made on purpose.
I wanted to know how common that was, so on 4 September 2026 I fetched robots.txt from the same 34 Nepali websites I audited earlier this year and searched each one for the named AI crawlers. Here is what came back.
The numbers
- 34 domains checked
- 31 served a usable robots.txt
- 3 did not — one returned 403 to an ordinary request, one timed out, one served something with no directives in it
- 7 named any AI crawler at all
- 27 named none
Twenty-seven of thirty-four have no position on AI crawlers whatsoever. Not permission, not refusal — no line in the file.
The crawlers named by the seven that did have a policy were GPTBot and Google-Extended (7 each), ChatGPT-User, ClaudeBot and PerplexityBot (6 each), then OAI-SearchBot, CCBot and Applebot-Extended (4 each), Bytespider (3) and anthropic-ai (2).
The detail I did not expect
Of the seven sites with a policy, three belong to SEO practitioners. Their own personal sites.
So of the actual businesses in the sample — the trekking companies, the hotels, the clinics, the news sites — four had made a decision. The people who have thought about AI crawler access in Nepal are, overwhelmingly, the people who sell SEO. That is not a criticism of them. It is a reasonable description of how new this is.
Most of the policies say yes. One says no, and probably did not mean to.
Among those that had rules, most were explicit permissions. A trekking company:
User-Agent: GPTBot
Allow: /A dental clinic had gone further and left a comment in the file explaining that Google-Extended is opt-out only and that disallowing it would remove the practice from AI results. The first half of that is right: a position has to be stated to count. The second half is a common misunderstanding — Google documents that Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search. Even the sites that had thought about this were working from the wrong model.
And then one site, a dental college, had this:
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /That blocks OpenAI's training crawler and Google's token for using content to train Gemini models. It is narrower than it looks: on Google's documentation it does not remove the college from Google Search or from the AI Overviews that appear inside it, and ChatGPT's search results are governed by a different crawler, OAI-SearchBot, which this file does not mention. I cannot know whether the block was deliberate. But blanket AI blocks are a common default in security plugins and copied robots.txt templates, and they are rarely checked against what each token actually controls.
That is what makes this different from ordinary SEO mistakes. A missing H1 is visible in the page source. This is invisible unless you go and look.
What these crawlers actually control
They are not interchangeable and blocking one does not do what people assume.
GPTBot is OpenAI's training crawler. Blocking it means your content is not used to train future models. It does not remove you from ChatGPT's live search results.
OAI-SearchBot and ChatGPT-User are the ones that matter for being cited in ChatGPT answers now. Blocking these is what removes you from the answer.
Google-Extended controls whether content Google crawls may be used to train future Gemini models. Google states that it does not affect a site's inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode are Search features, governed by Googlebot and by the normal snippet controls such as nosnippet — not by Google-Extended.
ClaudeBot and PerplexityBot are their respective assistants' crawlers.
CCBot is Common Crawl, which is an open dataset many other systems are built from. Blocking it has the widest and least predictable downstream effect.
The practical point: "block AI" is not one switch. Blocking training while allowing search is a coherent position. Blocking everything because a plugin suggested it is not a position, it is an accident.
What I would actually do
For most Nepali businesses — trekking, hospitality, clinics, services — I would allow the search-facing crawlers. If a visitor asks an assistant for trekking companies in Kathmandu, you want to be an option. Being absent from that answer costs you something real and gains you nothing.
The case for blocking is narrower and genuinely exists: original research you sell, subscription journalism, photography, licensed datasets. If your content is the product, opting out of training while allowing search is defensible.
Either way, make it explicit. A file that says nothing leaves the decision to whatever the default happens to be, and defaults change without telling you.
Check yours in thirty seconds
Type your domain followed by /robots.txt into a browser. Search the page for GPTBot.
If you find nothing, you have no policy. If you find Disallow: / under an AI crawler, confirm somebody chose that.
While you are there, check the file is not doing something worse. A robots.txt blocking your whole site to everything is the fastest way to disappear from search entirely, and it is one of the recurring faults in the technical problems I keep finding.
On llms.txt
There is a proposed convention called llms.txt — a plain-text summary of your site for language models. A lot of writing about AI visibility now treats it as a requirement.
I would be honest about its status: it is a proposal, not a standard, and none of the major assistants have committed to reading it. It is cheap to add and it might matter later. It is not a substitute for being crawlable, having clear structure, and saying things worth quoting, which is what actually gets a page cited today. If someone tells you an llms.txt file will get you into AI answers, treat that the way you would treat any other shortcut.
Why this matters more here than elsewhere
Nepal has relatively few authoritative sources per topic in English, and fewer still in Nepali. When an assistant answers a question about trekking permits or hospital departments in Kathmandu, it is drawing from a shallow pool. A site that is present in that pool has a disproportionate chance of being the source. A site that has quietly opted out has none.
That advantage will not last. It exists now because most sites have not made the decision — which is exactly what the twenty-seven tell us. I have written separately about what Nepali businesses should optimise for between Google and AI search, and about being machine-readable as a person.
The method here is deliberately reproducible: fetch robots.txt from a list of domains, search for the crawler names above, count. If you run it on your own sector I would be glad to hear what you find. The underlying sample and the technical measurements are published in full.
Correction, 15 September 2026: an earlier version of this article said Google-Extended controls whether a page can appear in AI Overviews. It does not; Google documents that Google-Extended does not affect Google Search, and AI Overviews are part of Search. The measurements are unchanged.