
More and more of your customers are asking an AI instead of Googling. In Sweden, the Internet Foundation’s Svenskarna och internet 2025 report (in Swedish) found that four in ten Swedes use AI tools, and ChatGPT alone is used by more than one in three. More than one in five now put questions to an AI that they would otherwise have Googled.
Here’s what that means in practice for you as a business owner: when someone asks ChatGPT about a supplier in your field, does your company make it into the answer? Or is your website invisible to the very tools your customers are starting to trust?
The answer often comes down to one small text file you may never have opened. It’s called robots.txt, and it decides what different bots are and aren’t allowed to fetch from your site. The problem is that “blocking AI” isn’t a single button. It’s many. And press the wrong one, and you can vanish from AI answers without even knowing it.
What robots.txt actually is
Robots.txt is an ordinary text file that sits at the top of your website, at the address example.com/robots.txt. Anyone can read it, including your competitors. It works like a sign at the entrance where you write instructions for the bots that visit: you may look in this folder, stay out of that one.
The important thing to understand is that the file is a polite request, not a lock. The standard behind it (RFC 9309, from 2022) states outright that the rules are not a form of access control. Well-behaved bots follow them. Less scrupulous ones don’t always. We’ll come back to why that matters.
No robots.txt at all? Then everything is allowed. That’s a perfectly normal and healthy state, nothing wrong in itself. It simply means no bot has been asked to stay away from anything.
Why “AI bot” isn’t one thing
Here’s the part most articles skip, and it’s the whole point.
Every big AI company runs not one bot, but several. And they do different jobs. Broadly, there are three kinds:
A bot that trains the model. It gathers content that can be used to teach the AI itself. Blocking it doesn’t affect whether you show up in answers, only whether your material gets to be part of the training.
A bot for visibility in AI search. It decides whether your business can appear when someone asks a question and gets an answer. This is the one you usually want to let in.
A bot that fetches a page when a human asks it to. When a user pastes in your link and says “summarise this,” the AI goes and fetches the page on the spot.
Take OpenAI as an example and it becomes clear. GPTBot is the training bot. OAI-SearchBot is the one that governs whether you appear in ChatGPT’s search answers. ChatGPT-User is the one that fetches a page at the user’s request. OpenAI’s documentation is explicit that these settings are independent of one another. You can let OAI-SearchBot in so you show up in answers, while shutting GPTBot out from training on your content.
According to its help page on the bots, Anthropic (Claude) follows the same pattern, with ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user-driven visits. Perplexity, according to its developer documentation, has PerplexityBot for its search index and Perplexity-User for what a user asks for.
See the trap? If you read “block AI” and shut out everything with GPT or Claude in the name, you can accidentally switch off the very bot that would have got you into the answers. You save your content from training, but the price is going invisible to the customer who’s asking.
The three traps that make people get it wrong
Three special cases cause the most trouble, because the names trick you into thinking they do something they don’t.
Google-Extended is not a search block. According to Google’s documentation, it affects neither whether your site is included in Google Search nor how it ranks. That includes AI Overviews, the AI answers at the top of Google, because they are part of Google Search and built from the regular index via Googlebot. But Google-Extended governs more than training. It decides both whether your content may be used to train Gemini models and whether the Gemini app may draw on it when answering. Blocking it therefore leaves your visibility in Google Search alone, but not in Gemini.
Applebot-Extended doesn’t crawl at all. It fetches no pages, according to Apple’s description of Applebot. It only governs whether content Apple has already collected may be used to train their models. Blocking it doesn’t make you less visible at Apple. A page that shuts out Applebot-Extended can still appear in Apple’s search results. If you also want to stop your content being used as source material in an AI-generated Apple answer, that’s the nosnippet tag’s job, not Applebot-Extended. So they’re two separate choices, and the block itself is only about training.
The user-driven bots often don’t care about robots.txt. This is the trickiest one. When a human themselves asks the AI to visit a page, several operators treat it as though the person had opened a browser. Perplexity states outright that their Perplexity-User generally ignores robots.txt. OpenAI says the rules “may not apply” to ChatGPT-User. That means in practice you can’t rely on robots.txt to shut them out. Anthropic is the exception and says all its bots respect the file, the user-driven one included.
The file is a request, not a lock
It’s worth pausing on one thing so you’re not lulled into false security. Robots.txt stops no one technically. It only asks nicely.
Most serious AI bots comply. OpenAI, Google, Anthropic, Apple and Common Crawl all say they respect the file. But compliance is neither total nor flawless. In August 2025, Cloudflare published a widely noticed report claiming Perplexity used hidden bots to get around both robots.txt and network blocks. Perplexity disputed the conclusions. The dispute is still unresolved, but it shows something important: if you really want to shut out a bot that won’t cooperate, robots.txt isn’t enough. That takes a real technical block, like a firewall or server-side bot protection.
Another thing that surprises many people: blocking a bot in robots.txt does not remove a page from search results. If you want a page genuinely not to show, you need a noindex tag or password protection, not a line in robots.txt. And a curious consequence of that: the bot has to be allowed to read the page in order to see your noindex tag at all. Shut it out in robots.txt and it can never read the instruction to stay away.
What you should actually do
Start by separating three questions that are easily bundled together. Do you want to show up in AI answers like ChatGPT, Perplexity and Claude? For the vast majority of small businesses the answer is yes. It’s free marketing to an audience that’s growing fast. If you don’t know how you show up there today, you can test it in ten minutes. Do you mind your content training AI models? And are there private areas, like customer portals or internal documents, that should never be crawled at all?
For most, the sensible setup is to be visible but skip the training: let the search bots in (OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot) and shut out the training bots if you want to (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot). One exception to weigh: because Google-Extended also decides whether the Gemini app may draw on your content, leave it open if you want to show up in Gemini.
Here’s what that might look like in practice:
# Be visible in AI search
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Shut out model training where possible
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /This example is not a one-size-fits-all solution. It only shows the principle: let in the bots that help your visibility and shut out the ones that are mostly about training, if that’s what you want. Always check your actual file, since other rules, firewalls and plugin settings can change the result.
One rule to remember when you name bots: a line for User-agent: * normally applies to every bot that doesn’t have its own, more specific group in the file. But if you create a separate group for, say, GPTBot or ClaudeBot, it’s that specific group that governs for that particular bot. So you need to be explicit about every bot you actually want to let in or shut out, rather than trusting a general rule to catch them.
And remember that robots.txt is never security. Private pages belong behind a login, not behind a polite request.
Is your website invisible to AI right now? Test it in seconds
A common fault isn’t a deliberate decision. It’s an old line in robots.txt that a plugin added, or that was left over from when the site was built, quietly shutting out the very bots you actually want in.
You can check it in seconds. Our free tool, the AI Crawler Check, fetches your robots.txt, reads what it tells the major AI bots, and shows in plain language whether you’re blocking any of them, which kind, and what that actually means. No hassle, no login.
Test shows everything is open but you still aren’t appearing in AI answers? Then the problem sits deeper, often in a bot protection or server setting that robots.txt doesn’t reveal. That’s exactly the kind of deeper review our GEO and AI visibility audit is for.
Your robots.txt is worth a look
This little file decides more than most people think. It stands between your business and a growing group of customers who would rather ask an AI than scroll through a list of results. And a line that shuts out the wrong bot is also one of the easiest problems to fix, once you see it.
Run the AI Crawler Check on your own address. If it shows a block you didn’t know about, it’s usually a single line to change. If it shows everything is open but you’re still missing from the answers, send us the result and we’ll dig further into bot protection and server settings.
Bot names and rules checked against OpenAI’s, Anthropic’s, Google’s, Apple’s and Perplexity’s documentation in October 2026. They change often, so check your file regularly.