AI Crawler Check
Is your robots.txt blocking the AI crawlers?
Paste a URL and we'll read its robots.txt and show whether the major AI crawlers are allowed to read your site — or being asked to stay away. No sign-up, nothing to install.
Reading the page's robots.txt…
That doesn't look like a checkable web address. Check it and try again.
That address can't be checked.
We couldn't find that address. Check the spelling and try again.
That address redirects somewhere that can't be checked.
You've reached today's limit for this check. Try again tomorrow — or get in touch for a full audit.
The tool is busy right now. Please try again in a minute.
The spam check didn't pass. Please reload and try again.
Something went wrong. Please try again in a moment.
No AI crawlers are blocked You have no robots.txt, which is perfectly fine — nothing stops AI from reading your site.
No AI crawlers are blocked Your robots.txt is in place and isn't shutting out the AI crawlers. This layer looks good.
An AI search crawler is blocked Your robots.txt asks one or more AI search crawlers to stay away. That can stop you appearing in AI answers.
You've opted out of some AI access This only affects AI training or user-triggered fetchers — not your AI-search visibility. It may well be intentional.
We couldn't read your robots.txt Something may be restricting automated access (a firewall or bot protection), or the site was temporarily unavailable. Worth a closer look.
- robots.txt Found and read None (no rules) Couldn't be read (bot protection) Couldn't be fetched
- Left today 0 checks
AI search visibility
Decides whether you can appear in answers from AI tools like ChatGPT and Perplexity. Blocking these can remove you from AI answers — this is what matters most.
- OAI-SearchBot OpenAI
- PerplexityBot Perplexity
- ClaudeBot Anthropic
AI training
Only controls whether your content may be used to train AI models. Opting out is a perfectly legitimate choice and does not affect whether you appear in AI search.
- GPTBot OpenAI
- Google-Extended Google
- Applebot-Extended Apple
- CCBot Common Crawl
- Bytespider ByteDance
User-triggered
Fetches your page when a person asks an AI to visit it. These may ignore robots.txt, so asking them to stay away is a request, not a guaranteed block.
- ChatGPT-User OpenAI
- Perplexity-User Perplexity
- Claude-Web Anthropic
This layer is clear. If you're still not showing up in AI answers, the cause is usually deeper — and that's exactly what our AI visibility audit looks at.
Explore AI visibilityIs this what you expected? If not, it's fixable. Want us to take a look and sort it out?
Look at our AI visibility Get in touchThis checks robots.txt — one important layer, but not the whole picture. We don't check whether you're actually cited in ChatGPT or Perplexity, nor meta-robots or server rules. That's part of a full AI visibility audit.
How it works
Three steps, a few seconds
Paste a URL
Enter your web address. No sign-up, no email, nothing to install.
We read your robots.txt
We fetch the robots.txt file at the address's root and read its rules exactly as an AI crawler does.
Get a clear verdict
We show each AI crawler grouped by what it controls — search visibility, training or user-triggered visits — and what blocking it actually costs.
What we check
Not all "AI crawlers" control the same thing
That's the whole point of this check. Blocking a training crawler does not make you invisible in AI search — we keep the three groups apart so you can see what actually matters.
AI search visibility
Decides whether you can appear in answers from AI tools like ChatGPT and Perplexity. Blocking these can remove you from AI answers — this is what matters most.
- OAI-SearchBot · OpenAI
- PerplexityBot · Perplexity
- ClaudeBot · Anthropic
AI training
Only controls whether your content may be used to train AI models. Opting out is a perfectly legitimate choice and does not affect whether you appear in AI search.
- GPTBot · OpenAI
- Google-Extended · Google
- Applebot-Extended · Apple
- CCBot · Common Crawl
- Bytespider · ByteDance
User-triggered
Fetches your page when a person asks an AI to visit it. These may ignore robots.txt, so asking them to stay away is a request, not a guaranteed block.
- ChatGPT-User · OpenAI
- Perplexity-User · Perplexity
- Claude-Web · Anthropic
FAQ
Questions about the AI Crawler Check
Is the tool really free?
Yes. You can run the AI Crawler Check without signing up, leaving your email or installing anything. There's a daily limit per check, so the tool stays fast for everyone.
Is it a problem to have no robots.txt?
No. With no robots.txt there are no rules, so every crawler may read everything — a perfectly normal, healthy state. We don't flag it as a problem.
I block Google-Extended — is that bad?
Not necessarily. Google-Extended only controls whether Google may use your content to train Gemini. It does not affect your normal Google ranking, nor AI Overviews. Opting out of training is a legitimate choice — which is why we mark it calmly, not red.
How is this different from SEO and schema?
robots.txt decides whether crawlers may read your site; schema and on-page SEO decide how well they understand it. They go together. Run our Schema Check and SEO Check on the same page too.
Do you store our data?
We read robots.txt only to give you the verdict. We record the domain (not the full URL, and no IP) to see how the tool is used and improve it. The result is shown only to you.
Want to show up in AI answers?
robots.txt is the first layer. An AI visibility audit looks at the whole picture — and what it takes to get cited.