Free tool · No account needed

AI Crawler Checker

Can GPTBot, OAI-SearchBot, PerplexityBot and the rest read your site? We read robots.txt the way Google documents it, name the line behind each block, and request your home page with each crawler's published User-Agent.

Check my site

Rules for 18 crawlers and tokens from

yourdomain.com/robots.txtIllustration

1User-agent: *2Disallow: /admin/3 4User-agent: GPTBot5User-agent: Google-Extended6Disallow: /7 8User-agent: OAI-SearchBot9Allow: /10 11Sitemap: https://yourdomain.com/sitemap.xml

How it reads for /robots.txt

GPTBotDisallow: / (line 6, User-agent: GPTBot)Blocked

Google-ExtendedDisallow: / (line 6, User-agent: Google-Extended)Blocked

OAI-SearchBotAllow: /

Illustration: an example robots.txt and how the checker reads it for the home page, following Google's robots.txt rules.Illustration: the rule that decides each crawler on an example home page, read the way Google reads robots.txt.

app.airankia.com/free-tools/ai-crawler-checker

We request your robots.txt once and your home page 11 times: once as a normal browser and once as each of the 10 crawlers that publish their User-Agent. It usually takes under 20 seconds.

What we check

18 crawlers, three different jobs

Grouped the way their companies separate them. 10 publish a full User-Agent, so we also request your page as them.

Search and AI answers6

OAI-SearchBotRequested
PerplexityBotRequested
Claude-SearchBotNo public User-Agent

How this tool works

One domain, three checks

robots.txt, read Google's way
For each crawler: every group that names it, combined; only if none does, the * group. The longest path wins; on a tie, Allow wins.
Your home page, as each crawler
Once with a normal browser User-Agent, once with each company's documented one. We compare the HTTP status of each answer.
The indexing rules on it
The X-Robots-Tag header and robots meta tags in the first 64 KB of the browser's answer, as written. A crawler that gets different rules is flagged.

Google's rules, not every crawler's.
We follow Google's robots.txt spec; other crawlers may read edge cases differently.

How to read the result

What each verdict means

The labels the result table uses. Green lets the crawler in, red keeps it out, amber means one check can't tell.

Lets it in

Allowed

robots.txt allows it and its User-Agent got a 2xx.

Allowed by robots.txt

robots.txt allows it; we don't request as this crawler.

Server allows it

robots.txt unreadable, but its User-Agent got a 2xx.

Keeps it out

Blocked by robots.txt

The rules that apply ask it not to fetch the page. We show the deciding line.

Blocked by server (mismatch)

robots.txt allows it and our browser got a 2xx, but its User-Agent got an error such as 403 or 429, a challenge page or over 3 redirects. Look at your firewall, CDN or app.

Refused by server

The same refusal, with no robots.txt to compare.

Can't tell

Inconclusive

Our browser got no 2xx either, or the crawler's request got no answer. A dropped connection can be a firewall or the network.

Unknown

No robots.txt to judge by, and not requested.

The most common mistake

A named group inherits nothing from *

Give GPTBot its own group and the private paths under * stop applying to it. A crawler with no group of its own inherits everything there, Disallow: / included.

The result lists each blocking line with the crawlers it blocks
Other Disallow lines are folded below: they can decide other paths

robots.txtExample

Before and after the fetch

When robots.txt fails, and what it can't do

12 hGoogle stops crawling a site after a 5xx on robots.txt, then keeps retryingSource: Google robots.txt spec

~24 hfor a robots.txt change to show in ChatGPT search resultsSource: OpenAI

What robots.txt answers

2xxWe read the rules and judge each crawler by them.This tool

401 · 403 · challengeWe never saw the file: every robots.txt result is unknown. Real crawlers may still get it.This tool

404 · other 4xxNo crawl restrictions (429 excepted), as Google reads it.Google · this tool

5xxWe couldn't read the rules. Google pauses crawling the whole site and keeps retrying (source).Google

After the fetch: indexing rules

nosnippet · noneGoogle: nosnippet keeps content from being "used as a direct input for AI Overviews and AI Mode"; none is noindex plus nofollow (source).Google

Disallowed pageThe rules are never seen: Google says they "will not be found and will therefore be ignored". OpenAI's crawler page doesn't mention them: for its crawlers, robots.txt is the control ().

Limits

What one check can't tell you

Our requests come from our server, not the crawlers' IPs.
A firewall that checks IPs can treat us differently: a 200 doesn't prove the real crawler gets through. Only your server or CDN logs show that.
We test the home page only.
A rule or firewall setting can treat other paths differently.
No JavaScript, first 64 KB of HTML.
A robots meta tag added by a script or placed further down won't show.
Some fetchers don't promise to follow robots.txt.
OpenAI says its rules "may not apply" to ChatGPT-User (source); Perplexity says Perplexity-User "generally ignores robots.txt rules" (source). There, robots.txt states a preference.

Your data

No AI, no score, no copy of your pages

What we don't do

A plain check, nothing kept

The result stays in memory for 10 minutes, so a repeat check doesn't hit your site again.

No AI in the resultNo scoreHome page onlyPages not saved

What we log

Enough to enforce the daily limits

Limits are counted per IP hash.

What we log 5 items

The domain checked and a short summary of the result
An unsalted SHA-256 hash of your IP, not the address: whoever holds it could recover the address by hashing every possible one
Your country as Cloudflare reports it, and whether the IP looks like a VPN or proxy
If you gave us your email, a link to it
For the VPN check, our own IP service answers from downloaded lists; if it fails or can't place the IP, it goes to ipquery.io. Cached 30 minutes

FAQ

Questions people ask

What is GPTBot?

OpenAI's crawler for content that "may be used in training our generative AI foundation models". It isn't the crawler behind ChatGPT search; that's OAI-SearchBot, and you can allow or block each one separately in robots.txt (source).

Does ChatGPT respect robots.txt?

It depends on the agent. OpenAI documents robots.txt controls for GPTBot and OAI-SearchBot; a change takes about 24 hours to show in its search. For ChatGPT-User, which fetches a page because a user asked, the rules "may not apply" (source).

How do I block AI crawlers?

Add a group per crawler, for example User-agent: GPTBot then Disallow: /. That works for crawlers that follow robots.txt; to actually refuse requests you need a firewall or CDN rule. Our robots.txt generator writes the groups.

Is Cloudflare blocking AI crawlers on my site?

It can. AI Crawl Control lets you "set allow or block rules for individual crawlers" (source). That block isn't in robots.txt unless you turn on managed robots.txt, which writes lines like User-agent: GPTBot / Disallow: / (source). Without it, it shows here as a mismatch or a challenge page; with it, also as "Blocked by robots.txt".

Crawlers can read you. Do AI answers name you?

AI Rankia runs your tracked prompts on up to 7 AI engines with live web search and shows whether they mention or cite you. Its LLM Readiness Audit also renders the page in a headless browser.

Free · live in ~30s · no credit card, no signup to see it

orCreate your account →

Running an agency? One plan covers 2 to 30 client brands, priced by how many you manage. See agency pricing →

Keep exploring

Related free tools

Robots.txt Generator and CheckerBuilds a robots.txt with the AI crawlers you choose, or reviews your current file against Google's robots.txt documentation.llms.txt Generator and CheckerDrafts an llms.txt from your sitemap and page titles, or checks an existing one against the llmstxt.org proposal.Free AI visibility checkerTwo checks with separate results: whether Google AI Mode, AI Overviews and two small Claude and Gemini models name your brand for buyer questions, and whether AI crawlers can read one URL of your site.

By Pedro Spota, Co-founder, AI RankiaLast updated: October 10, 2026Drafted with AI assistance. Pending review by Pedro Spota.