Choose, crawler by crawler, which AI bots may fetch your site and get a robots.txt that does what it says. Or check your live file against Google's robots.txt rules.
A
A
CB10 allowed
DMAExample: training blocked, search and user requests allowed1User-agent: *2Disallow: /admin/3 4User-agent: GPTBot5User-agent: Amazonbot6User-agent: ClaudeBot7User-agent: Google-Extended8User-agent: Applebot-Extended9User-agent: Meta-ExternalAgent10User-agent: CCBot11User-agent: Bytespider12Disallow: /13 14Sitemap: https://example.com/sitemap.xml15
AACB10 allowedDMA
AI Rankia/admin/ closed to everyone.One path, for every crawler (*) or one you name. A named crawler ignores *, so we copy the * rules into its group (except crawlers you block).
WordPress defaults: the two rules its built-in robots.txt writes for a public site, Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php (source). Not at the root? Adjust the paths.
1User-agent: *2Disallow:3
Generated in your browser; nothing is sent to us. Upload it to the root of each host, so it answers at https://yourdomain/robots.txt (www and non-www are separate hosts).
Allow or block each of 18 AI crawlers, grouped by what they do. Nothing leaves your browser.
A * group for private paths, one group for every blocked crawler, your Sitemap lines. No crawl-delay: Google doesn't support it.
We fetch your robots.txt once and show which of 20 crawlers are blocked, and by which line.
Both verdicts below are computed by the Check's own matcher.
Inside a crawler's rules, Google applies the matching rule with the longest path (source). Wildcards count toward the length.
Allow: /pub/ in a second GPTBot group still opens /pub/ under a Disallow: / in the firstAllow is flagged "changes nothing" only when it overrides no Disallow for the same crawlers, on any URL/docs/a.pdfAllowedBoth match. The longer pattern wins, and wildcards count toward its length.
/folder/pageAllowedSame length: a tie goes to Allow.
Grouped by what each company says its crawler does.
OAI-SearchBotPerplexityBotClaude-SearchBotDDuckAssistBotAAmzn-SearchBot
Disallow: / for * closes the site to every crawler without its own group. Fine on a staging site.Where the spec is silent, we follow Google's open-source parser.
| In your file | What Google's parser does |
|---|---|
A Crawl-delay between two User-agent lines | Doesn't split them. Only Allow or Disallow ends a group. |
User-agent: GPTBot/1.0 | Counts as GPTBot. |
Allow: /index.html or /index.htm | Also allows /: the folder URL opens too, even under Disallow: /. |
Misspellings such as dissallow or user agent | Still count for Google; we flag them as warnings. |
noindex for that (source).Crawl-delay "where appropriate" (source); Google ignores it.robots.txt: User-agent lines followed by Allow or Disallow rules, uploaded to your domain's root. Generate writes it; you upload it.User-agent: * group with the paths you don't want crawled (or an empty Disallow:), groups for crawlers you treat differently, and a Sitemap: line with your sitemap's full URL.Allow. Want it out of search? Use noindex and leave it crawlable: Google can't see a noindex it can't fetch.AI Rankia tracks whether AI assistants name or cite you, across the AI engines it monitors.
Allowing OAI-SearchBot or PerplexityBot lets them fetch your pages, not cite them. The free scan asks AI engines the questions buyers in your category ask, then shows whether they name you, who they name instead and which sources they cite.
* groupA crawler named in any group follows only the groups that name it (source). Give GPTBot its own Allow: / and your private paths no longer apply to it.
/private/report1User-agent: *2Disallow: /private/3 4User-agent: GPTBot5Allow: /
GPTBotfollows User-agent: GPTBotAllowedUser-agent: *BlockedGPTBotAAmazonbotClaudeBotGoogle-ExtendedAApplebot-ExtendedMeta-ExternalAgentCCCBotBBytespider
AAmazon: Amazonbot "may be used to train Amazon AI models"; Amzn-SearchBot "does not crawl content for generative AI model training". source
ChatGPT-UserPerplexity-UserClaude-UserMMistralAI-UserAAmzn-User