Why we built this
On itgrows.ai a robots.txt rule meant to hide one redirect page, /b, was written as a prefix. From 11 August to 21 October 2026 it also closed /blog, /business and every one of our guides to all crawlers. The pages loaded fine in a browser, the rule looked harmless, and Bing listed our business page as “discovered but not crawled” the whole time. Reading the rules did not reveal it; testing real URLs against them does.
What to do with the result
- Rules closing pages you want read: rewrite them so they match only what you meant, for example with a trailing
$. See is your website blocking AI crawlers. - A crawler refused by your server: it is your firewall or CDN, not robots.txt. See GPTBot blocked by Cloudflare.
- Everything open: the next question is whether the assistants name you. Run the AI visibility check.
Frequently Asked Questions
How does the check decide whether a page is blocked?
It reads your robots.txt the way crawlers do: the group for each crawler (or the * group), the longest matching rule wins, allow wins a tie, * and $ work as wildcards. Then it tests the pages listed in your sitemap against those rules, one crawler at a time.
Why would robots.txt block pages I never meant to block?
Rules are prefixes. A rule written for one page, such as Disallow: /b, also matches /blog and /business. We made exactly this mistake on itgrows.ai and it closed our guides to every crawler for ten weeks before we noticed.
My robots.txt allows everything. Can crawlers still be blocked?
Yes. A firewall, CDN or bot-protection setting can refuse a crawler even when robots.txt invites it. The check also asks your server as GPTBot, ClaudeBot and PerplexityBot and shows what it answered.
Which crawlers matter for being named in AI answers?
The ones that feed search and live answers: OAI-SearchBot and ChatGPT-User for ChatGPT, Bingbot because ChatGPT search and Copilot draw on Bing, Googlebot for Gemini and AI Overviews, PerplexityBot, and Anthropic's crawlers for Claude. GPTBot is OpenAI's training crawler.