Your website can block AI crawlers in two places. The first is robots.txt, whose rules GPTBot, ClaudeBot and PerplexityBot follow. The second is your CDN, firewall or bot protection, which can refuse their requests before a page is ever served.
To find out which applies to you, read robots.txt the way each crawler reads it. Then request a page with each crawler's user agent, and check your server logs for real visits and the status codes those visits received.
We got this wrong on our own site from 11 August to 21 October 2026. This guide covers the mistake we made and the checks that would have caught it.
Which AI crawlers do I actually need to check?
Each company runs more than one bot, and each bot has a different job. Blocking one does not block the others, and allowing one does not allow the others.
| Company | User agent | What it does |
|---|---|---|
| OpenAI | GPTBot | Collects content that may be used for model training |
| OpenAI | OAI-SearchBot | Crawls for ChatGPT search results |
| OpenAI | ChatGPT-User | Fetches a page when a person's conversation needs it |
| Anthropic | ClaudeBot | Collects training data |
| Anthropic | Claude-SearchBot | Crawls for Claude's search index |
| Anthropic | Claude-User | Fetches pages for a live user question |
| Perplexity | PerplexityBot | Crawls and indexes pages for Perplexity answers |
| Perplexity | Perplexity-User | Visits pages on behalf of a user's live query |
What the three companies document:
- OpenAI. Its crawler overview says these agents can each be allowed or disallowed independently. A site can permit OAI-SearchBot for ChatGPT search while blocking GPTBot from training.
- Anthropic. As Search Engine Journal reported, Anthropic's documentation warns that blocking ClaudeBot only stops training-data collection. It has no effect on Claude-SearchBot or Claude-User.
- Perplexity. Perplexity's documentation recommends explicitly allowing PerplexityBot in robots.txt and permitting its published IP ranges. It lists Perplexity-User separately.
One caveat about the live fetchers. PPC Land reported that OpenAI removed language saying robots.txt applies to ChatGPT-User, because its requests are triggered by a person rather than by autonomous crawling. For that agent, robots.txt is not the deciding layer. Your firewall is.
These bots are not occasional visitors. We counted requests in the nginx logs of pickaclass.com, an edtech platform we work with, from 28 July to 11 August 2026:
- ClaudeBot: 19,272,125 requests
- GPTBot: 2,744,985 requests
- Googlebot: 52,292 requests
Keep those numbers in mind for the firewall section.
How do I read robots.txt the way a crawler does?
Open https://yourdomain.com/robots.txt and read it with three rules in mind.
- A crawler obeys the group that names it. If there is a
User-agent: GPTBotgroup, GPTBot follows that group and ignoresUser-agent: *. It inherits nothing from the general group, even if its own group has only one harmless line. A generalDisallow: /applies to every bot that has no group of its own. - Paths match as prefixes.
Disallow: /bblocks/b, but also/blog,/businessand/b2b-pricing. - The more specific rule wins. If an
Allowand aDisallowboth match a URL, the longer, more specific path decides.
It also helps to know what robots.txt is not. RFC 9309, published by the IETF in September 2022, standardised the convention. It states that robots.txt rules are not a form of access authorization, so compliance is voluntary.
Well-behaved crawlers like GPTBot, ClaudeBot and PerplexityBot follow it. For them, robots.txt decides whether a page gets read. It does not lock anyone out.
The prefix trap we fell into
On itgrows.ai we wanted to hide a single redirect page, /b. The rule looked like this:
User-agent: *
Disallow: /b
Because robots.txt rules match prefixes, that line also closed /blog, /business and every article in our business blog to all crawlers. It did so from 11 August to 21 October 2026.
Nothing broke visibly, and the pages loaded fine in a browser. The only signals were quiet ones:
- Bing Webmaster Tools listed our business page as "discovered but not crawled" from 11 August 2026.
- Google had none of the business pages indexed.
On 21 October 2026 we changed the rule to:
User-agent: *
Disallow: /b$
The $ anchors the rule to the end of the URL, so it now matches only /b. If your file has short disallow paths, check every URL on your site that starts with the same characters.
Other common robots.txt blocks
Beyond the prefix trap, these are the usual ways a robots.txt file ends up shutting crawlers out:
- A
Disallow: /left over from a staging site that was copied to production. - A CMS setting that asks search engines not to index the site, which can rewrite robots.txt.
- A "block all AI bots" list pasted from a template. It blocks search and live-fetch agents along with training crawlers.
- A robots.txt that redirects, times out or returns a server error instead of a normal response, which can make crawlers hold back.
Can my CDN or firewall block AI crawlers even when robots.txt allows them?
Yes, and this block is harder to spot because robots.txt still says "welcome". The request never reaches your site, or it reaches a challenge page instead of your content. Typical causes:
- Bot management settings with a category for AI crawlers that is set to block.
- WAF rules that deny unknown or automated user agents.
- Rate limits that answer with 429 once a bot crosses a threshold.
- JavaScript or CAPTCHA challenges, which a crawler cannot solve. It receives the challenge HTML, not your page.
- Country or IP blocks that catch the data centres crawlers run from.
- Security plugins on the hosting or CMS level that act like a small firewall.
The volume figures above explain why rate limits trip so often. A limit tuned for Googlebot, which sent 52,292 requests in that fortnight, will not survive ClaudeBot sending more than 19 million.
Someone on the operations side sees a spike and adds a rule. The site disappears for one crawler, and nobody in marketing knows.
If you use Cloudflare, we wrote a separate walkthrough on fixing GPTBot blocked by Cloudflare. For Perplexity, follow its documentation and allow the published IP ranges, not just the user agent.
How do I test each crawler one by one?
Work through these steps for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot.
1. Read robots.txt per token. For each user agent, find the group that names it, or the * group if none does. Take five important URLs: home, a service page, a blog article, pricing and contact. Check each against every Disallow in that group, remembering prefix matching.
2. Request a page with each user agent. From a terminal:
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yourdomain.com/your-page
curl -s -o /dev/null -w "%{http_code}\n" -A "ClaudeBot" https://yourdomain.com/your-page
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://yourdomain.com/your-page
Compare the result with a normal browser user agent. If the browser gets 200 and GPTBot gets 403, 429 or 503, a rule keyed on the user agent is blocking it.
Treat a pass with caution. Some bot protection checks the IP address as well, so a spoofed user agent from your laptop can be treated differently from the real crawler.
3. Check your server logs. This is the ground truth. Search the access logs for each token and look at the status codes:
- Mostly 200: the crawler is getting through.
- A wall of 403 or 429: it is being refused.
- Zero lines for weeks: either the block happens before your server (check the CDN) or the crawler has not found you.
Our guide to finding AI crawlers in server logs shows the commands.
4. Check the CDN's security events. Filter by the same user agents. Blocked and challenged requests show up there even though they never appear in your origin logs.
5. Trigger a live fetch. Ask ChatGPT or Perplexity a question about a specific page on your site. A few minutes later, look for ChatGPT-User or Perplexity-User in the logs.
These fetches happen while an assistant answers a real person. In about two weeks of logs, the edtech platform received 1,918 of them on content pages. A client blog of a few dozen articles received 56 across 24 different articles in the same fortnight. A small site will see few, but it should see some.
6. Check webmaster tools. A page stuck at "discovered but not crawled" in Bing Webmaster Tools was our first visible symptom. Do not wait for a drop in traffic to notice it. How much Bing matters for ChatGPT is a separate question, covered in does ChatGPT use Bing.
What if nothing is blocked and assistants still don't name me?
Open access is a precondition, not a result. On 21 October 2026, the day we fixed our rule, we put 40 questions about getting a business into AI answers to ChatGPT, Claude, Gemini and Perplexity, three times each. We were named in 0 of 160 answers.
Being indexed is not the same as being found either. On 19 October 2026 we asked assistants for the exact titles of 18 articles, each older than three weeks, on three client sites:
- ChatGPT found 1 of 18.
- Claude found 7 of 18.
- Google had indexed 29 of 30 articles of one of those sites.
Size of site also matters. A site with complete markup but only 75 URLs stayed absent from answers on its own article's exact topic. A site with about 190,000 crawlable URLs was crawled constantly. Surface counts.
Do not judge your access by referral traffic. In about two weeks of logs from three of our course websites, human visitors arriving from AI assistants totalled one, from Copilot. Assistants mostly answer in place, so the crawlers' own log lines tell you more than analytics.
This is the work ItGrows does:
- publishing articles on the client's own domain that answer buyers' questions;
- fixing robots.txt, sitemaps, Bing and Google indexing, and IndexNow;
- every week asking ChatGPT, Claude, Gemini and Perplexity the client's buyer questions three times each, and counting the answers that name the client.
It costs $499 a month (plus a one-time $499 setup, free with 3 months or a year), and nobody can guarantee that an assistant will name a business. To see where you stand first, use the free check. In about 30 seconds it shows whether ChatGPT and Perplexity name your business for five of your buyers' questions.
Frequently Asked Questions
Does blocking GPTBot remove my site from ChatGPT?
Not by itself. OpenAI documents GPTBot as the training crawler and OAI-SearchBot as the crawler for ChatGPT search, and each can be allowed or blocked separately. If you block both, or a firewall blocks OpenAI's traffic in general, ChatGPT loses its ways of reading your pages.
Is robots.txt enough to keep AI crawlers out?
No. RFC 9309 states that robots.txt rules are not a form of access authorization, so following them is voluntary, and OpenAI no longer says robots.txt applies to ChatGPT-User. If you need to enforce a block, do it at the firewall or CDN.
How often should I re-check crawler access?
After every robots.txt edit, CDN or firewall rule change, site migration and CMS update, and once a month otherwise. Our own prefix mistake ran from 11 August to 21 October 2026 without any visible error on the site. That is why a routine check is worth doing even when everything looks fine.
Can I check crawler access without access to server logs?
Partly. Run the robots.txt reading and the curl tests yourself, look at your CDN's security dashboard, and check Bing Webmaster Tools for pages that are discovered but not crawled. For the logs themselves, ask your host or developer for a two-week export filtered to the AI user agents.