When Cloudflare blocks GPTBot, the cause is usually one of its AI crawler settings. It might be a managed robots.txt that disallows AI training, the edge blocking that comes with it, or a crawler-specific block in AI Crawl Control. To fix it, find which of those layers is doing the blocking and switch GPTBot to allowed. Allow OAI-SearchBot too if you want to appear in ChatGPT search. Then confirm in your server logs that OpenAI's requests actually arrive.
The rest of this page covers how each setting works, how to prove where the block sits, and what to decide before you switch everything to "allow".
Why does Cloudflare block GPTBot in the first place?
In July 2024 Cloudflare introduced a one-click option that lets customers block AI bots from scraping content for model training, built on top of the voluntary robots.txt mechanism (The Register). Many site owners turned it on, or a developer or agency did it for them, and then forgot about it. Later the same owners wonder why ChatGPT never seems to read their site.
Today there are two separate mechanisms, and they are easy to confuse. Cloudflare's robots.txt setting documentation says that enabling Bot Preference Sync with Training set to Disallow does two things. It adds managed robots.txt directives to your site, and it enables edge blocking for AI training crawlers. AI Crawl Control is the separate mechanism that enforces crawler-specific blocks.
So the block can live in either place, or in both. Cloudflare's bot reference lists GPTBot as operated by OpenAI, in the "AI Crawler" category, with its own detection IDs. Cloudflare recognises GPTBot specifically. A category-level or crawler-level rule will catch it even when your own robots.txt never mentions it.
There is a third, less obvious cause: general bot protection or custom firewall rules that you or a developer set up. These rules are not AI-specific, but they can challenge or block any automated visitor, including GPTBot. A crawler cannot solve a challenge page, so a challenge has the same effect as a block.
How do I check whether Cloudflare is actually blocking GPTBot?
Check from three angles, because each one catches a different layer.
1. Read your live robots.txt. Open yourdomain.com/robots.txt in a browser and compare it with the file on your server. If the live version contains AI crawler directives you never wrote, Cloudflare is adding rules at the edge.
2. Look at your Cloudflare settings and events. Open AI Crawl Control and see whether GPTBot is set to block. Then check your security events for requests identified as GPTBot that were blocked or challenged. Entries there tell you which rule is responsible.
3. Look at your origin server logs. This is the ground truth. If GPTBot requests reach your server and get 200 responses, Cloudflare is letting them through. If none appear over a couple of weeks, something upstream is stopping them.
For scale, we counted requests in the full nginx logs of an edtech platform we work with (pickaclass.com) from 28 July to 11 August 2026. GPTBot made 2,744,985 requests, ClaudeBot 19,272,125 and Googlebot 52,292. That is a very large site, and a small one will see far fewer, but zero is a strong signal of a block. Our guide Is My Website Blocking GPTBot, ClaudeBot and PerplexityBot? covers log checks step by step.
One warning about testing with curl and a fake user agent. Cloudflare identifies bots by more than the user-agent string. A request from your laptop that claims to be GPTBot does not show how the real crawler is treated. It may be blocked when the real one is not, or the other way round. Trust the dashboard and your logs instead.
How do I allow GPTBot in Cloudflare?
Work through the layers in this order. Fixing only one of them is the most common reason a block survives.
Step 1: Allow the crawler in AI Crawl Control
AI Crawl Control is where crawler-specific blocks are enforced, so set GPTBot to allow there. If you want to appear in ChatGPT search, check OAI-SearchBot as well. Review the other assistants you care about, such as ClaudeBot and PerplexityBot, while you are in that screen.
Step 2: Change the training preference in the managed robots.txt
GPTBot is a training crawler. While Bot Preference Sync is on with Training set to Disallow, Cloudflare keeps adding disallow directives and edge blocking for AI training crawlers. Allowing GPTBot elsewhere will not be enough while that setting stands. Change the training preference, or turn off the managed robots.txt and maintain the file yourself.
Step 3: Fix your own robots.txt
Your own file has to agree with Cloudflare. Add explicit groups for GPTBot and OAI-SearchBot, and make sure no wildcard rule closes paths you want read. How Do I Allow GPTBot and Other AI Crawlers in robots.txt? has the exact lines.
Read every rule as a prefix, because that is how robots.txt matches paths. On our own site, a rule meant to hide a single redirect page, /b, was written as a prefix. From 11 August to 21 October 2026 it also closed /blog, /business and every article in our business blog to all crawlers.
Bing Webmaster Tools listed our business page as "discovered but not crawled" from 11 August 2026, and Google had none of those pages indexed. We changed the rule to /b$ on 21 October 2026. A single missing character caused all of it.
Step 4: Remove IP blocks and challenges
Check whether anyone added firewall rules that block crawler IP ranges, or bot protection that challenges automated traffic. If so, add exceptions for the crawlers you want.
Do not use IP blocking as your control mechanism either. Anthropic warns that blocking its bots by IP address is unreliable because it "impedes our ability to read your robots.txt file". The company also commits to honoring robots.txt and not bypassing CAPTCHAs (ppc.land). Robots.txt and Cloudflare's crawler controls are the cleaner levers.
Step 5: Confirm in the logs
After a few days, check your origin logs for GPTBot and OAI-SearchBot requests with 200 status codes. The job is not done until you see them.
Should I allow GPTBot, OAI-SearchBot, or both?
This is the real decision, and it is easy to get backwards. OpenAI's crawler documentation says GPTBot and OAI-SearchBot are controlled independently in robots.txt. You can allow OAI-SearchBot so your pages can appear in ChatGPT search results, and still disallow GPTBot to opt your content out of model training.
| Crawler | Operated by | What it does | What blocking it means |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for model training | Your content is opted out of training; on its own this does not opt you out of ChatGPT search |
| OAI-SearchBot | OpenAI | Lets pages appear in ChatGPT search results | Your pages lose their path into ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page while ChatGPT is answering a real person | The assistant cannot read your page at the moment someone asks about your topic |
The last row matters more than most people expect. In the same fortnight, the edtech platform received 1,918 live fetches on content pages from retrieval agents such as ChatGPT-User, OAI-SearchBot and Perplexity-User. A client blog of a few dozen articles received 56 such fetches across 24 different articles. Each of those is an assistant reading a page while answering someone.
If you want ChatGPT to mention your business, do not block OAI-SearchBot. Allowing GPTBot is a separate choice about training. Some businesses allow it because they want models to learn what they do. Publishers protecting paid content often allow search and block training. Both are legitimate. For a full list of who runs which crawler, see What Are GPTBot, OAI-SearchBot, ClaudeBot and Other AI Crawlers?
What mistakes keep the problem alive after the fix?
Fixing one layer and stopping. You allow GPTBot in AI Crawl Control, but the managed robots.txt still disallows training, or a firewall rule still challenges bots. Check all three layers every time.
Allowing GPTBot but not OAI-SearchBot. The setting you changed is the one that matters least for appearing in ChatGPT search.
Trusting a spoofed test. A fake user agent from your own machine says nothing reliable about the real crawler.
Expecting mentions the next week. Unblocking gives crawlers access. It does not make you visible in answers.
On 19 October 2026 we asked assistants for the exact titles of 18 articles on three client sites, each older than three weeks. ChatGPT found 1 of 18 and Claude found 7. Google had indexed 29 of the 30 articles on one of those sites. Being indexed did not mean an assistant could find the page.
Size of the site matters too. A site with complete markup but only 75 URLs was still absent from answers on its own article's exact topic. ChatGPT search also leans on other indexes, which is why Bing matters; see Does ChatGPT Use Bing? Do I Need Bing Webmaster Tools and IndexNow?
What should I do once GPTBot can reach the site?
Measure the answers, not just the crawl. In about two weeks of logs from three of our course websites, human visitors arriving from AI assistants were close to zero: 1 from Copilot and none from the others. Assistants mostly answer in place, so referral traffic will not tell you whether you are being named. You have to ask the assistants your buyers' questions and read what comes back.
Ask more than once. The same question asked three times often names different businesses. We count a business as named for a question if any of the three asks names it. We compare weeks only on the same questions and the same number of asks.
To see where you stand now, the free check at itgrows.ai creates five buyer questions from your homepage. It shows in about 30 seconds whether ChatGPT and Perplexity name you.
If you want this handled ongoing, ItGrows fixes crawler access and indexing, including robots.txt, sitemaps, Bing and Google indexing and IndexNow. It also publishes articles on your own domain that answer your buyers' questions. Every week it puts those questions to ChatGPT, Claude, Gemini and Perplexity, three times each, and counts the answers that name you and the sites they cite. It costs $499 a month (plus a one-time $499 setup, free with 3 months or a year). Nobody can guarantee that an assistant will name a business, and we say so up front.
Frequently Asked Questions
How do I know whether I switched on an AI crawler block myself?
Compare your live robots.txt with the file on your server, then check AI Crawl Control and your security events in Cloudflare. If the live file contains AI directives you did not write, or GPTBot shows as blocked, a Cloudflare setting is responsible. Your origin logs confirm whether real GPTBot requests get through.
Will allowing GPTBot get my business mentioned in ChatGPT?
Not by itself. Unblocking gives crawlers access, but whether an assistant names you depends on what it finds and trusts when it answers. For ChatGPT search, OAI-SearchBot access matters more than GPTBot. No one can promise a mention.
Can I allow ChatGPT search but block training?
Yes. OpenAI says GPTBot and OAI-SearchBot are controlled independently, so you can allow OAI-SearchBot and disallow GPTBot. In Cloudflare, make sure the training disallow and your crawler settings match that choice.
Should I block AI crawlers by IP address instead of robots.txt?
Use robots.txt and Cloudflare's crawler controls instead. Anthropic warns that IP blocking is unreliable and stops it from reading your robots.txt file. IP rules also tend to outlive the reason they were added, and they quietly block crawlers you later want.