If you want ChatGPT, Claude, Gemini and Perplexity to name your business, you should not block the search and live-retrieval crawlers, because blocking them removes you from the answers you are trying to appear in. The only real decision is about training crawlers, and you can make that one separately, bot by bot, without touching the rest.
The rest of this page explains the three kinds of AI crawlers, what blocking each one costs and buys you, and the default we would set for a business that wants to be named.
Which AI crawlers are actually visiting your site?
"AI crawler" covers three different jobs. Robots.txt lets you treat each one differently, and that is the whole point of this decision.
Training crawlers collect pages that may be used to train future models. OpenAI's GPTBot is the best-known example. Google-Extended belongs in this group too, although it is not a crawler at all: according to Google's documentation on its common crawlers, it has no HTTP user-agent string of its own. It is a robots.txt token that controls whether content Google has already crawled is used to train or ground Gemini models, without affecting Google Search crawling.
Search crawlers build the index an assistant searches when it looks something up. OpenAI's OAI-SearchBot does this for ChatGPT search. PerplexityBot does it for Perplexity, and, citing Perplexity's own documentation, Grupa Insight notes that it respects robots.txt and is not used to pre-train Perplexity's foundation models.
Live retrieval agents fetch a page at the moment an assistant is answering a real person. ChatGPT-User and Perplexity-User are the common ones. In about two weeks of logs, an edtech platform we work with received 1,918 such fetches on its content pages; a client blog of only a few dozen articles received 56, spread across 24 different articles. Each of those is a person asking a question and an assistant reading the page to answer it.
| Type | Examples | What it does | What blocking it costs you |
|---|---|---|---|
| Training | GPTBot, Google-Extended (a token, not a bot) | Collects or permits content for model training; Google-Extended also covers grounding Gemini | Your pages are left out of future training data; for Google-Extended, also out of Gemini grounding |
| Search index | OAI-SearchBot, PerplexityBot | Builds the index the assistant searches when it looks things up | You drop out of ChatGPT search results and Perplexity's cited answers |
| Live retrieval | ChatGPT-User, Perplexity-User | Fetches a page while answering a specific person | The assistant cannot read your page at the moment someone asks about you |
For the full list of user agents and who runs them, see What Are GPTBot, OAI-SearchBot, ClaudeBot and Other AI Crawlers?
What do you lose if you block them?
Not much traffic, but a lot of visibility. In about two weeks of logs from three of our course websites, roughly 1.2 million requests in the two busiest logs, we counted exactly one human visitor arriving from an AI assistant, and that one came from Copilot. ChatGPT, Perplexity, Gemini, Claude and Meta AI sent zero. Assistants mostly answer in place rather than send a click.
That changes what you are protecting. The value is not the visit, it is being the name in the answer. Some of our clients report that this turns into business: FreightSizer, a B2B SaaS for freight dimensioning, reports two new B2B customers in one month, both from ChatGPT, and appslift, an app store optimisation service, reports 2 to 5 leads a month from AI assistants. Those are the clients' own figures, not ours.
Both paths into an answer matter. In a test on 23 September 2026 with 20 German business questions, the model answered 10 without searching at all and cited nobody, so whatever it said came from what it already knew. Where it did search, it cited primary sources such as laws, ministries, the OECD and large accounting firms on 9 of 10 questions, and it named firms only on "who should I hire" or "how do I choose" questions. Blocking search and retrieval crawlers takes you out of the second path entirely. Blocking training crawlers may thin out the first.
What do you gain by blocking them?
There are two legitimate reasons to block, and both are narrower than "block all AI".
The first is that your content is your product. A paid research library, a course platform or a news archive may reasonably not want its text used for training. OpenAI makes this easy to separate: its overview of OpenAI crawlers states that the robots.txt controls for OAI-SearchBot and GPTBot are independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot so its content is not used to train OpenAI's models.
The second is load. Across the full retained nginx logs of pickaclass.com, an edtech platform we work with, from 28 July to 11 August 2026 we counted ClaudeBot at 19,272,125 requests, Meta-ExternalAgent at 3,854,076, GPTBot at 2,744,985, Applebot at 2,655,555 and Amazonbot at 1,170,268. Googlebot made 52,292. If your server or hosting bill feels that, it is a real cost. Rate limiting at the server or CDN addresses it more precisely than a blanket Disallow does.
What blocking does not buy you is protection. The IETF standard, RFC 9309, says plainly that robots.txt rules are not a form of access authorization; compliant crawlers are asked, not forced. An academic analysis of robots.txt adds that it lacks formal enforcement and that smaller or malicious bots frequently disregard it. So a robots.txt block mainly stops the well-behaved bots, which are the ones run by the assistants that could cite you, while the bots you least want keep coming.
What is a sensible default for a business that wants to be named?
For most businesses whose website exists to win customers, this is the default we would set:
- Allow search crawlers. OAI-SearchBot and PerplexityBot should be able to read every public page you want cited.
- Allow live retrieval agents. ChatGPT-User and Perplexity-User are effectively a person asking about you right now.
- Keep Googlebot and Bingbot open exactly as you would for normal search.
- Allow training crawlers unless your content is the product. Your service pages, pricing and articles are marketing; having models know about you is the goal. If your text is what customers pay for, disallow GPTBot and similar training bots and leave the search bots open.
- Think twice about Google-Extended. Because it also governs grounding Gemini models, blocking it is not a pure training opt-out for a business that wants to appear in Gemini.
A robots.txt that keeps search and retrieval open while opting out of OpenAI training looks like this:
``` User-agent: GPTBot Disallow: /
User-agent: OAI-SearchBot Allow: /
User-agent: ChatGPT-User Allow: /
User-agent: PerplexityBot Allow: /
User-agent: * Allow: / ```
Delete the first block if you want training allowed too. The step-by-step version, including the other bots, is in How Do I Allow GPTBot and Other AI Crawlers in robots.txt?
Which mistakes block AI crawlers by accident?
Blocking crawlers without meaning to is easy. We did it ourselves.
On itgrows.ai, a robots.txt rule meant to hide a single redirect page, /b, was written as a prefix. Robots.txt rules match prefixes, so from 11 August to 21 October 2026 that one line also closed /blog, /business and every article in our business blog to all crawlers. Bing Webmaster Tools listed our business page as "discovered but not crawled" from 11 August 2026, and Google had none of the business pages indexed. We changed the rule to /b$ on 21 October 2026. On the same day we measured our own visibility: across 20 English and 20 German questions, put to four assistants three times each, we were named in 0 of 160 answers and none of our pages was cited.
Other causes to look for:
- CDN or firewall bot settings that block crawlers your robots.txt allows. The robots.txt looks fine and the bot still gets a 403. See How to Fix GPTBot Being Blocked by Cloudflare.
- Copied "block all AI bots" lists that include search and retrieval agents alongside training bots.
- Assuming open means found. On 19 October 2026 we asked assistants for the exact titles of 18 articles on three client sites, each older than three weeks. ChatGPT found 1 of 18 and Claude found 7 of 18, even though Google had indexed 29 of 30 articles on one of those sites. Being crawlable is the entry ticket, not the result.
Size of the crawlable surface matters too. One site with complete markup, server rendering, Article, FAQPage and Organization JSON-LD and a sitemap, but only 75 URLs, was still absent from answers on its own article's exact topic. A site with about 190,000 crawlable URLs was crawled constantly. Unblocking helps only if there are enough pages worth reading behind the door.
How do you check what your site is doing today?
Start with your server logs: search for the user agents above and look at the status codes they receive. A 200 means they got the page; a 403 or a challenge page means something is blocking them. How to Tell If AI Crawlers Visit Your Site and Find Them in Logs walks through it.
Then check your robots.txt line by line for prefix rules that catch more than you meant, as ours did.
Finally, check the outcome rather than the plumbing: do assistants name you when buyers ask? Answers vary, so one ask proves little. At ItGrows we put each client's buyer questions to ChatGPT, Claude, Gemini and Perplexity every week, three times each, and count a business as named for a question if any of the three asks names it. If you want a quick first look, the free check at itgrows.ai generates five buyer questions from your homepage and shows in about 30 seconds whether ChatGPT and Perplexity name you. Nobody can guarantee that an assistant will name a business, but you can at least make sure you are not the one keeping it out.
Frequently Asked Questions
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI documents that the robots.txt controls for GPTBot and OAI-SearchBot are independent. You can disallow GPTBot to opt out of training and still allow OAI-SearchBot so your pages can appear in ChatGPT search.
Does robots.txt actually stop AI bots?
Only the ones that choose to comply. RFC 9309 states that robots.txt rules are not access authorization, and smaller or malicious bots often ignore them. If you need to stop a bot for real, you have to do it at the server, CDN or firewall level.
Will blocking Google-Extended hurt my Google rankings?
Google says Google-Extended does not affect Google Search crawling. It does control whether Google-crawled content is used to train or ground Gemini models, so blocking it can matter if you want to appear in Gemini answers.
If I allow every AI crawler, will assistants start naming my business?
Not automatically. Access is a precondition, but in our 19 October 2026 test ChatGPT found only 1 of 18 client articles by exact title, even though Google had indexed 29 of 30 articles on one of those sites. Being named also depends on having enough pages that answer your buyers' questions and on sources that assistants trust mentioning you.