To allow GPTBot and the other AI crawlers, give them a User-agent group in robots.txt with Allow: /, or make sure no rule anywhere in the file disallows them, and repeat any paths you still want hidden inside that group. Then test the file against your real URLs, because robots.txt rules match prefixes, and one short rule can close far more of your site than you meant.
The rest of this page covers the exact rules, which bots matter for being named in answers and which only collect training data, the prefix mistake that closed our own blog for ten weeks, and how to test.
What exact rules allow GPTBot and the other AI crawlers?
If your robots.txt has no group for a bot and no User-agent: * rule that blocks it, that bot is already allowed. A block can also arrive by accident: a copied template, a CMS plugin, or a broad Disallow under * can shut out crawlers nobody meant to exclude. If you are not sure what your current file does, start with Is My Website Blocking GPTBot, ClaudeBot and PerplexityBot?.
If you want to be explicit, list the crawlers together in one group. Several User-agent lines can share one set of rules:
``` User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /cart/
User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/
Sitemap: [the full URL of your sitemap] ```
The detail people miss: a crawler that finds a group naming it follows only that group and ignores the * group. So if you add User-agent: GPTBot with Allow: / and nothing else, GPTBot may now crawl /admin/ even though * disallows it. Repeat every private path in the named group.
Also check what sits in front of robots.txt. A CDN or firewall rule can return a 403 to a bot that robots.txt allows; GPTBot Blocked by Cloudflare walks through that case.
Which bots matter for answers and which only for training?
The vendors now split their crawlers by purpose, and the settings are separate. OpenAI states that OAI-SearchBot and GPTBot are controlled by independent robots.txt settings, so a site can allow OAI-SearchBot, which serves ChatGPT search citations, while disallowing GPTBot, which collects training data. For a deeper look at each user agent, see What Are GPTBot, OAI-SearchBot, ClaudeBot and Other AI Crawlers?.
| Bot | Operator | What it is for | Does robots.txt control it? |
|---|---|---|---|
| GPTBot | OpenAI | Collecting training data | Yes, independently of OAI-SearchBot |
| OAI-SearchBot | OpenAI | Pages ChatGPT search can show and cite | Yes, independently of GPTBot |
| ChatGPT-User | OpenAI | Fetches a page when a user's request triggers it | OpenAI says robots.txt rules may not apply |
| ClaudeBot | Anthropic | Anthropic's web crawler | Yes; Anthropic also honours Crawl-delay |
| PerplexityBot | Perplexity | Indexing pages for Perplexity's search results | Yes; Perplexity recommends allowing it |
| Perplexity-User | Perplexity | Fetches triggered by a user's question | Perplexity documents it as user-triggered |
| Google-Extended | Use of crawled content for Gemini training and grounding | Yes; it does not affect Google Search |
A few notes on the rows:
- OpenAI updated its documentation so that robots.txt compliance language applies to OAI-SearchBot and GPTBot but not ChatGPT-User, explaining that because these actions are initiated by a user, robots.txt rules may not apply.
- Anthropic's support article commits that its bots honour industry standard robots.txt directives and support the non-standard Crawl-delay extension, which is useful if crawl volume worries you.
- Perplexity says that to appear in its search results you should allow PerplexityBot in robots.txt and permit its published IP ranges. The IP part matters if your firewall filters by address.
- Google says the Google-Extended token controls use for Gemini training and grounding and does not impact inclusion in Google Search. Blocking it will not hurt your rankings, but grounding is how Gemini ties answers to sources, so think before you block it.
The live fetchers are the ones closest to a real buyer. In a fortnight of logs, an edtech platform we work with received 1,918 fetches from ChatGPT-User, OAI-SearchBot and Perplexity-User on content pages, each one an assistant reading a page while answering a person. A client blog of a few dozen articles received 56 such fetches across 24 different articles. The training crawlers are much louder: across pickaclass.com's nginx logs from 28 July to 11 August 2026 we counted 19,272,125 requests from ClaudeBot, 3,854,076 from Meta-ExternalAgent, 2,744,985 from GPTBot, 2,655,555 from Applebot and 1,170,268 from Amazonbot, against 52,292 from Googlebot.
The trade-off is straightforward. If you want assistants to read you when answering, allow the search and user agents at minimum. Blocking the training crawlers is a legitimate choice if server load or content reuse is the concern, but decide it per bot, not with one blanket rule.
How does prefix matching work, and how did it cost us ten weeks?
A robots.txt path is a prefix, not a page name. Disallow: /b blocks every URL whose path starts with /b: /b, /blog, /business, /b2b-pricing, /brand.png. There is no warning; the crawler just obeys.
This happened to us. On itgrows.ai, a rule meant to hide a single redirect page at /b was written as a prefix. From 11 August to 21 October 2026 it also closed /blog, /business and every article in our business blog to all crawlers. Bing Webmaster Tools listed our business page as "discovered but not crawled" since 11 August 2026, and Google had none of the business pages indexed. We changed the rule to /b$ on 21 October 2026.
The $ anchors the end of the path, so /b$ matches only /b exactly. Google and the large crawlers support $ and the * wildcard; check a crawler's documentation if you rely on them. Safer patterns:
Disallow: /b$for one exact page.Disallow: /admin/with the trailing slash, so/administration-servicesstays open.Disallow: /*?sessionid=for a query parameter, not the page behind it.
When a short rule and a longer Allow collide, the major crawlers apply the more specific (longer) match. That rescues you in some cases, but do not design a file around it.
How do I test that the rules actually work?
Testing is quick, and it catches exactly the kind of prefix mistake described above.
- Read the live file. Open your robots.txt in a browser or with
curl. Confirm it is the file you think it is, served with a 200 status, not a staging copy or an HTML error page. - List your money URLs. Write down the homepage, every service or product page, the blog index and three articles. Check each against every
Disallowline, character by character, as a prefix. This is the check that catches a rule like/b. - Use the search engines' tools. Google Search Console reports how it fetched robots.txt, and Bing Webmaster Tools shows crawl status per URL. An important page stuck at "discovered but not crawled", as our business page was from 11 August 2026, is worth investigating straight away. Bing matters here as well; see Does ChatGPT Use Bing?.
- Fetch as the bot. Request a page with each crawler's user-agent string. A 403 or a challenge page means a firewall, not robots.txt, is blocking it.
- Read your server logs. After a few days, look for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the user agents on the pages you care about. No visits to your articles after a change is a sign something is still in the way.
- Re-test after every deploy. Robots.txt is often regenerated by a CMS or framework, so a file that was correct last month can change without anyone editing it by hand.
Will allowing AI crawlers get my business named in answers?
No, not on its own. Allowing crawlers removes a barrier; it does not create a reason to be cited. After we fixed our rule, our own day-0 measurement on 21 October 2026 put 40 questions to ChatGPT, Claude, Gemini and Perplexity three times each: we were named in 0 of 160 answers and none of our pages was cited.
Other things we have measured point the same way. On 19 October 2026, assistants asked for the exact titles of 18 client articles, each older than three weeks, found few of them: ChatGPT 1 of 18, Claude 7 of 18, even though Google had indexed 29 of 30 articles on one of those sites. A site with complete markup but only 75 URLs was absent from answers on its own article's topic. And in a 23 September 2026 test of 20 German business questions, the model answered 10 without searching at all.
What gets a business named is content that answers the questions buyers ask, on pages assistants can reach. That is the work ItGrows does on clients' own domains, alongside fixing crawler access and indexing, and measuring weekly how often ChatGPT, Claude, Gemini and Perplexity name the client. Nobody can guarantee an assistant will name a business. To see where you stand today, the free check at itgrows.ai shows in about 30 seconds whether ChatGPT and Perplexity name you for five of your buyers' questions.
Frequently Asked Questions
Do I need to add GPTBot to robots.txt to allow it?
No. A crawler with no group of its own follows the User-agent: * group, and if that group does not block it, it is allowed. An explicit group is useful for clarity, but then you must repeat your private paths inside it.
Can I allow ChatGPT search but block training?
Yes. OpenAI treats OAI-SearchBot and GPTBot as independent settings, so you can allow OAI-SearchBot and disallow GPTBot. Note that user-initiated fetches by ChatGPT-User may not follow robots.txt at all, according to OpenAI's documentation.
Does blocking Google-Extended hurt my Google rankings?
Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It does control use of your content for Gemini training and grounding, so blocking it is a decision about Gemini, not about Search.
How quickly do crawlers notice a robots.txt change?
Crawlers re-read robots.txt periodically rather than instantly, so expect a delay before behaviour changes. Watch your server logs and Bing Webmaster Tools for the pages you reopened, and resubmit your sitemap to speed up rediscovery.