On 21 October 2026 we put 40 questions about getting a business into AI answers to ChatGPT, Claude, Gemini and Perplexity. ItGrows was named in 0 of 160 answers and none of our pages was cited. The main reason was our own mistake: a one-line robots.txt rule had closed our blog and business pages to every crawler for ten weeks, and the assistants cited the blogs of GEO agencies and tool vendors instead.
Below is what we measured, what went wrong, who got cited in our place, and what we are changing. Each step is something you can check on your own site in an afternoon.
What exactly did we measure?
We run this measurement for clients every week, so we ran it on ourselves. We wrote 20 English and 20 German questions a buyer might ask, such as how to get a company recommended by ChatGPT, or which agency helps with generative engine optimization. Each question went to ChatGPT, Claude, Gemini and Perplexity, three times each.
The three asks matter. Asked the same question three times, an assistant often names different businesses. We count a business as named for a question if any of the three asks names it. That turns 40 questions times 4 assistants into 160 answers, and we compare weeks only on the same questions with the same number of asks.
The result on day 0: named in 0 of 160. Cited in 0 of 160. No partial credit, no near misses.
This kind of measurement is cheap. With API prices of October 2026, one question asked once with web search costs us about $0.069 with Claude, $0.010 with ChatGPT, $0.009 with Gemini and $0.0013 with Perplexity. A report of 20 questions across 4 assistants with 3 asks costs about $5.40, so our 40-question run cost roughly twice that.
Why were we invisible? The robots.txt prefix mistake
We wanted to hide one redirect page, /b, from crawlers. So this line went into robots.txt:
Disallow: /b
Robots.txt rules match prefixes. That line does not mean "the page called /b". It means "every path that starts with /b". On itgrows.ai that included /blog, /business and every article in our business blog. From 11 August to 21 October 2026, all of it was closed to all crawlers.
The mistake stays hidden because browsers ignore robots.txt. The pages load normally for visitors and for anyone checking by hand. Only the crawlers are turned away.
The symptoms were there. Bing Webmaster Tools had listed our business page as "discovered but not crawled" since 11 August 2026. Google had none of the business pages indexed. Google states that a page must first be indexed and eligible to be shown in regular Search before it can appear in its generative AI features. The same logic holds for any assistant that searches the web before answering: a page nobody may crawl is a page nobody can cite.
On 21 October 2026 we changed the rule to:
Disallow: /b$
The $ marks the end of the path, so the rule now matches /b and nothing that merely starts with it. For the longer version of how these rules behave, see how to tell whether your website is blocking AI crawlers and how to allow GPTBot in robots.txt.
Who do the assistants cite instead?
The same day-0 measurement recorded every site the assistants cited. Most cited sources were the blogs of GEO agencies and tool vendors themselves: firms that publish answers to these questions on their own domains.
| English answers | German answers | |
|---|---|---|
| Most cited | firstpagesage.com and llmpulse.ai (8 answers each) | geoagentur.de (6 answers) |
| Next | help.openai.com (7) | help.openai.com (5) |
| Then | blog.hubspot.com, ahrefs.com, searchengineland.com (6 each) | seranking.com, geoagenturen.de, getsichtbar.com, blog.hubspot.com, llmpulse.ai (4 each) |
When the English answers named tools, Profound appeared in 12 answers, Semrush in 9, Otterly.AI and Peec AI in 8 each, and Ahrefs in 7. For "which agency" questions, the English answers included iPullRank, First Page Sage, Go Fish Digital, Siege Media and Directive; the German answers included Aufgesang, Claneo, Suchhelden and geoagentur.de.
Two patterns are worth taking to your own market. First, an official help page (help.openai.com) sits near the top in both languages. That fits a separate test we ran on 23 September 2026 with 20 German business questions: the model answered 10 without searching at all and cited nobody, and where it searched, it cited primary sources such as laws, ministries, the OECD and large accounting firms on 9 of 10. Second, in that test firms were named only on "who should I hire" or "how do I choose" questions, and two runs of the same question named two different sets of firms.
We saw the same shape outside our field. On 20 October 2026, for an orthopaedic practice in Bonn, the four assistants named the practice in 12 of 80 answers to 20 patient questions. Competing practices' own websites were the most cited sources, the directory jameda.de was cited in 12 answers, and questions about costs, risks and treatments named no practice at all. The sites that win are the ones that published crawlable answers on their own domain, which is exactly what our robots.txt rule had made impossible for us.
Was the robots.txt rule the only reason?
No, and it would be convenient to pretend otherwise. Opening the door is the entry ticket, not the result.
Our business blog is young and small. We have seen what that means on other sites. One site with complete markup (server rendering, Article, FAQPage and Organization JSON-LD, a sitemap) but only 75 URLs was still absent from answers on its own article's exact topic. A site with about 190,000 crawlable URLs was crawled constantly. Surface beats markup.
Indexing is not findability either. On 19 October 2026 we asked assistants for the exact titles of 18 articles, each older than three weeks, published on three client sites. ChatGPT found 1 of 18; Claude found 7 of 18. Google had indexed 29 of 30 articles on one of those sites. Being in Google's index did not mean an assistant could find the page.
Content itself still counts, and it is the part we control. The Princeton GEO paper presented at KDD 2024 reports that adding citations, quotations and statistics to content can raise a page's visibility in generative-engine answers by up to 40%.
What are we changing?
Here is the plan, with no promise attached to any step.
- The rule. Changed from
/bto/b$on 21 October 2026. That is the only fix already made. - Indexing. The next work is the same crawler access and indexing work ItGrows does for clients: sitemaps, Bing and Google indexing, IndexNow. Google says site owners can request recrawling after changes, but that crawling can take anywhere from several days to several months. We do not expect next week's numbers to move. Why Bing matters for ChatGPT is covered in does ChatGPT use Bing.
- Answers on our own domain. Articles that answer the 40 questions directly, with our own measured data. This article is one of them. Google's 2026 guide on generative AI in Search repeats that traditional SEO practices remain the foundation, which matches what we see.
- Measuring against day 0. Same 40 questions, same four assistants, three asks each. We track whether we are named, which of our pages are cited, and who is cited instead.
- Watching the right signal. In about two weeks of logs from three of our course websites, human visitors arriving from AI assistants totalled one, from Copilot. Assistants mostly answer in place. The more useful signal is live retrieval agents (ChatGPT-User, OAI-SearchBot, Perplexity-User), which fetch a page while an assistant answers a real person. In the same fortnight, a client blog of a few dozen articles received 56 such fetches across 24 different articles.
We know the route can work because we watch it weekly. In our measurement of 20 October 2026, Perplexity cited an article we had published on a client's site, a German company-formation advisory. Clients also report results to us: FreightSizer says it got 2 new B2B customers in one month, both from ChatGPT; MagiScan reports hundreds of installs a month from ChatGPT, Claude and Gemini; appslift reports 2 to 5 leads a month from AI assistants. None of that guarantees anything for us, or for you.
How do you check the same things on your own site?
Our mistake was not exotic. It was one line. Check these five things:
- Read your robots.txt. Open yourdomain.com/robots.txt. For every
Disallow, ask which other paths start with the same characters. - Look at indexing status. In Bing Webmaster Tools and Google Search Console, find your most important pages. "Discovered but not crawled" for weeks is a warning sign.
- Read your server logs. Are AI crawlers getting normal responses on content pages? When they are allowed, they come in volume: on an edtech platform we work with, pickaclass.com, the logs from 28 July to 11 August 2026 show 19,272,125 requests from ClaudeBot, 2,744,985 from GPTBot and 52,292 from Googlebot.
- Ask the assistants your buyers' questions. Three times each, in every assistant your buyers use. Write down who is named and which sites are cited.
- Study who wins. The cited sources tell you what kind of page earns the answer. If it is competitors' own sites, you need answers on your domain; if it is a directory, you need to be listed there.
If competitors keep appearing where you do not, why ChatGPT names your competitors but not you goes through the usual causes. For a quick start, the free check at itgrows.ai generates five buyer questions from your homepage and shows in about 30 seconds whether ChatGPT and Perplexity name you.
Frequently Asked Questions
How can a single robots.txt line block a whole blog?
Robots.txt rules match prefixes, so Disallow: /b blocks every path that begins with /b, including /blog and /business. Adding $ at the end, as in /b$, limits the rule to that exact path. Browsers ignore robots.txt, which is why such mistakes go unnoticed.
Will fixing robots.txt get us named by AI assistants?
Not by itself. It makes your pages eligible to be crawled and cited, but crawling can take days to months, and the assistants still need pages that answer the question better than the sources they cite today. Nobody can guarantee an assistant will name a business.
Why did ItGrows publish its own zero result?
Because the mistake is common and easy to check, and because we measure clients the same way we measured ourselves. A day-0 number is only useful if it is honest; later weeks are compared against it on the same questions and the same number of asks.
What does ItGrows do about this for clients?
It publishes articles on the client's own domain that answer buyers' questions, fixes crawler access and indexing (robots.txt, sitemaps, Bing and Google indexing, IndexNow), and every week asks ChatGPT, Claude, Gemini and Perplexity the client's buyer questions three times each. It costs $499 a month (plus a one-time $499 setup, free with 3 months or a year), and makes no promise that any assistant will name you.