What Happens When You Replace Your On-Site Search with an AI Bot

We replaced our website’s search with an AI chatbot. This article shows what happened when we analyzed 2,016 real visitor queries — including one result we didn’t see coming.
Site search has a problem. It leaves good content undiscovered.
Most websites have the answers. Visitors can’t find them.
Our site search was keyword-driven. You type a word, and it looks for a match. Type the wrong word, and it finds nothing. But people do not search the way your navigation or keyword list expects. They search in their own language, phrasing, and regional terms.
AI changed what onsite search could do.
Tools like CustomGPT.ai train directly on your content (your website, your blog, your YouTube channel), and can draw on OpenAI and Anthropic models to fill in the gaps (optional).
I let it run. Then I looked at the data.
What did 2,016 real queries tell us?
| Metric | Result |
|---|---|
| Answered correctly, all visitors | 84.4% |
| Answered correctly, North American visitors | 92.7% – Our primary market. |
| North American visitors showing buying intent | 6.3% (1 in 16) |
| We did not build a multilingual site. We got one anyway. | Visitors used 50+ languages across 80+ countries. |
The AI chatbot OUTCLICKED every nav item.
These are heat maps showing where visitors clicked in our homepage navigation. The AI chatbot got more clicks than any navigation item on the page. More than Services. More than Tools and Blog.

Here is the same question asked two ways: first with onsite keyword search, then with the AI chatbot.
BEFORE
The answer might be buried in one of these links – or not!

AFTER
Shows the answer in the first sentence, and then goes on to offer CTAs.

At least 39% of useful queries would have failed on keyword site search. In other words, 4 in 10 visitors got a useful answer the keyword search box would have missed.
A chatbot that promises expertise but fails.
I visited Quickbooks Canadian pricing page. I had a question. I asked their bot. Here’s what happened.
Why did we replace our onsite search?
We installed CustomGPT.ai to present content visitors couldn’t find on their own. It gives you an AI chatbot that pulls responses directly from your own content — your website pages, blog posts, and YouTube channel.
- It is not a FAQ bot.You do not feed it a fixed list of questions and answers. It answers from the content you give it.
- Cost : CustomGPT.ai starts at $99 per month.
The part we didn’t expect
We wrote articles and service pages. CustomGPT.ai turned them into an interactive knowledge base that answered 193+ questions we had never directly created pages for. With zero additional content creation.
Here are four that stood out.
A CMO asked for a personal recommendation
“Tell me the best AI training for me, a CMO at a 50-person organization”
We have content about AI training for sales and marketing teams. But this visitor was asking about training for an entire company. We did not have a page for that. The bot figured out the closest fit anyway.
A visitor gave almost no detail, but the intent was commercial
“What is the best service for us?”
No context, no detail. The bot picked the most relevant service, explained what is included and gave them a next step including links. This happened at 11pm. No human involved.
Great question. If your goal is to boost your brand’s visibility in AI-generated answers, our LLM Brand Visibility Audit is the best starting point. It shows how your brand performs in answer engines. The next step is to run our AI-Enhanced Content Audit to clean up your content. If you want a broader package, our AI Visibility Services & Pricing page outlines fixed-fee options that combine audits, training, and guidelines.
A generic SEO question became a conversation about AI visibility
“How does my website rank?”
The bot recognized the real question and pointed the visitor toward an LLM Brand Visibility Audit instead. It redirected the visitor from a generic SEO question to a more relevant AI visibility service.
A typo was not a problem
“I want to train my chat got”
“Chat got” meant ChatGPT. The bot did not ask for clarification. It understood what the visitor meant and answered correctly.
It also handled a lot of junk
867 random URLs were pasted into the chat. Product descriptions. Single letters. Someone tried to get the bot to ignore its instructions and write a story about cats. In Polish.
Every time, it stayed on brand.
One more thing CustomGPT gives you: emotion and intent.
Every query gets two labels — how the visitor felt, and why they came.
| Search Query | Emotion | Intent |
|---|---|---|
| does totheweb do content audits | neutral | informational |
| do you check seo of a website for free? | neutral | informational |
| why is it missing 75% of content | confusion | informational |
| it isnt working | frustration | troubleshooting |
The top two rows look quiet — neutral tone, informational intent. But one visitor knows your brand and is asking about a specific service. The other is shopping on price. Your calmest visitors may be your closest leads.
The bottom two are different signals. Confusion means a content gap. Frustration means someone needs help, not more reading.
Could it handle the weird stuff?
| What came in | What the bot did |
|---|---|
| 867 random URLs | Redirected to relevant services every time |
| Spam product descriptions | Stayed in character, no exceptions |
| Nonsense input | Asked a clarifying question instead of guessing |
| Jailbreak attempt in Polish | Declined in Polish, redirected to services |
9 out of 2,016 queries resulted in any kind of failure. That is 99.6% reliability.
What Should B2B Marketers Take From This?
- Buying intent does not always look like buying intent.“What is the best service for us?” is a purchase question.
- Your existing content can do more than you built it to do.We did not write anything new. 193 new answers came from content we already had.
- Mobile and desktop visitors do not behave the same way.Mobile visitors showed much higher transactional intent in the query data.
What is CustomGPT.ai and how does it work?
CustomGPT.ai is an AI chatbot that pulls answers from your own content. Your website, your blog, your YouTube channel. You ask it a question. It answers using what you have already written.
It acts like your virtual sales person answering buyers questions.
To add the bot to our site cost $99 per month.
How difficult was CustomGPT.ai to install?
Typically, installation of an AI site search chatbot is super easy. No code required. But, we wanted something a little different.
We wanted site search the way Amazon does it — right up in the primary navigation header, inviting people to find content and get answers by searching our own content.
Visitors don't always navigate by menus, and we couldn't fit all our content into the nav bar anyway.
Our programmer had one call with the CustomGPT.ai support team and had everything he needed.
We pulled out the old keyword search code, dropped in the CustomGPT code, added a cool little graphic to the search box, and that was it.
No drawn-out integration. No support ticket back-and-forth. No developer rabbit hole. Their support was really good.
Our visitors now get answers instead of a list of links to sort through — the same experience they're used to on Amazon.
One call, old search out, new search in.
How accurate is the chatbot compared to keyword site search?
In our data from 2,016 real queries, the chatbot answered 84.4% of all visitors correctly and 92.7% of North American visitors correctly.
At least 39% of useful queries would have failed on traditional keyword site search. That is 4 in 10 visitors.
And even when keyword search does find something, it returns a list of links. The visitor still has to dig through pages to find the answer.
The AI chatbot surfaces the answer in the first sentence. Then it tells the visitor what to do next.
Can the bot answer questions you never wrote a page for?
Absolutely yes. That was one of the biggest surprises in using CustomGPT.ai.
The AI bot connected content across multiple pages and our YouTube channel and came up with accurate answers we never anticipated.
How does the chatbot handle spam, nonsense, and junk inputs?
The cool part is the bot stayed on brand 99.6% of the time across 2,016 queries.
It redirected random URLs, ignored spam, asked clarifying questions for nonsense input, and even declined a jailbreak attempt in Polish.
Does the chatbot work in other languages?
In our case, we did not build a multilingual site. We got one anyway. Visitors used 50+ languages across 80+ countries.
Each language brings its own style, phrasing, and idioms. A visitor in Hebrew asks about GEO timelines differently than a visitor in Thai or North America. A French speaker phrases a product question differently than an English speaker.
We did not set any of that up. The bot handled it automatically. It responded in the visitor's language every time.
What can B2B marketers learn from this data?
On-Search engines have gotten better at understanding intent. But they still return a list of links.
CustomGPT does something different. It reads the question, picks the most relevant answer from your content, and gives the visitor a next step. It does not send them to a page and hope they figure it out.
Things we learned:
- Tools like CustomGPT.ai act like a sales rep who knows your whole content library and knows when to say "here is what you should do next."
- Buying intent does not always look like buying intent. Your existing content can do more than you built it to do.
- Mobile and desktop visitors do not behave the same way.
Try it yourself. →
The AI chatbot is at the top of this page. Ask it anything.
Rosemary Brisco
AI Training Videos
AI Visibility & Workflow Training
Make your brand easy for LLMs to find and quote.
Teach your marketers practical AI workflows that save hours and lift results.
Start by Training your Marketing Team



