Back to Expert Advice

What Happens When You Replace Your On-Site Search with an AI Bot

Featured image for “What Happens When You Replace Your On-Site Search with an AI Bot”
We replaced our website’s search with an AI chatbot. This article shows what happened when we analyzed 2,016 real visitor queries — including one result we didn’t see coming.

Site search has a problem. It leaves good content undiscovered.

Most websites have the answers. Visitors can’t find them.

Our site search was keyword-driven. You type a word, and it looks for a match. Type the wrong word, and it finds nothing. But people do not search the way your navigation or keyword list expects. They search in their own language, phrasing, and regional terms.

AI changed what onsite search could do.
Tools like CustomGPT.ai train directly on your content (your website, your blog, your YouTube channel), and can draw on OpenAI and Anthropic models to fill in the gaps (optional).

I let it run. Then I looked at the data.

What did 2,016 real queries tell us?

MetricResult
Answered correctly, all visitors84.4%
Answered correctly, North American visitors92.7% – Our primary market.
North American visitors showing buying intent6.3% (1 in 16)
We did not build a multilingual site. We got one anyway.Visitors used 50+ languages across 80+ countries.

The AI chatbot OUTCLICKED every nav item.

These are heat maps showing where visitors clicked in our homepage navigation. The AI chatbot got more clicks than any navigation item on the page. More than Services. More than Tools and Blog.

Before and After screenshot of the heat maps

Here is the same question asked two ways: first with onsite keyword search, then with the AI chatbot.

BEFORE

The answer might be buried in one of these links – or not!
BEFORE example of the question answered by On-Site Search

AFTER

Shows the answer in the first sentence, and then goes on to offer CTAs.
AFTER EXAMPLE of the same question answered by my AI chatbot

At least 39% of useful queries would have failed on keyword site search. In other words, 4 in 10 visitors got a useful answer the keyword search box would have missed.

A chatbot that promises expertise but fails.

I visited Quickbooks Canadian pricing page. I had a question. I asked their bot. Here’s what happened.

An example of Intuit's chatbot "expert" that couldn't answer the question.

Why did we replace our onsite search?

We installed CustomGPT.ai to present content visitors couldn’t find on their own. It gives you an AI chatbot that pulls responses directly from your own content — your website pages, blog posts, and YouTube channel.

  • It is not a FAQ bot.You do not feed it a fixed list of questions and answers. It answers from the content you give it.
  • Cost : CustomGPT.ai starts at $99 per month.

The part we didn’t expect

We wrote articles and service pages. CustomGPT.ai turned them into an interactive knowledge base that answered 193+ questions we had never directly created pages for. With zero additional content creation.

Here are four that stood out.

A CMO asked for a personal recommendation

“Tell me the best AI training for me, a CMO at a 50-person organization”

We have content about AI training for sales and marketing teams. But this visitor was asking about training for an entire company. We did not have a page for that. The bot figured out the closest fit anyway.

A visitor gave almost no detail, but the intent was commercial

“What is the best service for us?”

No context, no detail. The bot picked the most relevant service, explained what is included and gave them a next step including links. This happened at 11pm. No human involved.

Bot response

Great question. If your goal is to boost your brand’s visibility in AI-generated answers, our LLM Brand Visibility Audit is the best starting point.  It shows how your brand performs in answer engines. The next step is to run our AI-Enhanced Content Audit to clean up your content. If you want a broader package, our AI Visibility Services & Pricing page outlines fixed-fee options that combine audits, training, and guidelines.

A generic SEO question became a conversation about AI visibility

“How does my website rank?”

The bot recognized the real question and pointed the visitor toward an LLM Brand Visibility Audit instead. It redirected the visitor from a generic SEO question to a more relevant AI visibility service.

A typo was not a problem

“I want to train my chat got”

“Chat got” meant ChatGPT. The bot did not ask for clarification. It understood what the visitor meant and answered correctly.

It also handled a lot of junk

867 random URLs were pasted into the chat. Product descriptions. Single letters. Someone tried to get the bot to ignore its instructions and write a story about cats. In Polish.

Every time, it stayed on brand.

One more thing CustomGPT gives you: emotion and intent.

Every query gets two labels — how the visitor felt, and why they came.

Search QueryEmotionIntent
does totheweb do content auditsneutralinformational
do you check seo of a website for free?neutralinformational
why is it missing 75% of contentconfusioninformational
it isnt workingfrustrationtroubleshooting

The top two rows look quiet — neutral tone, informational intent. But one visitor knows your brand and is asking about a specific service. The other is shopping on price. Your calmest visitors may be your closest leads.

The bottom two are different signals. Confusion means a content gap. Frustration means someone needs help, not more reading.

Could it handle the weird stuff?

What came inWhat the bot did
867 random URLsRedirected to relevant services every time
Spam product descriptionsStayed in character, no exceptions
Nonsense inputAsked a clarifying question instead of guessing
Jailbreak attempt in PolishDeclined in Polish, redirected to services

9 out of 2,016 queries resulted in any kind of failure. That is 99.6% reliability.

What Should B2B Marketers Take From This?

  1. Buying intent does not always look like buying intent.
    “What is the best service for us?” is a purchase question.
  2. Your existing content can do more than you built it to do.
    We did not write anything new. 193 new answers came from content we already had.
  3. Mobile and desktop visitors do not behave the same way.
    Mobile visitors showed much higher transactional intent in the query data.

CustomGPT.ai is an AI chatbot that pulls answers from your own content. Your website, your blog, your YouTube channel. You ask it a question. It answers using what you have already written.

It acts like your virtual sales person answering buyers questions.

To add the bot to our site cost $99 per month.

Typically, installation of an AI site search chatbot is super easy. No code required. But, we wanted something a little different.

We wanted site search the way Amazon does it — right up in the primary navigation header, inviting people to find content and get answers by searching our own content.

Visitors don't always navigate by menus, and we couldn't fit all our content into the nav bar anyway.

Our programmer had one call with the CustomGPT.ai support team and had everything he needed.

We pulled out the old keyword search code, dropped in the CustomGPT code, added a cool little graphic to the search box, and that was it.

No drawn-out integration. No support ticket back-and-forth. No developer rabbit hole.  Their support was really good.

Our visitors now get answers instead of a list of links to sort through — the same experience they're used to on Amazon.

One call, old search out, new search in.

In our data from 2,016 real queries, the chatbot answered 84.4% of all visitors correctly and 92.7% of North American visitors correctly.

At least 39% of useful queries would have failed on traditional keyword site search. That is 4 in 10 visitors. 

And even when keyword search does find something, it returns a list of links. The visitor still has to dig through pages to find the answer.

The AI chatbot surfaces the answer in the first sentence. Then it tells the visitor what to do next.

Absolutely yes.  That was one of the biggest surprises in using CustomGPT.ai.

The AI bot connected content across multiple pages and our YouTube channel and came up with accurate answers we never anticipated.

The cool part is the bot stayed on brand 99.6% of the time across 2,016 queries.

It redirected random URLs, ignored spam, asked clarifying questions for nonsense input, and even declined a jailbreak attempt in Polish.

In our case, we did not build a multilingual site. We got one anyway. Visitors used 50+ languages across 80+ countries.

Each language brings its own style, phrasing, and idioms. A visitor in Hebrew asks about GEO timelines differently than a visitor in Thai or North America. A French speaker phrases a product question differently than an English speaker.

We did not set any of that up. The bot handled it automatically. It responded in the visitor's language every time.

On-Search engines have gotten better at understanding intent. But they still return a list of links.

CustomGPT does something different. It reads the question, picks the most relevant answer from your content, and gives the visitor a next step. It does not send them to a page and hope they figure it out.

Things we learned:

  • Tools like CustomGPT.ai act like a sales rep who knows your whole content library and knows when to say "here is what you should do next."
  • Buying intent does not always look like buying intent. Your existing content can do more than you built it to do.
  • Mobile and desktop visitors do not behave the same way.

Try it yourself. →

The AI chatbot is at the top of this page. Ask it anything.

How helpful was this content?

Click on a star below to rate our tool out of 5 stars

Average rating 5 / 5. Vote count: 45

No votes so far! Be the first to rate this.

AI Visibility & Workflow Training

Make your brand easy for LLMs to find and quote.

Teach your marketers practical AI workflows that save hours and lift results.

Start by Training your Marketing Team

Read the CMO roadmap for AI visibility

Explore AI visibility services and pricing