How to A/B Test Two Chatbot Scripts (Simple 2026 Guide)

By Marcus Reed · Updated 2026-11-21 · reviewed before publication

Most small business owners write one chatbot script, turn it on, and hope. That is a mistake. A greeting that feels friendly to you might read as pushy to your customers. A question that seems clear might confuse the people who see it.

A/B testing fixes that. You show two versions of a script to different visitors, then keep the one that works. It sounds technical, but the basic version fits in an afternoon.

This guide covers the whole process: what to test, how to split your traffic, how to read the numbers, and which tools make it manageable. The focus is on three platforms (Tidio, Crisp and Chatbase) because each one is priced for small teams. Prices and limits below come from the vendors' public pricing pages and documentation, not from paid testing.

Our top picks for this guide
TD
Tidio ★★★★★
from $24.17/mo billed yearly (Starter, 100 billable conversations) · Live chat + Lyro AI chatbot for small businesses
Review
CR
Crisp ★★★★★
from $45/workspace/mo (Mini, 4 seats) · Affordable all-in-one inbox: chat, chatbot, emails
Review
CB
Chatbase ★★★★★
from $40/mo (Hobby, 700 message credits) · Train a ChatGPT-style bot on your own content
Review

What A/B testing a chatbot script actually means

A/B testing means running two versions of the same thing at the same time and letting visitors decide which one wins. Version A is your current script. Version B is the change you want to try: a different greeting, a shorter menu, a softer close.

The key phrase is "at the same time." If you run version A for a week and version B the next, you learn little. A sale, a holiday, or a random slow Tuesday skews the comparison. Running both at once cancels those outside factors.

You do not need a large audience to start, but you do need a clear goal before you begin.

Decide what "better" means before you touch anything

This is the step most people skip, and it is why many tests end without an answer. "Better" is not a feeling. It is a number.

Pick one goal per test. Common ones for small businesses:

  • **Answer rate** — the share of chats where the bot resolves the question without a human.
  • **Lead capture** — how many visitors give you an email or phone number.
  • **Handoff rate** — how often the bot passes someone to a real person. Lower is usually better, unless the bot is pushing people away.
  • **Conversation length** — sometimes longer is good (people are engaged), sometimes shorter is good (they got answers fast).

Write the goal down before you launch. If you change it halfway, you will talk yourself into whatever result appears.

Set up two versions in your chatbot tool

Here is where platform choice matters. Some tools offer a built-in split. Others leave the mechanics to you.

**Tidio** is a common starting point for small e-commerce stores. The vendor describes a working live chat in under 10 minutes, and its Lyro AI answers questions from your FAQ automatically. The free plan gives you 50 billable conversations a month plus 50 Lyro AI chats as a one-off, enough for a first test. The catch: Lyro AI credits run out quickly on cheaper plans, and advanced analytics sit on higher tiers, so plan to track results in a spreadsheet.

**Crisp** is the budget pick. Pricing is flat per workspace rather than per seat, with 4 seats included on the Mini plan, so confirm that the people running your test fit inside that cap. Chatbot, shared inbox and campaigns come in one place. The trade-off is lighter AI than Tidio's and a free plan with zero AI credits. For a scripted bot test, though, it does the job.

**Chatbase** works differently. You upload docs or a website and it builds a ChatGPT-style bot trained on that content, which makes it strong for testing knowledge scripts: different ways of phrasing an answer, different fallback lines. There is no free plan anymore, so testing means the 7-day trial or Hobby at $40/mo. Helpdesk with human handover requires Basic at $150/mo, so budget for that if handoff is part of the test.

Whichever tool you use, build both scripts fully before launch, and resist tweaking version B mid-test. That contaminates the data.

Split your traffic fairly

The cleanest split is 50/50. Half your visitors see script A, half see script B. Tools with a built-in A/B feature usually handle this for you.

Without a native split, you have two options. The first is random assignment per visitor. The second is a time-based split, alternating hours or days. Time-based is messier but workable when traffic is steady.

One thing to watch: make sure the split is random and not tied to something like device type. If all mobile visitors get script A and all desktop visitors get script B, you are testing devices rather than scripts.

The same failure has a second form, an assignment rule that quietly favors one group. If a tool only shows the new script to returning visitors, the "winner" is whatever returning customers happened to see. Read how your tool assigns variants before you trust the result.

Run it long enough to trust the numbers

There is no magic number of days, but there is a firm rule: do not stop early because one version is "winning." Early results bounce around. A script can look excellent on day two and poor by day ten.

A practical approach is to fix the calendar first, with two full weeks as a common placeholder, and let the conversation count grow inside that window. If traffic is thin, extend the window rather than guessing from a handful of chats.

Also, avoid unusual periods. The week before a big sale, or the week after a holiday, will not show normal behavior. Pick a boring stretch.

Read the results without fooling yourself

When the test ends, compare your one chosen metric between A and B. Ignore everything else for now. If A captured more emails, that is your answer, even if B had longer conversations.

A few honest cautions:

  • **Small differences are noise.** A gap of a couple of percentage points is probably not real. Look for a difference you would also notice in the transcripts before you change anything permanently.
  • **Watch the handoff rate.** A script that boosts leads but floods your inbox with confused customers is not a win.
  • **Read the actual chats.** Numbers tell you what happened; the transcripts tell you why. Skim a sample from each version and you will spot patterns no dashboard shows.

If B wins, make it your default and test again, maybe B against a C. If A wins, you learned something for free and can try a different change next month.

A simple testing schedule that works

You do not need to test constantly. One clean test per month is plenty for most small businesses.

  • **Week 1:** Pick your goal and write both scripts.
  • **Week 2-3:** Run the test with a 50/50 split.
  • **Week 4:** Read results, pick a winner, and note what you learned.

Keep a simple log with the date, what you changed, and what happened. After six months you will have a playbook written for your own business rather than borrowed advice.

Frequently asked questions

How many conversations do I need before I can trust an A/B test?

A rule of thumb is two full weeks at a settled traffic level, with enough conversations that a couple of percentage points would not swing the result. If traffic is very low, run longer rather than guessing. The number matters less than keeping both versions live for the same period.

Can I A/B test chatbot scripts on a free plan?

Sometimes. Tidio's free plan includes 50 billable conversations a month plus 50 Lyro AI chats as a one-off, which is enough for a small first test. Crisp has a free plan with 2 seats but no AI credits. Chatbase no longer has a free tier, so you would need the 7-day trial or the $40/mo Hobby plan.

What should I test first, the greeting or the whole script?

Start with the greeting or the first question. It is the highest-impact change and the easiest to isolate. Testing a whole new script at once makes it impossible to know which part helped. Change one thing, measure, then move to the next.

What if my chatbot tool has no built-in A/B testing?

You can still test. Build both scripts as separate bots or flows, then alternate them by hour, day, or random assignment, and track results in a spreadsheet. It is less clean than a native split, but it works. Just make sure the assignment is truly random and not tied to visitor type.

Ready to pick one?

TD
Tidio ★★★★★
from $24.17/mo billed yearly (Starter, 100 billable conversations) · Live chat + Lyro AI chatbot for small businesses
Review
CR
Crisp ★★★★★
from $45/workspace/mo (Mini, 4 seats) · Affordable all-in-one inbox: chat, chatbot, emails
Review
CB
Chatbase ★★★★★
from $40/mo (Hobby, 700 message credits) · Train a ChatGPT-style bot on your own content
Review
MR
About the author

Marcus Reed — Marcus Reed is the senior editor at ChatbotRadar. He tracks live chat and AI chatbot tools for small teams, verifying pricing on vendor sites and trialling tools on their free plans. How he tests · Report an error

Disclosure: this article contains affiliate links. If you buy through them we may earn a commission at no extra cost to you. Pricing and features are re-checked regularly but can change — always confirm on the vendor's site.