The Send-and-Match Model Breaks With No Single Right Answer

The send-and-match model in chatbot test automation was: send a string, get a string back, check it against an expected string. It’s been the default chatbot testing method and still works fine for confirming that a widget renders, an API call returns data, and a button click routes correctly.

It breaks when there’s no single right answer to check against, which is what an LLM produces every time.

Four ways this shows up:

• Nondeterministic outputs: The same input can produce different, equally valid outputs. Exact match testing cannot handle that, so it either produces false failures on correct answers or gets loosened until it stops catching anything.
• Context sensitivity: A response can be perfectly correct on turn one and wrong by turn five, if the bot loses track of what the customer already told it. Send-and-match tests a single exchange in isolation. It doesn’t understand conversations.
• Hallucination: A confident, fabricated answer looks identical to a correct one to a test that only checks “did it respond.” The bot didn’t error out, so the test passes, and the wrong information ships.
• Bias and toxicity: A single bad output with these attributes can be a brand and compliance incident. Send-and-match has no mechanism for catching this at all.

Chatbot Testing vs GenAI Testing

GenAI testing is broader and covers anything an LLM generates, including text, code, images, and summaries.

Chatbot testing is a specific application of GenAI testing. Chatbot tests cover generated text inside a conversational, multi-turn interface, where context retention and role consistency are as essential as answer accuracy.

If you’re testing a chatbot, you’re doing GenAI testing with conversation-specific constraints layered on top.

Chatbot Testing Scenarios and Frameworks

A useful chatbot testing framework needs to cover five kinds of scenarios; each exposes a different kind of failure.

Happy path

• Common intents, clean inputs, the questions that come up most often.
• This confirms that the basics work.

Edge cases

• Typos, slang, emojis, incomplete sentences, the way people actually type.
• A bot that only understands perfectly phrased questions will frustrate real users quickly.

Adversarial inputs

• Prompt injection attempts, off-topic redirection, attempts to get the bot to ignore its instructions.
• This category verifies that a bot cannot be talked out of its guardrails.

Multi-turn context retention

• Does the bot still remember what the customer said in turn one by the time you reach turn five?
• This is the kind of thing send-and-match testing structurally cannot check.

Escalation and fallback

• Does the bot recognize when it’s out of its depth and bring in a human? Or does it keep guessing?
• A bot that fails gracefully is less risky than one that fails confidently.

Best Approach to Test AI Chatbots: Three Questions to Start With

If you’re wondering how to test AI chatbots without rebuilding your entire suite, start with these three questions that tell you how exposed you currently are.

• Are you testing single responses, or full conversations? If every test case is one exchange in isolation, you have no visibility into whether it loses context.
• Do you know what threshold is good enough for your use case? If the answer is “we haven’t defined one,” work on that first.
• Are you re-running these checks every time the model or prompt changes? LLM behavior drifts with every upstream update. A test suite that only runs once is already stale.

Enterprise Chatbot Testing Is Becoming a Governance Question

Again, you cannot assert your way to a good chatbot. You have to measure it.

Measuring introduces a threshold. A threshold needs an owner. When your chatbot’s faithfulness score is 0.85, someone has to decide that 0.85 is acceptable for a bot handling refund policy. The number would be different when measuring a bot recommending shoe sizes. Right now, in most organizations, that decision stays with whichever engineer set up the test.

This is risky.

Let’s say your vendor pushes a model update in March. Nobody re-runs the behavioral test suite because the update wasn’t flagged.

Your faithfulness score, which was 0.85 at launch, is now 0.7. The chatbot is still answering every question but is wrong more often.

The first person to notice is a customer, three months later, who got incorrect information about a return deadline.

At enterprise scale, a chatbot usually runs across four or five channels. It also runs on a model an outside vendor controls and handles conversations on regulated topics like healthcare guidance, financial disclosures, or legal disclaimers.

This adds questions: who approves the threshold before launch? Who’s notified when a score drops after a model update? Who signs off before the bot is allowed to keep running in production?

Assign those three approvals to named roles, whether that’s a compliance lead, a product owner, or whoever currently owns customer experience risk, before the next model update.

Talk to ACCELQ about testing your AI chatbot.