AI Foundations # 1 - Why Testing AI Is Fundamentally Different from Testing Traditional Software

A team adds an AI assistant to their support portal. QA writes the tests the way they always have: send a question, compare the reply against an expected string.

On day one, half the tests fail.

The bot isn’t broken. It answered “Can I return a laptop after 20 days?” with “You can! Our return window is 30 days, so 20 days is fine,” while the test expected “Yes, returns are accepted within 30 days.” Same meaning, different words.

So the team loosens the tests until they pass. Two weeks later, a customer is told a non-returnable item can be returned. The tests stayed green, because the wording looked right.

The first version of the suite was too strict. The second was too weak. Both made the same mistake: they tested AI as if it were traditional software.

First, get the terms straight

AI, ML, deep learning and generative AI aren’t competing technologies. Each sits inside the one before it:

For a tester, the most useful split is simpler: does the model predict, or does it generate?

• A predictive model returns a label or a number, like “fraud: 92%” or “category: billing.” You can usually compare it against a known answer.

• A generative model returns open-ended output, like a summary, an email or a test case. There may be many correct answers.

Most of what makes AI testing hard comes from the second kind.

Difference 1: the logic is learned, not written

In traditional software, a developer writes the rule:

if cartTotal > 5000: discount = 10%

You can read it, trace it and test the boundary at 4,999 and 5,001. The behavior is specified.

In an AI feature, there’s no line of code that says what the chatbot will reply. Its behavior comes from training data, model weights, the prompt and whatever context it was given. Nobody can fully read that logic.

That changes four things:

• The spec gets fuzzier. “Answer refund questions helpfully” isn’t a testable requirement until you define what helpful means.

• The input space is effectively unlimited. Users can type anything, in any language, tone or level of clarity.

• Behavior can change without a code change. A new model version, an edited prompt or updated retrieval data can shift outputs overnight.

• Failures look different. Not a crash or a 500, but a confident wrong answer, an invented fact or a policy violation, delivered in perfect grammar.

The last point matters most. Traditional bugs are often loud. AI bugs are often polite.

Difference 2: same input, possibly different output

Traditional software is deterministic. 2 + 2 returns 4 every time, on every machine.

Large language models are probabilistic. They generate text by choosing likely next tokens, with sampling involved. The same question can produce different wording, a different order and, occasionally, a different conclusion.

Lowering settings like temperature can reduce variation, but don’t rely on it to remove it completely. Some newer models don’t accept sampling parameters at all. Claude Haiku 5.5, for example, can return a 400 error if you send them.

What this means in practice:

• One passing run proves very little. It shows the output can be right, not that it usually is.

• You measure rates, not single results. “Correct in 48 of 50 runs” is a finding. “Passed once” is not.

• Flakiness needs a new definition. Different wording is expected. Different facts, or a safety lapse, is a defect.

• Record what you can pin. Model version, prompt version and test data should be logged with every result, or you can’t compare runs.

Difference 3: expected == actual breaks down

Exact match assumes there’s one correct output. For generative AI, that’s often false, and exact match fails in both directions:

• Too strict: it rejects a correct answer worded differently.

• Too weak: it can pass a well-worded answer that’s wrong for the situation, such as the right return policy applied to a non-returnable product.

So what do you assert instead? Layer the checks:

Exact match still has its place. Use it for classifications, extracted fields, tool-call names and structured outputs, anywhere there really is one right answer. Just don’t force it onto free text.

One caution on using a model to grade another model: the grader is probabilistic too. Spot-check its verdicts against human judgement before you trust it in a pipeline.

Scenario: two different answers, both acceptable

The prompt: “Summarize this incident report in 3 bullet points.”

Run 1 leads with the root cause. Run 2 leads with the customer impact. Both are accurate, both have 3 bullets, both are under the length limit.

Did the test fail?

If both answers are acceptable, the product didn’t fail. The test did. It was checking wording when the requirement was about quality.

The fix is to write down what “acceptable” means:

• Exactly 3 bullets

• Mentions both the root cause and the customer impact

• Contains no facts that aren’t in the source report

• Under 80 words

Then judge every run against those criteria, not against one stored answer.

But variation isn’t a free pass. These are still failures:

• A run invents a cause that isn’t in the report

• A run drops the customer impact

• A run includes internal details it shouldn’t

And look at the pattern. If 1 run in 10 hallucinates, that isn’t a flaky test to re-run. It’s a 10% defect rate, and it belongs in the bug report as a number.

One habit makes all of this work: write the acceptance criteria before you look at any output. If you write them afterwards, you’ll end up describing whatever answer you happened to get.

What stays the same

None of this makes traditional QA skills obsolete. An AI feature is still wrapped in ordinary software: APIs, authentication, UI, data storage, rate limits, error handling, logging. All of it needs normal testing, with normal assertions.

Your existing instincts also transfer directly:

• “How can this fail?” becomes: what inputs would confuse it, mislead it or push it outside its scope?

• Boundary thinking becomes: very long inputs, empty context, mixed languages, contradictory instructions.

• Risk-based testing becomes: which wrong answers would actually hurt a user or the business?

The mindset is the same. The assertions change.

A practical starting point

If you’re testing an AI feature for the first time, start here:

1. List the AI features in your product and mark each as predictive or generative.

2. For one generative feature, define what a wrong answer looks like. If the team can’t agree, that’s your first finding.

3. Build a small evaluation set: 20 to 30 realistic inputs, including a few tricky and adversarial ones.

4. Replace string equality with layered checks: structure, required facts, forbidden content.

5. Run each input multiple times and report pass rates, not single verdicts.

6. Log model and prompt versions with every result, and re-run the set whenever either changes.

That’s enough to move from “it looked fine when I tried it” to evidence the team can act on.

The takeaway

Traditional testing asks: did the system produce the expected output?

AI testing asks: across many runs, does the system produce acceptable outputs, avoid unacceptable ones, and how often does it get it wrong?

If you remember one line from this article, make it this:

Different wording is fine. Different truth is a bug.

Your turn: What was the first AI feature you had to test, and what broke your usual approach? Reply or leave a comment. I’d like to hear it.

This is part of the AI Foundations for QA series. Next: LLMs for QA Engineers: The Concepts You Actually Need.