"Is it tested?" used to be a question with a mechanical answer. The suite ran, the suite was green, and green meant the build still did what the last build did. Put a language model in the request path and the same question turns into an opinion.
Somebody in your organization is signing off on releases using a definition of "passing" that stopped describing the system. If you run engineering or QA at a company that has shipped an LLM-backed feature, that somebody is probably you.
The instinct is to go looking for a testing tool that handles AI. There are many, and some of them are good. But no tool will tell you what score is high enough to ship, who is allowed to decide that, or what happens when a borderline result lands on a Friday afternoon. Those are the questions that block a release, and none of them are answered by installing anything.
What follows is the decision structure underneath the tooling: how to build the dataset you test against, how to score outputs that have no single right answer, how to write acceptance criteria a person can actually sign, and how to keep all of it working when the model underneath you changes. The goal is not a green build. The goal is a defensible confidence level, at a known cost, owned by a named person.
Why the test suite stopped working the day the model arrived
Start with the part everybody skips, because it is the part that decides everything downstream: two entirely different strings can both be correct. A support reply that opens with an apology and one that opens with the fix are both good answers. A summary that leads with the decision and one that leads with the context are both faithful.
Equality was never the right check for this kind of output. It only worked before because the outputs it was applied to happened to be enumerable.
Non-determinism is the reason people usually give, and it is true but almost useless as a diagnosis. Four distinct mechanisms sit underneath it, and each one breaks a different assumption your existing tests rely on.
Sampling is the obvious one. Most production configurations generate text by sampling from a probability distribution, so identical inputs produce different outputs across runs. Set temperature to zero and the variance narrows, but it does not vanish, because floating-point non-associativity and batching effects still shift results between runs on the same hardware.
Model versions move underneath you. Providers update weights, deprecate snapshots, and change default routing. Your code did not change, your prompts did not change, and your output quality did.
Prompt sensitivity is worse than most teams expect. Reordering two sentences in a system prompt, or adding a clarifying instruction, can move behaviour on cases that seemed unrelated to the edit. There is no compiler to tell you what you just broke.
Retrieval adds its own variance. If the feature pulls context from an index, then re-embedding the corpus, changing chunk size, or simply adding documents changes which context reaches the model, and therefore what it says.
Some things do stay deterministic. Keep those tests. You can and should still assert on the shape of the response: schema validity, required fields, JSON parseability, tool-call structure, refusal behaviour on prohibited inputs, latency ceilings, and token-cost ceilings. They belong in the ordinary suite and they run on every commit, protecting the plumbing around the model.
What they do not do is tell you whether the output was any good. Before you can build the apparatus that answers that, there is a distinction that decides where the work goes.
Testing the model and testing your product are different jobs
A lot of wasted effort at this stage comes from confusing two kinds of evaluation that have almost nothing to do with each other.
Model-level evaluation asks whether a given model is capable. This is what public benchmarks measure, and it is what model cards report. It is useful for exactly one decision: choosing a candidate model to build on.
System-level evaluation asks whether your feature, with your prompt, your retrieval configuration, your guardrails, your tool definitions, and your users' actual inputs, does the job. IBM's guidance on LLM evaluation draws the same line, separating evaluation of the model from evaluation of the system built around it (IBM, 2025).
The gap between them is large enough to invalidate intuitions. A model that scores well on a reasoning benchmark can fail badly on your task because your prompt is ambiguous, your retrieval returns the wrong document, or your users phrase things in ways the benchmark never contained. The reverse also happens. A mid-tier model with a well-constructed prompt and good retrieval frequently beats a frontier model wired up carelessly.
Most teams over-invest in the first and under-invest in the second, and the reason is not technical. Benchmark comparisons are easy, public, and shareable in a slide. Building an evaluation harness for your own feature is unglamorous work that nobody outside the team will ever see.
Everything after this section is about system-level evaluation, because that is what you ship and that is what your users experience. One more piece of vocabulary has to be cleared out of the way first, because half the material published on this subject is answering a different question.
Testing AI is not the same as using AI in your tests
Two different subjects share almost the same vocabulary, and conflating them is expensive.
AI for QA means using machine learning to make conventional testing cheaper: self-healing element locators, generated test cases from requirements , visual regression detection, flaky-test triage, coverage-gap analysis. The system under test is ordinary software. The AI is in your tooling. The benefit is speed and maintenance cost.
QA of AI means the system under test is itself probabilistic. The AI is in the product. The benefit is knowing whether you can ship.
Different budgets, different skills, different risk profiles. The dangerous position is the common one: a team excellent at the first and holding nothing at all for the second, because the tooling investment creates a feeling of testing maturity that does not transfer.
Most of the search results for phrases like "AI testing" describe the first. This article is about the second.
Evaluation datasets, and why most teams do not have one
An evaluation dataset is a fixed collection of inputs, paired with either expected outputs or grading criteria, that you run your feature against to produce a score. It is the closest thing AI features have to a test suite, and it is the single highest-leverage artifact your team can build.
Almost nobody starts with one. The usual state is a handful of prompts in a Slack thread that somebody tries by hand before a release.
Where the examples come from
Production logs are the best source, by a wide margin. Real inputs contain the phrasings, the typos, the half-formed questions, and the edge cases that nobody sitting in a planning meeting would invent. If the feature is not live yet, the nearest equivalents are support tickets, sales call notes, existing search queries, or whatever corpus of real user language your organization already has.
Then add the cases you know are hard. Every team can name them: the ambiguous request, the request that should be refused, the input in the wrong language, the question whose answer is not in the knowledge base, the adversarial user testing the boundaries. These are worth writing by hand because they are rare in logs and expensive in production.
And add every failure you have already had. A production incident that does not become a permanent row in the evaluation set will happen again.
How many examples you need
Fewer than people fear to get signal, more than people expect to get a defensible threshold.
As a practical range rather than a benchmark: thirty to fifty examples is enough to catch gross regressions and to tell whether a prompt change helped or hurt. Two hundred to five hundred examples, spread deliberately across the categories your feature handles, is roughly where a pass rate becomes something you would defend in a release meeting. Beyond that, marginal examples add less than better coverage of the categories you are weakest in.
The composition matters more than the count. A thousand examples that all exercise the same happy path tell you almost nothing. A hundred examples spanning ten input categories, weighted toward the ones with real consequences, tell you a great deal.
Labelling ground truth when several answers are right
This is where exact-match thinking has to be abandoned properly. For most generative tasks you are not writing down the correct output. You are writing down what makes an output correct.
That means a rubric: a small set of named dimensions, each with a plain-language definition and a scale. Three to five dimensions is the workable range. For a document summarizer, that might be faithfulness (every claim is supported by the source document), completeness (the decisions and owners in the source appear in the summary), and concision (no restating of information the reader already has).
Write the definitions so that two different people applying them to the same output would land in the same place. If they would not, the rubric is not finished. That single test will come back three more times before this article is done, because it turns out to govern almost everything.
The dataset is a living asset
The habit that separates teams who improve from teams who guess is simple. Every production failure becomes a new row. Every ambiguous case a reviewer flags becomes a new row. Every regression discovered after a model upgrade becomes a new row.
Do this for six months and you have an asset that encodes everything your organization has learned about what its AI feature must not do. That asset is worth more than the prompt it protects.
Scoring an output when there is no exact answer
Once you have a dataset, you need a way to turn an output into a number. There are three mechanisms, and they differ enormously in what they cost and what they can see.
Programmatic checks are code. Does the response parse as JSON. Does it contain a citation. Is it under the length limit. Does it avoid a banned term. Did it call the right tool with the right arguments. These are cheap enough to run on every single request in production, and they are reliable within their narrow scope.
Model-graded evaluation, usually called LLM-as-a-judge, uses a language model with a rubric to score an output. This is what makes qualitative dimensions like faithfulness measurable at scale, and it is the technique that turned evaluation from a manual exercise into an automated one.
Human review is a person applying the rubric. It is the most expensive mechanism, the slowest, and the only one that reliably catches the failure mode where an output is fluent, plausible, well-structured, and subtly wrong.
The practical answer is all three, layered by cost:
| Failure mode | How you catch it | What it costs | What confidence it buys |
|---|---|---|---|
| Broken format or schema | Programmatic check | Negligible, runs on every call | High, and deterministic |
| Fabricated or ungrounded claim | Model-graded faithfulness score against the source | Low per run, plus the work to validate the judge | Medium to high, once the judge is calibrated |
| Subtly wrong reasoning | Human review on a sample | Reviewer-hours per release | High on the sample, inferential beyond it |
| Tone or brand violation | Model-graded rubric score | Low | Medium |
| Regression after a model change | Full evaluation-set re-run, gated on threshold | One evaluation run per release | High, but only for the cases in your set |
| Novel adversarial input | Red-team pass and synthetic generation | Bounded project effort | Only for the attacks you thought of |
Read the last column carefully, because it is the honest part. Every technique here buys a specific and bounded kind of confidence. None of them buys the general assurance that the feature is correct, and a testing practice that pretends otherwise is worse than one that admits its limits.
When a model judging a model is trustworthy
Model-graded evaluation works, with conditions attached, and the conditions are where teams lose the plot.
It is most reliable on dimensions with a checkable referent. Asking a judge whether every claim in a summary appears in the source document is a comparison task with a right answer. Asking a judge whether an answer is helpful is a taste judgement, and the score will drift with the judge.
The failure mode to plan around is self-preference. A model tends to rate outputs from its own family favourably, so grading with the same model that generated the response builds in a bias you cannot see in the numbers. Use a different model as the judge where you can.
Calibration is the step you cannot skip. Before trusting a judge, have humans score a subset, then check how well the judge agrees with them. If agreement is poor, the rubric is ambiguous or the judge is unsuitable, and the automated scores mean nothing. Teams skip this step constantly, and it is the reason an evaluation dashboard can look healthy while quality falls.
Score each rubric dimension separately rather than collapsing to one number. When a regression appears, a per-dimension score tells you that faithfulness dropped while tone held steady. A single composite score tells you only that something moved.
None of which matters until somebody decides what number means yes.
Writing acceptance criteria somebody can actually sign
Here is where most AI quality efforts stop short, one step before the thing the release process actually needs. Metrics without a threshold are a dashboard. A threshold without an owner is a suggestion.
A usable acceptance criterion has five parts:
- The metric. Which rubric dimension, measured how, by which mechanism.
- The threshold. The number that constitutes passing.
- The sample. Which slice of the evaluation dataset, and how many examples.
- The reviewer. Who signs off, and who they escalate to.
- The failure action. What happens when it fails. Block the release, ship behind a flag, ship with a wider human-review net, or roll back.
Written out for a support-reply drafting feature, one criterion might read: faithfulness scored by the calibrated judge across the full 300-example evaluation set must reach 95 percent, with all failures reviewed by the support QA lead before release, and any failure involving a billing or account-security topic blocking the release outright.
That is a sentence a human can sign. "Faithfulness looks good" is not.
The threshold is a business decision wearing technical clothes
Ninety-five percent faithfulness is an excellent result for a marketing summary and an unacceptable one for medication guidance. The number does not come out of the evaluation harness. It comes from what happens when the feature is wrong.
The practical method is to tie the threshold to consequence. For low-consequence outputs, where a bad response is mildly annoying and easily corrected by the user, a lower bar plus a broad review sample is proportionate. For high-consequence outputs, where a bad response causes financial, legal, or safety harm, you want a hard gate, a narrow tolerance, and a defined escalation path.
Say the reasoning out loud when you set the number. A threshold nobody can justify will be quietly renegotiated the first time it blocks a release somebody wants.
Somebody has to own the number
Engineering proposes the threshold, because engineering understands what is achievable and what each point of improvement costs. The business owner accepts it, because they own the consequence of being wrong.
Skipping this conversation does not avoid it. It relocates it to the incident review, where the owner turns out to be whoever is available to blame.
Once agreed, the criterion belongs in the definition of done for the feature , next to the ordinary engineering requirements. If it lives only in a document nobody opens during release, it is not a gate.
The Discovery Sprint
Standing this up from nothing is where most teams stall. No individual piece is difficult. The sequencing is, and a first attempt usually produces an evaluation set that measures the wrong things.
Our Discovery Sprint is a one-week engagement that maps which of your AI features need which confidence level, what belongs in each evaluation dataset, which scoring mechanism fits each failure mode, and what thresholds your business can actually defend. You finish the week with acceptance criteria written down and an owner attached to each one, rather than a plan to write them later.
Regression testing when the model moves underneath you
Traditional regression testing rests on an assumption that quietly stopped being true: that the system changes only when you change it.
Providers update models. Snapshots get deprecated on a schedule. Default routing changes. Your retrieval index gets rebuilt with new documents or a new embedding model. Any of these can move output quality without a single commit to your repository.
The evaluation dataset is the answer, because it doubles as the regression suite. Run it on every prompt change, every model upgrade, every retrieval configuration change, and on a schedule regardless, so that provider-side drift surfaces before a customer finds it.
Version everything together
A result is only reproducible if you captured everything that produced it . That means recording, with every evaluation run: the exact model identifier including snapshot, the prompt version, the retrieval configuration and index version, the judge model and rubric version, and the evaluation dataset version.
Teams routinely record the score and none of the inputs. Six weeks later, when quality has drifted, there is no way to establish what changed. This is the cheapest discipline in the whole practice and the one most often skipped.
What this looks like in CI
Run a fast subset, perhaps thirty to fifty examples covering the main categories, on every pull request that touches a prompt, a retrieval configuration, or model routing. It should complete in the time a normal test suite takes.
Run the full evaluation set nightly and on every release candidate.
Gate on the threshold from your acceptance criteria, not on a diff against the previous run. Scores fluctuate between runs by their nature. A gate that fires on any downward movement will be disabled within a fortnight, which is worse than having no gate.
Human review, sampled rather than exhaustive
A language model that is confidently and fluently wrong passes every automated check you have. That is the whole argument for keeping humans in the loop, and it is also why teams get this stage wrong in both directions, either skipping review entirely or attempting it exhaustively.
You cannot review everything. You do not need to. What you need is a sampling design.
Weight the sample toward the outputs that matter. Low-confidence outputs, where the judge scored near the threshold. High-consequence categories, where being wrong is expensive. New or changed functionality, where you have the least evidence. Categories where the judge has historically disagreed with humans. A uniformly random sample spends most of its budget confirming that easy cases are still easy.
Reviewers have to agree with each other first
Put two reviewers on the same subset periodically and measure how often they reach the same verdict. If your own people disagree with each other about what passes, then every threshold built on their judgements is measuring noise, and no amount of automation downstream will fix it.
Disagreement is diagnostic rather than embarrassing. It almost always points at a rubric dimension whose definition is too vague, which is the same test from the labelling section arriving a third time.
The honest cost
Review is labour, and it should appear in the plan as labour. Budget it in reviewer-hours per release rather than as an unowned expectation that somebody will look at the outputs .
The cost curve is favourable, though. Early on, human review is doing most of the work, because you have no calibrated judge and no history. As the judge is validated against human labels, the sample rate for well-understood categories can come down, and human attention concentrates where automation is weakest. The human loop exists to teach the automated layer what good looks like, then to watch the places automation cannot see.
Synthetic and adversarial testing for the cases you have not seen
Everything so far is grounded in what your users have already done. It says nothing about what they will do next, or what a motivated attacker will do deliberately.
Synthetic generation fills part of that gap. Take your existing evaluation examples and generate variants: the same request phrased ten different ways, with typos, in another language, at a different level of formality, truncated mid-sentence. This is a direct test of prompt sensitivity, and it frequently reveals that a feature which handles the canonical phrasing beautifully falls apart on a casual one.
You can also generate edge cases from a specification. Enumerate the categories your feature is supposed to handle, then generate examples for the intersections nobody wrote tests for.
Red teaming is the adversarial half. Prompt injection through user-supplied content or retrieved documents, jailbreak attempts against the system prompt, attempts to extract the prompt itself, attempts to trigger tool calls with harmful arguments, and probing for outputs that are offensive or legally problematic. The ISTQB's AI testing syllabus carries dedicated sections on adversarial attacks and data poisoning, treating them as a core competency for testing AI-based systems rather than an optional extra (ISTQB CT-AI v2.0, 2026).
State the limit plainly, because it is easy to lose sight of. Synthetic data only tests what you thought to generate. It broadens coverage of the failure space you can imagine, and it does nothing about the failure space you cannot. It supplements the production-derived dataset. It never replaces it.
Testing does not stop when the feature ships
Every mechanism described so far runs offline, against a fixed dataset, before release. Offline evaluation cannot see the thing that most often degrades a live AI feature, which is that reality moves.
Online evaluation closes that gap. Sample live traffic, score it with the same rubric and the same judge you use offline, and track the result over time. The infrastructure is the same. Only the input source changes.
Three kinds of drift call for different responses:
- Model drift. The provider changed something. Your inputs are stable, your scores moved. The response is to re-run the full evaluation set, identify which categories regressed, and adjust the prompt or pin a snapshot.
- Data drift. Your retrieval corpus changed, or the documents users reference are different from the ones the feature was built against. The response is to refresh the evaluation dataset from recent production data.
- User drift. People are asking different things. New use cases have emerged that the feature was never designed for and nobody decided to support. The response is a product decision, not an engineering one.
Concept drift is recognized in the testing standards literature as a distinct reason AI systems need continuous monitoring rather than release-time validation alone (ATSQA, accessed 2026). The practical consequence is that release-time testing and production monitoring are two halves of one practice, sharing a rubric, a judge, and a dataset.
Which closes the loop back to the beginning. A production failure becomes a new row in the evaluation dataset. That row protects against the same failure recurring. The dataset gets slowly, permanently better at describing what your feature must not do, and the confidence you can claim about it stops being a feeling and becomes an artifact.
Key takeaways
- Keep your deterministic tests for schema, latency, cost, and refusal behaviour, and stop expecting them to say anything about output quality. They are testing the plumbing, not the water.
- Build the evaluation dataset before buying any tool. Seed it from production logs, add your known-hard cases, and make every incident a permanent new row. Without it, no tool has anything to measure.
- Score with a rubric of three to five named dimensions rather than one composite number, and calibrate any model-based judge against human labels before you trust a single score it produces.
- Write acceptance criteria with all five parts present: metric, threshold, sample, reviewer, and failure action. Engineering proposes the threshold, the business owner accepts it, and it belongs in the definition of done.
- Version the model snapshot, prompt, retrieval configuration, and dataset with every evaluation run, and keep running the set against live traffic after release. Drift arrives without a commit.
What to do first
If none of this exists yet, the first move is not a platform decision. It is fifty examples.
Pull fifty real inputs from production logs, write down the three dimensions that define a good response for your feature, and score the current output by hand. That afternoon of work will tell you more about your AI feature's actual quality than any dashboard you could buy, and it produces the seed of the dataset everything else is built on.
Then write one acceptance criterion, with a threshold and a named owner, and put it in front of the person who would have to answer for a bad output. Their reaction to the number is the conversation that has been missing.
If you would rather not spend three months discovering the sequencing the expensive way, our Discovery Sprint compresses it into a week: your features mapped to confidence levels, your evaluation datasets specified, your thresholds set, and an owner attached to each one.
