A normal feature has a normal test: give it an input, check the output, ship when it passes. An AI feature breaks that deal. Ask the same question twice and you may get two different answers, both fluent, one of them wrong. OpenAI’s own guidance on evaluation says it plainly: models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures.
That does not mean AI features cannot be tested. It means the test suite looks different, and so does the release decision. Here is a practical approach for engineering leaders who need to say “yes, ship it” with evidence rather than a good feeling in a demo.
Start with an evaluation set built from real tasks
The core tool is the eval: in Anthropic’s definition, a test for an AI system where you give it an input, then apply grading logic to its output. An eval set is a library of those tests, and its quality decides everything else.
Build it from reality, not imagination. Pull real tickets, documents, queries and edge cases from the workflow the feature will serve. Add every failure a pilot user reports. Anthropic’s engineering team notes that converting user-reported failures into test cases ensures your suite reflects actual usage, and that 20 to 50 simple tasks drawn from real failures is a great start. You do not need thousands of cases on day one. You need cases that matter, with a clear definition of a good answer for each.
Include the awkward ones: empty inputs, contradictory documents, questions the system should decline, requests in the wrong language, and anything where the correct answer is “I don’t know.”
Grade with the cheapest method that works
Not every check needs a model. Use layers:
- Code checks first. Is the output valid JSON? Does it cite a document that exists? Does the extracted amount match the source? These are deterministic and cheap.
- Model-based graders for judgment calls. A second model can score tone, completeness or faithfulness to a source at scale. Research on LLM judges found that strong ones can reach over 80% agreement with human preferences, the same level as agreement between humans, but also found position, verbosity and self-enhancement biases.
- Human review to keep the graders honest. OpenAI’s guidance says to use human feedback to calibrate automated scoring, and lists not calibrating your automated metrics against human evals as an anti-pattern.
Anthropic adds the habit that separates serious teams: you will not know if your graders are working well unless you read the transcripts and grades from many trials. Someone should read real outputs every week.
Run it more than once
Because answers vary, a single pass proves little. Run each case several times and look at consistency. Anthropic describes two useful measures: pass@k, the likelihood of at least one correct answer in k attempts, and pass^k, the probability that all k trials succeed. For a customer-facing feature, the second number is the honest one. If a task succeeds 9 times out of 10, a user who runs it daily will very likely hit a miss within two weeks.
Test for wrong and ungrounded answers on purpose
The failure that damages trust is the confident wrong answer. Test for it directly. Write cases where the source material does not contain the answer and check that the system says so. Check that every claim traces to a retrieved document. Check numbers, dates and names against the source of truth. We looked at what this failure costs in retail in our piece on AI hallucination in e-commerce, and the pattern holds in any domain where users act on what the system says.
Add adversarial tests before someone else does
If your feature reads emails, web pages, tickets or uploaded files, it reads text written by people you do not control. OWASP ranks prompt injection first in its 2025 Top 10 for LLM applications and explains that indirect injection happens when an LLM accepts input from external sources, such as websites or files. It also admits that it is unclear if there are fool-proof methods of prevention.
So treat it as something to test continuously, not something to declare solved. Plant instructions inside documents and check that the system ignores them. Try to extract the system prompt. Try to make it call a tool it should not. OWASP lists adversarial testing and attack simulations among its mitigations, with the model treated as an untrusted user. Defences themselves belong in your design; we cover them in our guide to AI agent guardrails. This section is about proving they hold, and re-proving it after every change.
Regression-test every prompt and model change
In an AI feature, a prompt edit is a code change and a model upgrade is a dependency upgrade. Both can quietly break things that worked. Anthropic separates capability evals, which ask what the system can do well, from regression evals, which ask whether it still handles all the tasks it used to. Your release pipeline needs the second kind.
In practice: keep a fixed regression set that must not get worse, version prompts alongside code, and run the full set on every prompt edit, model swap, retrieval change or tool change. Compare results against the last release, not against an absolute score, and look at which cases flipped from pass to fail, not only the average.
Set release gates before you need them
Agree on thresholds while nobody is under deadline pressure. A workable gate has three parts:
- Hard floors on critical cases. Zero tolerance for leaking data, following injected instructions or inventing figures in regulated content.
- Minimum consistency on the core tasks, measured across repeated runs.
- No regression against the current production version beyond an agreed margin.
Pair that with a staged rollout: internal users, then a small share of customers, then everyone, with a one-step rollback at each stage. Human sign-off stays for anything high-risk.
Keep testing after launch
Production will find inputs your eval set never imagined. OpenAI’s guidance suggests logging everything so you can mine your logs for good eval cases, and monitoring the app to identify new cases of nondeterminism and grow the eval set over time.
A simple operating loop works for most teams: sample a fixed slice of live conversations each week, score them with your automated graders, have people review the lowest scores plus a random handful, and add every confirmed failure to the regression set. Track cost, latency, refusals and user corrections alongside quality, so a quiet drift shows up as a trend line rather than a customer complaint.
What to do this quarter
Pick one AI feature already in production or close to it. Collect 30 real tasks, including 10 that should fail safely. Write a grader for each, run them five times, and read the transcripts. Add a regression run to the pipeline and agree on one release gate. That is a few weeks of work, and it turns “we think it works” into a number you can defend.
Test it with 247 Labs
247 Labs builds AI features with evaluation and release gates designed in from the start, and can add them to systems you have already shipped. If you want a second set of eyes on how your AI features are tested, talk to our team about reviewing one real workflow.
