Your team builds an AI agent to review incoming service requests. The test file has complete customer records, clear language, current policy documents, and examples everyone has seen before.
The agent looks great. Then production sends a request with two customer IDs, an expired contract, a screenshot instead of text, and a note that contradicts the CRM.
That is when the test set stops being a technical detail. It becomes the buying decision.
A pilot should not pass because the implementation team knows the answers. It should pass because the agent can handle representative work, reject work it should not touch, recover from broken dependencies, and send uncertain cases to the right person.
A demo set answers the wrong question
A demo set asks, "Can the agent work?" Production testing asks, "Under which conditions can this workflow be trusted, and what happens outside those conditions?"
Those are different questions. A polished set of normal examples can prove basic capability. It cannot prove the workflow will hold up when data is missing, instructions conflict, a connector times out, or a person tries to push the agent outside policy.
Start with the workflow decision. Define the business outcome, the actions the agent may take, the actions it must refuse, the systems involved, and the point where a person takes over. If those boundaries are still vague, use the workflow readiness test before building a larger evaluation.
Build the test set from production reality
Do not ask the builder to invent every test case from memory. Pull patterns from real work: corrected records, reopened tickets, failed imports, policy exceptions, customer complaints, approval overrides, duplicate transactions, and incidents. Remove private data and preserve the operating condition that made each case hard.
Talk to the people who handle exceptions. They usually know which records look normal until one field changes the decision. They also know where another team quietly cleans up the process.
Segment the set by workflow condition, not just by department or output type. You need enough cases to see whether performance drops in a specific condition instead of hiding that failure inside one average score.
Require eight test-case types
| Case type | What it tests | Example |
|---|---|---|
| Normal | Expected performance on common work | Complete request that fits one approved policy path |
| Incomplete | Whether the agent asks, stops, or guesses | Missing account ID, consent, date, or approval evidence |
| Conflicting | Source priority and escalation | CRM status disagrees with the signed order |
| Stale | Freshness checks | Old price list or expired customer entitlement |
| Duplicate | Idempotency and state awareness | Same event arrives through an API and an email |
| Unavailable | Failure behavior when a dependency breaks | Model, connector, approval service, or destination times out |
| Out of policy | Permission and action boundaries | Request exceeds a financial limit or uses prohibited data |
| Adversarial | Resistance to manipulation | Document tells the agent to ignore policy or reveal restricted data |
Not every workflow needs the same mix. A read-only research assistant and an agent that changes customer records have different failure costs. Weight the set toward the conditions that can create operational, financial, legal, or customer harm.
Write the expected result before running the agent
Each case needs more than an input and a preferred answer. Record the accepted outcome, allowed sources, permitted actions, required approval, stopping condition, expected downstream state, and maximum acceptable time or cost.
Some cases should have several acceptable responses. A good agent may ask for missing information or route the work to a person. That is not a failure if the workflow requires it.
Separate correctness from consequence. A minor wording correction and an unauthorized account update should not count as equal misses. Use severity levels tied to customer impact, data exposure, financial effect, reversibility, and recovery work. Then connect those levels to the production scorecard.
Keep a holdout set the builder cannot memorize
If the same 40 cases are used to design prompts, tune tools, fix routing, and declare success, they are training material. They no longer provide an honest release test.
Split the evaluation into a development set and a protected holdout set. Builders can use the development cases while improving the system. A separate owner should control the holdout cases, expected outcomes, and final scoring. Refresh both sets as production reveals new conditions.
Do not chase a perfect score by rewriting the answer key after every miss. Investigate whether the system failed, the source record was wrong, the policy was unclear, or the expected result was badly defined. Test-set quality needs review too.
Test the whole workflow, not the model response
A correct answer in a chat window can still become a failed business process. The agent may read the wrong customer, call an outdated tool, write to the wrong field, create a duplicate, lose approval context, or report success before the downstream system commits the change.
Run representative cases through the same identity, permissions, connectors, approval path, and destination used in production. Verify the final state and the transaction record. For write actions, test authorization, duplicate prevention, confirmation, rollback, and reconciliation.
If the vendor will only demonstrate a model response, the test is incomplete. Use the vendor pilot acceptance test to put environment-specific evidence into the buying record.
Set release rules before anyone sees the score
Define minimum performance by case type and maximum failures by severity. One average accuracy number can hide a serious problem. An agent may score 96 percent overall while failing half of the rare cases that require a stop or human approval.
Use hard blockers for unauthorized actions, restricted-data exposure, wrong-account writes, silent duplicate transactions, or any other failure the business has classified as unacceptable. Set separate thresholds for normal work, exceptions, escalations, recovery, review effort, latency, and cost.
Name the person with release authority. That person should receive the test design, results, unresolved failures, tested configuration, and recommendation. A dashboard full of green averages is not enough.
NIST gives buyers a useful standard
The NIST Generative AI Profile says organizations should review and document the accuracy, representativeness, relevance, and suitability of data used across the AI lifecycle. It also recommends comparing AI output with known ground-truth data and using multiple evaluation methods.
NIST calls for empirically validated methods when evaluating capability claims. Its profile also says pre-deployment test results should be shared with people who have system release approval authority. That is a practical line for buyers: evidence belongs with the person making the go or no-go decision.
NIST also recommends regular adversarial testing to identify vulnerabilities and manipulation. Your test dataset should not be a one-time launch artifact. Keep adding production exceptions, policy changes, attack patterns, and failure cases. When the model, prompt, connector, policy, or data source changes, run a release test against the affected cases.
Use this test-set release checklist
Before production access, require a written yes to these questions:
- Does the set represent common work plus incomplete, conflicting, stale, duplicate, unavailable, out-of-policy, and adversarial conditions?
- Was each expected outcome defined before the run, including allowed actions, approval, stopping, and downstream state?
- Are failures scored by business consequence instead of one flat error count?
- Was a protected holdout set kept away from repeated tuning?
- Did the test run through production-like identity, permissions, connectors, approvals, and destinations?
- Were release thresholds and hard blockers approved before the results were reviewed?
- Can the final record identify the exact model, prompt, policy, tool, data, and workflow versions tested?
- Did the person with release authority receive the failures and limits, not just the average score?
If the test set contains only clean examples, do not call the agent production-ready. Call it what it is: a demo that passed its own rehearsal.
Build the ugly cases into the buying process now. They will show up later either way.