Before changing an AI model, collect a manageable set of requests that reflect the work your application needs to do. Include routine tasks, incomplete information, requests the system must refuse, and workflows involving tools. Record what a satisfactory result looks like before comparing the current model with a candidate.
A public benchmark can help identify models worth considering. It cannot establish whether a model will work with your prompts, documents, permissions, and connected systems. Test the application behavior that matters to your users.
Define the decision the evaluation should support
Start with the reason for considering a change. You may want better answers, lower operating cost, shorter response times, or support for a different deployment arrangement. Decide which improvements matter and which existing behaviors must remain acceptable.
Write down the failure conditions as well as the desired results. An answer that sounds better is not an improvement if it invents a price, discloses restricted information, or changes a record outside the user's authority. Treat those failures separately from preferences about wording.
The National Institute of Standards and Technology's initial public draft on automated benchmark evaluations emphasizes defining evaluation objectives and reporting qualified claims. Its scope is automated benchmark evaluation, rather than instructions for building a new business-specific dataset. Use the same care when interpreting your own results.
Collect cases from the actual work
Begin with requests people already test manually, reported failures, and recurring support questions. Use production records only where their use is authorized. Remove unnecessary personal information and secrets, and restrict access to any retained sensitive data.
Include cases from several parts of the workflow:
| Case type | What to test |
|---|---|
| Routine work | Whether the system completes common tasks using the correct sources |
| Missing or conflicting information | Whether it asks for clarification or explains the limitation |
| Restricted requests | Whether it respects data permissions and action limits |
| Tool use | Whether it selects the right operation and supplies valid inputs |
| Interrupted work | Whether it reports an uncertain result and follows the supported recovery process |
Choose the mix according to use and risk. There is no fixed percentage that fits every application. A rare but serious failure deserves an explicit test even when it represents little of the normal workload.
You can add constructed cases to test situations that have not occurred yet. Record which cases come from observed work and which were created for testing. Neither source is useful if the expected result is unclear.
Describe success in terms a reviewer can apply
For each case, save the request, the necessary context, the permitted actions, and the expected result. Record the version of the source documents or test data so another reviewer can reproduce the comparison.
Use checks that fit the task. A document summary may need accurate source references and coverage of specific points. A tool request may need the correct record identifier and a valid operation. A workflow may need a verified change in a test system, followed by confirmation that no extra action occurred.
Separate the final result from the route taken to obtain it. Several tool sequences may be acceptable. Require a particular sequence only when the order matters for correctness, permissions, or recovery.
Define how to handle an ambiguous result. A short scoring guide with examples is more useful than a broad instruction to judge “quality.” Review disagreements can reveal unclear requirements as well as model weaknesses.
Run a comparable test
Save the prompts, retrieval inputs, tool definitions, application version, and model settings used in each run. Hold them constant where possible when comparing models. If the candidate needs different settings or an adapter, document that difference and test the resulting application configuration.
Run tool-based cases in an isolated environment or with controlled substitutes. Do not issue real refunds, send customer messages, or change production records just to compare models.
Model outputs can vary between attempts. Repeat important cases and retain the raw outputs and tool results. A fixed seed, where supported, is not a guarantee of identical behavior across providers or versions.
Where reviewers compare written answers, hide the model names and vary the display order. Review the result against the scoring guide rather than choosing whichever response is longer or more polished.
Review failures before relying on the average
Summarize results by case type and by the business areas that matter. An overall pass rate can hide a regression in restricted requests or a workflow used by a small but important group.
Report the number of cases and attempts alongside the results. A small evaluation set is useful for finding regressions and clarifying requirements; it does not establish a precise production failure rate or prove that rare failures cannot occur.
Anthropic's engineering guidance on agent evaluations recommends starting with existing manual checks and real failures, then expanding the suite as the system develops. Use that approach to make an initial comparison useful without treating it as complete coverage.
If you use another model to score outputs, compare its judgments with human review on representative cases. Investigate disagreements, including whether answer order or wording affects the score. An automated judge is another component to evaluate, not an independent guarantee of correctness.
Plan a limited release and recovery
Passing offline tests supports a release decision, but live use may expose conditions the test set missed. Decide how much work the candidate may handle initially, which signals to monitor, and who can stop the rollout.
Retain a tested configuration that the application can use if the candidate performs poorly. Check that the earlier model remains available and compatible with the current prompts and tools. Switching configuration does not undo messages already sent or records already changed; those need their own recovery procedures.
Monitor answer quality, tool outcomes, response times, and cost for the actual workload. Investigate a regression before assuming the model caused it. Retrieval changes, source updates, and service failures can also affect the result.
Add confirmed failures to the evaluation set with clear expected behavior. Retest the affected cases and the existing suite before the next change. Keep some cases separate from prompt development so the team can check whether improvements extend beyond the examples it has been tuning against.
Explore X-Centric's AI consulting services for help defining an evaluation and deciding whether a model change improves the work your application needs to do.