Compare AI models by giving them a defined task under recorded conditions, applying acceptance rules chosen before seeing the outputs, and accounting for every attempt. Decide in advance what evidence would justify changing your current approach.
This article provides Agent Model Fit’s proposed comparison worksheet. It contains no AMF model results, tested rankings, or recommended universal trial count. It is for teams choosing a model for a particular workflow, with a human reviewer responsible for judging whether the evidence supports that choice.
Our model escalation guide explains when a different model may be worth considering. This worksheet turns that question into a comparison you can inspect.
Define the choice before the test
Write a one-sentence decision: “Choose a candidate for extracting approved product facts into a draft catalog record, subject to human review.” Include the current manual process or a simpler tool when either is a realistic alternative.
Next, identify the unit being compared:
- Model substitution: The surrounding workflow stays fixed while the model changes.
- Whole-system comparison: The candidates use different prompts, tools, context, or operating procedures.
Both can inform an implementation decision. A whole-system result, however, cannot tell you that the model alone caused the difference. Our agent architecture guide explains the relevant system boundaries.
Write the required output, permitted data and tools, prohibited actions, and responsible reviewer. Mark critical requirements separately from preferences. An unauthorized external action should remain visible even if the output is otherwise excellent.
NIST’s discussion of AI risks and trustworthiness places the choice of metrics and thresholds in the context of use and human judgment. A suitable threshold therefore needs an owner and a reason; it should not emerge from whichever candidate looks best afterward.
Record the candidate conditions
Complete this table before the assessment. “Unavailable” is an acceptable record of an unobservable setting; an assumed version is not.
| Condition | Candidate A | Candidate B |
|---|---|---|
| Model and observable version | — | — |
| Prompt revision | — | — |
| Tools and relevant versions | — | — |
| Input/context snapshot | — | — |
| Data access and permissions | — | — |
| Runtime and output limits | — | — |
| Allowed human intervention | — | — |
| Other settings or unknowns | — | — |
Use identical conditions where the intended comparison allows it. Record necessary differences and explain what they prevent you from concluding. If one application hides its prompt or model version, the comparison may still support a choice between applications, but the attribution is narrower.
NIST’s AI RMF Core calls for documenting evaluation tools and demonstrating performance under conditions resembling deployment. Those principles support recording the environment instead of reporting a score without its conditions. See MEASURE 2.1 and 2.3.
Separate practice cases from assessment cases
Use development examples to clarify instructions and repair ambiguous criteria. Reserve a separate assessment set for the comparison you intend to report.
Include ordinary cases and relevant exceptions: missing fields, conflicting records, unusual units, and cases that should be referred to a person. Select exceptions because they matter to the intended work, not because they make a favored candidate look stronger.
Choose the number and mix of cases according to the decision’s consequences and the evidence you need. A small demonstration can expose a defect without establishing dependable performance across the workflow.
If an assessment case changes your prompt or acceptance rules, record that change. Treat subsequent work as a new assessment version; do not present the revised attempt as an untouched first result.
Decide how attempts will be counted
Use task for a case with defined inputs and success criteria, and trial for an attempt at that case. This terminology follows Anthropic’s vendor engineering guidance. Keep the two denominators separate: repeating one case gives more attempts, not more distinct cases.
Before running, record the planned trials per case, allowed retries, stopping conditions, and treatment of human assistance. Decide whether you need success on the first attempt or whether a reviewed retry is acceptable. Preserve failed attempts when a later retry succeeds.
Define how each trial starts. Anthropic also recommends isolated trial environments because leftover state can distort results. Record your reset procedure, including any shared files, conversation history, or retained outputs.
An environment failure needs a diagnosis, not an automatic low model score. If a documented test setup fault prevents one candidate from receiving the required input, label the affected comparison invalid and explain the rerun. Keep that attempt in the accounting. If a tool failure is part of the deployed system you are evaluating, it may instead be a relevant operational failure.
Keep one result record per trial
Leave results blank until evidence exists. For each case/candidate/trial combination, complete the following record.
| Field | Entry after the trial |
|---|---|
| Case, candidate, and trial ID | — |
| Criterion outcomes | — |
| Output and supporting evidence | — |
| Human intervention | — |
| Missing observations | — |
| Validity and any exclusion reason | — |
| Reviewer’s decision | — |
Use explicit outcomes such as pass, fail, and unresolved for each criterion. Record invalid trials separately. When summarizing, distinguish planned attempts, attempts actually started, valid completed attempts, invalid attempts, and unresolved attempts. State the denominator behind each percentage. Do not silently remove failures or count an unobserved result as success.
Keep critical failures separate from averages. Record review effort if it matters to the choice, using the same definition for both candidates.
Illustrative task: product-description extraction
This example is hypothetical. Candidate A and Candidate B are placeholders; neither has been run or scored.
The task is to extract product name, material, dimensions, and warranty from approved specification sheets into a draft catalog record. Each populated field must point to its supporting passage. Missing fields must say “not supplied.” The agent may not search elsewhere or publish the record.
A sheet describes a fictional Harbor Desk Tray, provides its material and dimensions, and says nothing about a warranty. The expected behavior is to flag the missing warranty. Inferring a one-year warranty would fail the evidence requirement, however plausible it sounds.
A reviewer should establish that interpretation before seeing either candidate’s answer. The same rule then applies to both.
Make a bounded decision
The available decisions are to prefer a candidate for the named use, retain the current approach, narrow the task, or collect more evidence. State what remains unknown alongside the choice.
Revisit affected results when the model, prompt, tools, input sources, permissions, or intended use changes materially. Results for catalog drafts do not establish suitability for publishing them automatically.
Your next action is to complete the conditions table and acceptance rules for one real workflow. Resolve disagreements about success before spending effort running the comparison.
Provenance and review status
Prepared by Agent Model Fit with AI assistance for research and drafting on October 2, 2026. This is a proposed evaluation method, not a completed comparison. No candidate outputs or performance results are reported. Human editorial review by the Agent Model Fit team was completed on October 2, 2026.