Skip to content
Agent Model Fit

GuidesPractical scorecard

Did the AI Agent Actually Finish? A Practical Evaluation Scorecard

By Agent Model Fit

Published

Sources (2)

On this page

An AI agent has finished a task when a reviewer can find the intended deliverable, check it against the agreed requirements, and account for any unresolved issues or external actions. Permission to use the result is a separate decision. A correct purchasing recommendation, for example, does not authorize a purchase.

For leaders and business owners accepting agent-delivered work, the scorecard below provides a small record of that decision. It is Agent Model Fit’s proposed method for reviewing one output. We have not validated it as a predictor of reliability or a way to reduce rework.

Name the result you are reviewing

Start with four details: the output and its version, the inputs used, the intended use, and the person responsible for accepting it. “The inventory analysis” is too vague. “Replenishment recommendation version 2, using Friday’s approved stock snapshot, for the operations manager’s review” identifies something a reviewer can inspect.

Then write the acceptance criteria in observable terms. Replace “accurate and useful” with requirements such as “every recommendation includes its source stock record and calculation” or “missing delivery information is flagged rather than treated as zero.”

Keep these four questions separate:

  • Existence: Can the reviewer open the requested output?
  • Correctness: Does that version satisfy the agreed criteria?
  • External effect: If an action was requested, does the destination system confirm it?
  • Authorization: May the result be used for the named purpose?

Anthropic’s vendor engineering guidance distinguishes an agent’s transcript from the outcome recorded in its environment. That distinction matters when the agent reports that an action succeeded: the message and the resulting system state are different evidence. See Demystifying evals for AI agents.

For workflows that change records or publish material, our practical agent architecture guide explains where those external effects belong.

Use five review dimensions

Choose the checks that matter for this deliverable. The dimensions below are prompts for review, not five equally weighted scores.

DimensionQuestion to answer
Acceptance criteriaDoes the delivered version meet each agreed requirement?
Output correctnessDo facts, calculations, references, and exclusions agree with the approved inputs?
InterventionWhat clarification, correction, or manual assistance was needed?
Rework and recoveryWhat was repaired, and what evidence shows the repair worked?
Open issues and authorityWhat remains unresolved, and who can authorize the named use?

Record assistance even when it was appropriate. A reviewer supplying a missing definition differs from a reviewer rewriting an incorrect result. Both belong in the record because they explain what was actually delivered and what human work remains.

For a repaired output, identify the new version and the checks affected by the change. Recheck those checks and any dependent conclusions. Retain earlier results when the correction does not affect their inputs or meaning.

Copy this blank acceptance record

Create one entry per criterion, using the same output/version reference throughout. Keeping the fields vertical makes the record usable in a document, ticket, or ordinary spreadsheet.

FieldEntry to complete
CriterionSpecific requirement being checked
Evidence locationOutput section, source record, test result, or system receipt
Actual resultPass, fail, unavailable, or not applicable, with the observation
LimitationWhat this check does not establish
Responsible reviewerPerson or role accountable for this check
Unresolved actionSpecific next action and its owner, or “none”

Use fail when the evidence shows a requirement was missed. Use unavailable when you cannot determine whether it was met. Use not applicable only with a reason. A blank cell should never quietly become a pass.

The NIST AI Risk Management Framework’s MEASURE 1.1 calls for documenting risks or characteristics that are not measured; MEASURE 2.1 calls for recording test sets, metrics, and tools. These principles inform the record’s evidence and limitation fields. NIST has not approved this scorecard. See the AI RMF Core.

Illustrative example: a replenishment recommendation

This scenario and every value below are hypothetical. An agent receives approved inventory records and prepares a draft recommendation. It has no purchasing capability.

The agreed calculation is:

Suggested quantity = target stock − (on hand − reserved + confirmed inbound), with a minimum of zero.

The agent must flag missing inputs instead of guessing. The operations manager reviews the resulting recommendation.

ProductOn handReservedConfirmed inboundTarget
Cedar802010100
Slate400Missing60

Cedar’s calculation is 100 − (80 − 20 + 10) = 30 units. Slate has no supported quantity until its inbound information is resolved. Recommending 20 units would silently assume there is nothing inbound.

Suppose the draft recommends 30 Cedar units and 20 Slate units. The acceptance record would distinguish these findings:

CheckObservationNext action
Cedar calculationPass: 30 agrees with the supplied valuesNone
Missing-input ruleFail: Slate treats missing inbound as zeroReplace the quantity with a missing-data flag
InterventionReviewer identified the unsupported assumptionRecord the correction
Repair verificationUnavailable: revised output not yet checkedInspect the revised Slate entry
Purchasing authorityNo purchase belongs to this taskRetain the draft-only boundary

The disposition is revise, with one precise correction. If the manager needs an actionable Slate quantity, the missing inbound record must also be obtained and the calculation checked. A well-formatted draft does not resolve that missing input.

Record the decision and its scope

Close the record with one of three decisions:

  • Accept for the named use: Identify the accepted version, reviewer, date, and any conditions.
  • Revise: Specify the changes and affected checks required before acceptance.
  • Hold: Identify the missing evidence or authority and who can resolve it.

Avoid a numerical total that lets a serious defect disappear among successful checks. An unsupported recommendation remains unsupported even if formatting, arithmetic elsewhere, and delivery all pass.

Use appropriately qualified reviewers for consequential work. This record organizes their evidence; it does not replace their judgment or establish how an agent will perform on future tasks.

Start with one already-delivered, reversible task and its real reviewer. If the work fails repeatedly, use our model escalation guide to consider the next approach. If the task itself is a poor fit, revisit how to identify suitable automation opportunities.

Provenance and review status

Prepared by Agent Model Fit with AI assistance for research and drafting on October 2, 2026. The scorecard is a proposed framework; the inventory example is illustrative and is not an AMF experiment. Human editorial review by the Agent Model Fit team was completed on October 2, 2026.

Sources

  1. NIST AI RMF 1.0 Core
  2. Anthropic: Demystifying evals for AI agents