Compare the complete agent on the work you need done, not just the model name on its settings screen. Start with the same model in two execution systems, hold the task and permissions steady, and inspect both the final result and how the run ended. Then decide whether the next change belongs in the model or the system around it.
For a team choosing a research assistant, reporting agent, or internal workflow tool, this is a practical buying distinction: access to a strong model is not the same thing as a reliable service.
What an agent harness changes
An agent harness is the software around a model that manages its interaction with tools and the environment, including context, execution, and stopping. MedicalHarness, introduction and problem formulation.
Consider a hypothetical weekly reporting job. Three candidate agents receive the same approved sales export and must deliver a reconciled summary with an exception list.
Agent A writes a plausible explanation in chat but never saves the requested report. Agent B saves a report, but drops the exception list after shortening its conversation history. Agent C delivers both files with correctly reconciled totals.
Calling the first two outcomes “the model got it wrong” leaves important questions unanswered. Was file delivery an explicit requirement? Did the agent have permission to write the file? Were the exceptions still visible when it prepared the final report? Did execution end before all required outputs existed?
These are diagnostic questions, not a claim that any particular product behaves this way. The example is hypothetical. Its point is to make the missing deliverable or lost condition observable rather than hide it inside a single quality score.
What the research adds
Wang, Zhao and Ding’s October 5 MedicalHarness preprint compares five open-weight models in five harnesses on 107 medical tasks, with three seeds: 8,025 episodes. The authors report that harness effects and model-harness interactions together explain roughly 24% of outcome variance, averaged across four environments. That is not a 24% accuracy gain. No harness wins across every model and task; only two of twenty model-environment comparisons show a leader-runner-up gap whose 95% interval excludes zero. These controlled medical results are not a ranking of default consumer products or proof of performance on business reporting. Study setup and results.
AMF’s takeaway is a selection rule, not a new leaderboard: make the model, surrounding system, and task explicit in the comparison. Treat a promising result elsewhere as a reason to investigate a combination, not a substitute for judging your own deliverable.
A small comparison that produces a decision
The following is AMF’s proposed procedure. We have not run this experiment, and the suggested sample is a starting point, not a statistical guarantee.
1. Define an acceptable result before running either agent
Choose one recurring workflow with outputs a reviewer can check. Write down the deliverables, the evidence they must use, and the conditions that require a stop or escalation.
For the reporting example, acceptance might require:
- The report and exception list both exist and open correctly.
- Totals reconcile to the supplied export.
- Every excluded row has an explained treatment.
- Missing inputs remain visible rather than being filled with guesses.
- No external message or production change is made.
Keep unauthorized action separate from incorrect output. A correct-looking report does not excuse sending it to someone the user never authorized.
2. Hold the comparison steady
Use two harnesses that can actually run the same model version. Record that version, provider, reasoning settings where exposed, source files, tool permissions, time limit, and retry allowance. Keep those conditions the same wherever possible.
Start with a small set of representative cases, including a normal export, missing data, contradictory inputs, and a task requiring more than one output. Repeat cases rather than selecting the best attempt.
If a product does not expose the model version or comparable controls, you can still compare the complete products. Label it a product comparison, not an isolated harness test. Acknowledge the differences you cannot control.
3. Keep unfinished work in the denominator
For every attempt, record one final disposition: accepted, rejected, blocked, or unfinished. Preserve a short reason and the resulting artifacts.
An agent that returns an answer on eight of ten tasks has not necessarily succeeded eight times. A missing file, unsupported claim, or unreconciled total can still make an answered task unacceptable. Conversely, stopping because required permission is missing may be the correct behavior, even though the business task remains incomplete.
Report these outcomes separately. Do not quietly remove timeouts or failed tool calls from the comparison.
4. Count the whole cost of getting usable work
Track model and tool charges where available, elapsed time, retries, and human correction or review time. Do not infer a saving from a shorter final answer.
Use two views: accepted results out of all attempts, and total measured run cost divided by accepted results. Report human effort alongside those figures rather than burying it inside an invented hourly rate. If no result is accepted, cost per accepted result is undefined, not zero.
The AI pilot cost worksheet and human-work log provide complementary records.
5. Change the part implicated by the failure
If the model never received a decisive exception, test evidence retention or delivery before paying for a larger model. If the evidence was present but repeatedly misinterpreted, a model change becomes a more specific hypothesis. If tools were unavailable, resolve that access boundary rather than relabeling it a reasoning failure.
Do not change the model and harness simultaneously and then credit one for the improvement. First compare two harnesses with a fixed model. If a model change is still worth testing, hold the chosen harness steady for that comparison.
Use the evidence-delivery guide when the failure sits between finding a source and using it correctly.
Limitations and fit
This guide does not certify a harness, validate a medical application, or establish that a workflow is safe to automate. AMF has not reproduced the cited study. Small internal trials also cannot establish reliability for every workload or rare failure.
A simpler tool may be enough for a one-step lookup. The proposed comparison is most useful when the same recurring job can fail in several distinguishable ways and the team is deciding what to replace.
Record the test date and configuration. A later model or software update is a changed candidate, not automatically the same tested system.
Next action
Take one disappointing agent result and classify the failure: missing evidence, incorrect reasoning, failed tool execution, unfinished delivery, or a legitimate permission stop. Then choose one controlled comparison that could distinguish the likely causes.
The goal is not another evaluation program. It is a concrete decision about which change is worth making before the next run.
Provenance
Prepared for Agent Model Fit with AI-assisted primary-source inspection, analysis, and drafting on October 23, 2026. The research summary is attributed to Wang, Zhao and Ding’s MedicalHarness preprint. The reporting scenario, comparison procedure, and business interpretation are AMF proposals, not measured results. No benchmark replication, product certification, human review, or business saving is claimed.