A practical log for tracking the clarifications, corrections, and reviews behind an AI-assisted result.
An agent produces a polished comparison in three minutes. Before anyone can use it, a person spends another twenty checking source documents, correcting assumptions, and requesting revisions.
Was the task completed successfully? Perhaps. Did it require only three minutes of work? That describes a different measurement.
A completion score tells you whether an output met the requirements. An intervention log records what people had to do to get it there. Together, they give you a more useful picture of whether an agent fits a task.
Start with one task, not a dashboard
Choose a recurring piece of work with a result someone can check: comparing supplier quotes, preparing a meeting brief, or turning approved source material into a draft.
Write down what an acceptable result must contain before starting. For a quote comparison, that might mean using the supplied prices, preserving delivery terms, showing the source for each entry, and leaving missing information marked unknown.
Record the agent, model, tools, and relevant configuration when available. If a tool does not expose its underlying model, record “not disclosed.” Don’t guess from the product name.
Then use these counting rules consistently:
- Task: the business outcome you want.
- Run: one attempt to produce it. For this log, a deliberate restart counts as a new run; ordinary follow-up instructions remain within the current run.
- Review pass: a person checks an output against the requirements.
- Intervention: a person changes the input, instructions, workflow, or output to help complete the task.
- Revision round: the agent produces another version in response to feedback.
A single feedback message can contain several corrections. That can be one intervention and one revision round. The number of defects, human actions, and generated versions need not match.
Count review time even when nothing goes wrong
A useful log includes work that confirms an output is acceptable, not just work that fixes it.
Separate active human time into three buckets:
- Preparation: assembling inputs and defining the request.
- Review: checking sources, calculations, completeness, and requirements.
- Intervention: clarifying instructions, correcting inputs, editing output, managing a retry, or escalating a problem.
Give each work period a unique event ID. If a person spends four minutes checking an output and three minutes correcting it, record seven minutes split across review and intervention. Don’t count all seven minutes in both categories.
Track waiting separately. Time spent doing another task while the agent runs is not active work on this task. If someone must actively supervise the agent, record that supervision as human work.
Keep one-time setup outside the run totals, but visible. Building a reusable template and reviewing its first result are different costs. Neither should disappear from a later assessment.
A copyable log
At the top of your record, capture:
- Task ID and description:
- Acceptance requirements:
- Agent, model, tools, and version:
- Input and instruction references:
- One-time setup effort:
For each event, capture:
- Event ID:
- Run ID:
- Actor:
- Activity and reason:
- Preparation minutes:
- Review minutes:
- Intervention minutes:
- Time measured or estimated:
- What changed:
- Evidence or output reference:
At the end of each run, capture:
- Outcome: accepted, failed, stopped, or unfinished
- Revision rounds:
- Elapsed time:
- Waiting or blocking time, if known:
- Remaining issues:
Use “unknown” when you did not measure something. Zero means you observed none. An estimate should remain labeled as an estimate.
Worked example: comparing three vendor quotes
The following example and every number in it are hypothetical. They demonstrate the logging method, not the performance of a real agent or model.
The task is to prepare a factual comparison of three quotes for printing an event brochure. The agent must preserve quantities, prices, delivery terms, and source references. It must not invent a missing delivery date. A person will make the purchasing decision separately.
One-time setup: 12 human minutes to create the reusable comparison template. This is recorded separately from the attempts below.
Run A
- E01 — Prepare the request. The reviewer selects the three supplied PDFs and attaches the acceptance requirements. Preparation: 4 minutes.
- E02 — Review the first output. The reviewer finds that one quote uses the wrong print quantity. Review: 6 minutes.
- E03 — Correct the input and stop the attempt. The reviewer discovers an outdated PDF in the supplied inputs, replaces it, and records the reason for restarting. Intervention: 3 minutes.
Run A ends without an accepted deliverable. Active human time: 13 minutes. Revision rounds: zero. Its costs remain in the task record.
Run B
- E04 — Review the restarted output. Quantities and prices now match, but the comparison gives a definite delivery date where the vendor only supplied an estimate. Review: 5 minutes.
- E05 — Request a correction. The reviewer asks the agent to preserve the vendor’s uncertainty and cite the relevant passage. Intervention: 2 minutes. The agent produces one revised version.
- E06 — Review the revised comparison. The reviewer checks it against all acceptance requirements and accepts it. Review: 4 minutes.
Run B produces an accepted deliverable. Active human time: 11 minutes. Revision rounds: one.
Read the totals without losing the story
Across this hypothetical task:
- Two runs produced one accepted deliverable.
- Preparation took 4 minutes.
- Review took 15 minutes.
- Interventions took 5 minutes.
- Total active human time across the attempts was 24 minutes.
- Including the separate 12-minute setup, this first use required 36 human minutes.
- Agent waiting and total elapsed time were not measured.
Reporting only Run B would leave out 13 minutes needed to reach the usable result. Reporting only correction time would hide the larger review burden.
The reasons also matter. Run A encountered a supplied-input problem. Run B required a correction to the generated output. Calling both “model errors” would obscure what to improve.
What this evidence can tell you
The record makes the human contribution visible. It shows where effort went, why a restart happened, and what the reviewer checked before accepting the result.
It does not establish time saved. That requires a comparable measurement of the existing way of doing the task. It does not establish that one model is better than another. That requires comparable tasks, conditions, and enough observations to support the claim.
Across repeated tasks, look for patterns worth investigating. Repeated input corrections may point to preparation problems. Repeated unsupported assumptions may justify changes to instructions, workflow, or model choice. Long review times may mean the output is difficult to verify, even when few corrections are needed.
Keep failed and unfinished attempts in that review. Their effort counts, even when their outputs never reach the final folder.
Try it on the next task
For your next AI-assisted task, record the human work as it happens. Keep the log small enough that someone will actually use it.
Before judging the result, answer three questions: Did it meet the requirements? How much human work did it require? What caused that work?
That is a more useful starting point for agent–model fit than the polished output alone.