When a search agent produces an incomplete answer, inspect the evidence path before changing the model. Did the system find the relevant document? Did it send the decisive passage to the model? Did the answer preserve the passage’s meaning and conditions?
Those questions point to different repairs. A better query may help when a document was missed. A different evidence-selection step may help when the document was found but the important paragraph was dropped. If the paragraph reached the model and the answer still misrepresented it, check whether later summarization or truncation removed it, then investigate reasoning, instructions, and answer verification.
This article explains the relevant research, then proposes a small evaluation a team can run on its own work. The evaluation procedure and business example are AMF proposals, not experiments reported by the researchers.
What Programmatic Search Agents changes
Qian and colleagues’ Programmatic Search Agent (PSA) retains candidates across steps, composes search operations in executable cells, and selects the evidence returned to the deciding model. Its tool-based comparator shares the primitives and persistent workspace; PSA additionally composes dependent operations within a cell. Method, section 3.
In its motivating study of one query-based agent, the paper tracked verified supporting passages that entered extraction candidates but never appeared in returned observations. This suggests an evidence-delivery gap, rather than showing that the model ignored evidence it had seen. Passage identifiers alone do not establish whether semantically equivalent support was absent. The cohort was combined after inspecting two runs and should be treated as descriptive, rather than a general failure-rate estimate. Evidence tracing, section 2.1 and Appendix B.
AMF’s practical interpretation is to make the boundary between stored material and model-visible material inspectable. For a buyer evaluating a research tool, “we retrieved the document” should lead to a follow-up question: “Which part of it was actually available when the answer was written?”
What the reported results establish
Across five backbones, PSA improves macro-averaged task success over the query-based baseline by 4.00 percentage points on InfoSeek-Eval and 7.56 on BrowseComp-Plus. These are fixed-benchmark, model-judged results; PSA does not lead every comparison. Results, section 4 and Table 1.
The paper’s final-step token metric counts input and output for the last decision step, plus independent fallback finalization when used. Some historical traces use calibrated reconstructions, and tasks without usable token estimates are excluded. It is not total run cost. Equal decision budgets also permit different amounts of work per decision. Setup and accounting.
Treat these results as a reason to test the interface on your tasks. They do not establish how much your team would save, whether your sources contain the needed answer, or whether your current tool loses evidence in the same way.
A supplier comparison with one missing condition
Consider this hypothetical task: compare three supplier proposals and identify which suppliers can meet a delivery deadline. Each recommendation must cite the delivery condition and distinguish a commitment from an estimate.
One proposal says delivery is available within ten working days, provided the customer approves the artwork by noon on Monday. Imagine that the search system retrieves the proposal but returns only the sentence about ten working days. The answer then recommends the supplier without mentioning the artwork deadline.
There are at least three places to investigate:
- Document selection: was the current proposal retrieved, or an older version?
- Evidence delivery: did the model receive the artwork condition alongside the delivery statement?
- Answer construction: if both were visible, why did the final recommendation omit the condition?
For this example, a proposed evidence-processing step would inspect the delivery passage together with its adjacent qualification, retain a source reference, and return both as one reviewable item. The answer should preserve the conditional delivery timeframe and distinguish a commitment from an estimate. It should compare that timeframe with the required delivery date and leave feasibility unresolved if the artwork approval date or other necessary information is unknown.
That is a design sketch. It is not PSA code, a tested integration, or a claim that preserving adjacent text always captures the necessary context. A condition may live in another section or document.
Start with a trace the team can inspect
Choose one recurring, checkable task, such as answering questions over approved policy documents or comparing supplied proposals. Before testing, write down what an acceptable answer must contain.
For each task, keep a compact record:
- Task ID, question, and acceptance requirements
- Source versions, access dates, and relevant permissions
- Agent, model, retrieval settings, and limits, where disclosed
- Retrieved document and passage identifiers
- Evidence included in actual model inputs, including the answer-producing call, and any intervening summaries or truncation
- Final answer, citations, unresolved conditions, and reviewer decision
- Retries, execution errors, elapsed time, and human effort
Treat a missing trace as unknown. A citation in the final answer does not, by itself, prove that the model received the passage supporting that claim.
Keep the record proportionate to the task. Use authorized storage and limit access to source text. A useful evaluation should not create a second uncontrolled copy of confidential documents.
Score the handoffs separately
A single answer score hides where the work failed. AMF proposes checking four outcomes for every case.
1. Relevant evidence was found
Before comparing systems, have a reviewer identify the passages needed to support the expected answer, including exceptions and contradictory evidence. Allow more than one valid evidence set when the question permits it.
Record how many required facts have support in the retrieved material. A page-level match is insufficient when the relevant paragraph is absent from the captured text.
2. Required evidence reached the model
For the facts with retrieved support, check whether a sufficient supporting passage appeared in a model-visible observation before the answer.
Also check whether that support, including its qualifications, remained available in the answer-producing call, directly or through a faithful intermediate summary. Record truncation, summarization, and handoff losses separately from initial delivery failures; mark this unknown if the effective model inputs are unavailable.
Report the count and denominator. For example, “support for six of eight required facts was retrieved; support for four of those six was shown.” This hypothetical count separates a retrieval gap from a delivery gap without pretending that either is answer accuracy.
3. The answer used the evidence correctly
Check each material claim against the cited passage. Did the answer preserve dates, scope, exceptions, uncertainty, and conflicting information? Did it answer all required parts? Did it leave unsupported conclusions unresolved?
Score a conditionally correct answer differently from an unsupported definite recommendation. Keep the reviewer’s reason for rejection, not just a pass/fail label.
4. The complete task stayed within budget
Count work across the full run: model calls, retrieval and extraction calls, failed attempts, waiting, and review. If prices or token usage are unavailable, record that limitation rather than estimating a saving from a shorter final context.
For the human side, use the human-work intervention log. A system that requires fewer searches but much more source checking may still be a poor fit.
Compare a small change before a large replacement
Start by locating the dominant failure in the traces. If source discovery is reliable but qualifications are frequently dropped, test a narrowly changed evidence-selection step against the existing system.
Hold the questions, source versions, access permissions, answer requirements, and model settings constant where possible. Record any difference that cannot be controlled. Include cases with missing evidence, contradictory documents, outdated versions, and long qualifications.
Set the acceptance rule before looking at results. For the supplier example, an omitted delivery condition can be a rejection even when the supplier name is otherwise correct. Define acceptable latency and human review effort for the actual business process rather than borrowing a universal threshold.
Repeat cases and report the number of attempts, including failures and unfinished runs. Where practical, hide the system identity from the reviewer. A hand-selected success can illustrate a mechanism; it cannot establish a reliability rate.
If a selective evidence view is the proposed improvement, compare it with a fuller view of the same retrieved material. Check whether brevity removes distracting text or hides qualifications the reviewer needs. Preserve enough source context to inspect either outcome.
Limitations and fit
As of October 17, 2026, arXiv still lists only the October 5 v1 paper, and code release remains subject to approval; no official implementation was linked in the paper or abstract. Release status.
AMF has not reproduced PSA’s benchmarks or tested the proposed procedure. Our supplier example is hypothetical, and the scoring approach needs adaptation to the task. Reviewer judgments can disagree, and a reference evidence set can itself be incomplete.
More control over evidence processing introduces more choices to get wrong. An overly aggressive filter can discard an exception; a concise extraction can remove uncertainty. If executable actions are introduced, review the allowed operations, access boundaries, resource limits, and recovery behavior before using them with organizational data. This article is not a security assessment.
A straightforward lookup may be adequately served by a simpler search interface. Conversely, repeatedly comparing conditions across long documents is a sensible place to investigate evidence handling. Choose based on the observed failure and the cost of the proposed repair.
Next action
Take one recent weak answer and reconstruct three things: what was found, what the model saw, and what the answer claimed. If the logs cannot distinguish them, improve that visibility first.
The next decision should follow the trace: improve discovery, improve evidence delivery, or improve answer verification. That gives the team a specific hypothesis to test before replacing the agent or buying a stronger model.
Provenance
Prepared for Agent Model Fit with AI-assisted primary-source inspection, analysis, and drafting on October 14, 2026. Primary-source claims and release status were rechecked with AI assistance on October 17, including clarification of evidence tracing and token accounting. Research claims are attributed to Qian and colleagues’ paper; the diagnostic procedure, evaluation criteria, and supplier scenario are AMF proposals. No independent PSA execution, measured business outcome, or cost saving is claimed.
Human editorial approval is recorded separately against the exact manuscript revision before publication.