Who&When benchmark tests AI agents’ ability to diagnose failure
Illustration, not documentary evidence of the event.
Researchers from Penn State University, Duke University and collaborating institutions introduced Who&When, a benchmark for identifying which AI agent caused a task to fail and where the decisive mistake occurred. A June 16, 2025 research account published by syncedreview.com described failure logs from 127 multi-agent systems, with human annotations naming the responsible agent, the error step and an explanation.
The account reported that the paper had been accepted as an ICML 2025 Spotlight presentation and that its code and dataset were publicly available. These are claims from the supplied research account, rather than independently verified results.
The reported accuracy gives developers a reason to keep human review in the loop. The account puts the best single-method results at about 53.5% for identifying the responsible agent and 14.2% for locating the exact error step. Experiments primarily used GPT-4o and tested attribution both with and without access to the correct answer to the underlying task. The account does not specify which setting produced those headline figures, limiting their usefulness as an estimate of performance on a particular workflow.
The methods also serve different debugging needs. All-at-Once examines the complete log in one pass and performed better at identifying the agent. Step-by-Step checks the sequence incrementally and was better at locating the mistake, but incurred greater cost and risked accumulating errors. Binary Search repeatedly narrows the search to a portion of the log, offering an intermediate tradeoff. Combining methods improved results at additional computational cost, according to the account.
For a team choosing an approach, this suggests that the first decision is what the diagnosis must accomplish. Directing a developer toward a suspect agent requires less precision than identifying the exact action to change. The reported gap between those tasks makes automated attribution easier to justify as an aid to investigation than as sufficient evidence for an automatic repair.
Evaluation conditions deserve equal attention. A team investigating tasks without known correct answers should prioritize the benchmark’s corresponding results. The account also reports declining accuracy as logs grow longer, especially for identifying the error step. A useful local evaluation would therefore measure whether the tool finds actionable mistakes in the team’s own long traces, and whether the review time saved justifies the added computational cost.
Correction, September 19, 2026: This article was revised to attribute reported claims, remove unsupported statements and clarify the limits of the available evidence.