Benchmarks do not stop being useful when they saturate: they change nature. Researchers retrospectively simulated saturation using historical model and benchmark releases, and checked whether complete transcripts of correctly solved tasks could predict capabilities on unsaturated items. The answer is yes, with an important caveat: the signal must be extracted with robust protocols, not with a simple average of scores.

The work compares three approaches. The first is pairwise judging, which directly compares two transcripts; the second assigns an absolute score to a single transcript; the third uses deterministic features or embeddings of transcripts. Among them, pairwise judging reduces prediction error by 5.3 times compared with a baseline that uses only saturated benchmark scores. That is not a marginal difference: it suggests execution traces contain information not captured by the final score.

The most interesting finding is not just that the signal exists, but where it concentrates. The authors observe that capability-relevant information is distributed across transcript sections, but clusters in specific channels such as reasoning steps and tool calls. This is consistent with model evolution: as scores flatten, the difference between systems emerges in the process rather than in the final answer.

For teams running self-hosted models, this changes the value of inference logs. Cloud platforms often return aggregate metrics; an on-premise deployment keeps the full trace, from generated tokens to tool-call sequences. This is not about collecting more data for its own sake: the research shows that those logs, when queried with appropriate protocols, can separate models that a leaderboard no longer distinguishes. This tension between public evaluation and local control is part of the trade-offs AI-RADAR examines on /llm-onpremise.

But there is a limit that deserves attention. The protocols do not always generalise across distribution shifts. The study notes that the first reasoning models introduced in early 2025 were underestimated. That detail matters: an evaluation based on historical transcripts risks rewarding a well-established reasoning style and penalising new capabilities expressed in different forms. This shifts the problem from collecting the signal to the robustness of extraction methods.

The second-order implications are just as relevant. If transcripts become an evaluation source, incentives change for model builders: optimising for a final score is no longer enough if traces are analysed in detail. At the same time, pairwise judging has non-trivial computational and operational costs, because it requires multiple comparisons and can introduce biases linked to the model used as judge. Companies evaluating models for regulated contexts or safety cases must treat evaluation as a pipeline, not as a single number.

Who wins and who loses? Those who control raw execution data gain an edge, typically teams operating on-premise or in hybrid environments with complete logs. Those who rely only on public leaderboards and have no access to full traces lose some signal. For vendors, the message is twofold: saturated benchmarks are not dead, but value shifts from scores to process metadata; at the same time, models introducing new paradigms may be poorly assessed by protocols trained on the past.

It is not a comfortable conclusion. Extending the useful life of benchmarks is not automatic: it requires designing extraction methods that hold up under distribution shifts. The research leaves a clear indication: transcripts retain predictive signal beyond accuracy, but reading that signal reliably is the next problem to solve.