Benchmarks for AI scientists have so far rewarded the right answer and ignored the path. OpenDiscoveryTrace flips the perspective: 558 complete agent trajectories, recorded step by step, with thoughts, tool calls, observations, errors, revision triggers, and self-reported confidence. The dataset spans 124 scientific tasks across drug discovery, materials science, genomics, and scientific literature analysis. It covers seven models: GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro with 124 trajectories each, plus four open-weight models (Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B) with 30 each, and 60 live-retrieval variant trajectories.

A pilot analysis on 363 LLM-judged trajectories shows something output-only benchmarks cannot see. The three frontier models achieve comparable success rates, between 84 and 89 percent. Yet Claude Opus 4.6 produces 30 times more errors per trajectory than GPT-5.4: 2.5 versus 0.08. That is not a statistical detail; it is the difference between a system that accumulates visible, correctable errors and one that appears cleaner only because its failed steps do not surface in final-output evaluation.

For teams managing LLMs in controlled environments, process traceability matters beyond research. A scientific agent working with sensitive data or intellectual property cannot be judged on final results alone: you need to know whether it followed a verifiable method, used tools as expected, and revised wrong steps. In an on-premise deployment, where logs stay under the organization's control, datasets like OpenDiscoveryTrace offer a reference for building process-level evaluations without sending data to external services. At AI-RADAR, those evaluating on-premise deployments can find analytical frameworks at /llm-onpremise to compare these trade-offs.

The inclusion of open-weight models is the most relevant signal for local deployment. The smaller models have only 30 trajectories each, but their presence shifts attention away from a comparison of large vendors alone. For a team evaluating a model to run on its own infrastructure, being able to inspect the traces of a Qwen or a Phi model becomes a concrete selection criterion: what matters is not just how many tasks it solves, but how it solves them and how often it corrects itself.

One tension remains: judging traces with an LLM adds a second layer of subjectivity. But that is also the point: if a process must be auditable, the judge needs to see not only the answer, but the path that produced it. OpenDiscoveryTrace turns that path into public data, not a black box.