The problem is not whether an LLM agent produces a correct answer, but whether it does so for the right reason and without unapproved steps. The trajectory-judge study starts from a deterministic support-desk environment with tools, a scripted oracle policy that always solves it, and a fault injector that breaks exactly one step at a known point. Across 400 trajectories, five judges—programmatic rules, an outcome-only judge, step-rubric judges at two model sizes, and a self-consistency ensemble—were scored on detection, step localisation, fault typing, calibration, and cost.

The numbers make the blind spot concrete. The outcome-only judge catches 84% of loud faults, those that change the customer-visible answer, but only 45% of silent faults, which leave the outcome unchanged. Meanwhile, it flags 33% of correct trajectories as errors. The step-rubric judge reaches 77% silent recall with zero false alarms, but at 3x the cost.

The distinction between loud and silent faults changes the incentive for anyone evaluating agents. A team using outcome-only metrics can report a high success rate without noticing that the agent used an unauthorised path or ignored an intermediate constraint. In on-premise deployment, where access to sensitive data and compliance matter, this is not a minor detail: a silent fault can be an unnecessary read of personal data or a call to an unexpected service, even when the customer receives the correct answer. The report also shows that judge calibration is uneven: the outcome-only judge produces false positives on one third of correct trajectories, eroding trust in the evaluation system itself.

The final reply remains invisible

The most severe finding concerns the final reply. No judge actually reads it: an invented promise appended to an otherwise perfect trajectory evades the rules entirely and the step judge in 82% of cases. This shifts the problem from trajectory evaluation to the real content that reaches the customer. An agent can complete all expected steps, satisfy the rubric, and then promise the customer something that will not be kept. That is not an execution error but a final-generation error, and current judges do not catch it. The self-consistency ensemble triples the cost while improving nothing, a sign that sampling redundancy does not replace a rubric that looks at content.

What changes for on-premise deployment evaluation

For those evaluating on-premise deployment, the study offers a rare artifact: the environment, the injector, all raw verdicts, and an analysis pipeline that rebuilds every number offline. This matters for data sovereignty because it allows the evaluation to be repeated on local data without sending trajectories to external services. But it also introduces a cost trade-off: the step rubric triples cost compared with the outcome-only judge. Teams running local infrastructure can quantify that cost in TCO and decide whether silent-fault coverage justifies the investment. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these trade-offs, but the choice depends on the specific operational risk.

The structural lesson is that LLM agent evaluation cannot keep measuring only the visible answer. The distinction between silent and loud faults should become a first-class metric, and reading the final reply should become a separate control. Until judges read the content that is delivered to the customer, apparent success remains an unverified promise.