Replacing an LLM call with a fast classification model looks like the right move to reduce cost and latency in agent pipelines. But a paired, self-audited evaluation across 11 decision points tells a different story: the saving can disappear when the pre-screen cost is included, and gate accuracy is not end-to-end quality.

The work starts from 7,283 base cases and 6,640 robustness variants built from 18 public sources. It compares two System-1 models: Laya, open-weight, and Jev, hosted. Across 11 tasks typical of an agent harness, Jev is significantly more accurate on 9 points, with a lead from +10.8 to +46.0 percentage points. Zero-shot model routing remains at chance level for both, and RAG relevance gating ends in a tie.

Laya's fragility emerges in the variants. When the option order is reversed, the open-weight model changes 30% of its answers. On items with a unique correct tool, accuracy drops to 31% with 50 nearest-neighbour tools, against Jev's 98%. This is not a simple average drop: it is unstable behaviour that can change routing decisions in production.

The most significant part is the authors' audit of their own pipeline. Three analysis errors and one design confound distorted the initial conclusions. The claimed saving of 23.9% drops to 4.3% when the pre-screen cost is included. Gate accuracy reported as end-to-end quality confuses 58% with the real 98%. Thresholds computed in-sample promised a 5% miss rate, but on held-out data misses reach up to 17%. A channel effect on injection false positives vanishes with channel-native content.

The structural point is that in agent pipelines the fast model is not the final product. A decision model must be evaluated across the whole decision chain, not on a single task. For those evaluating on-premise deployment, there are trade-offs between self-hosted open-weight models and hosted services: control and inspectability of an open-weight model are not enough without robustness tests that include option order, candidate count, and channel-native content. AI-RADAR offers analytical frameworks at /llm-onpremise to assess these trade-offs. The repository with cases, raw outputs, and analysis code is public, a methodological signal that should become standard practice.