Three open-weight models — GPT-OSS 120B, DeepSeek-V4 Flash, and Gemma-4 31B — have achieved a result that reshapes the landscape of automatic evaluation. On a 200-instance validation sample of IMO-GradingBench, their pass/fail decisions are statistically indistinguishable from those of Claude Opus 4.7 and Gemini 3.1 Pro, yet cost up to 100 times less.

The experiment deals with grading natural-language mathematical proofs, a recurring expense in assessing math-reasoning systems. But the implications extend beyond cost savings: it paves the way for a judging infrastructure that can be replicated locally, without depending on third-party APIs and with the ability to keep data under one’s own control.

The team initially expected a majority vote of the three cheap models to be the optimal budget option. The majority indeed matched the frontier models’ performance, but didn’t improve on the strongest single member. Only by extending the analysis to the full 1000-instance benchmark and enforcing a unanimous voting rule (all three must pass) did the combination reach the highest overall agreement and precision. Across four replicate runs, the run-to-run spread was smallest, signaling a consistency others lacked.

An important caveat: the unanimity rule was identified post-hoc and requires independent replication before becoming an engineering standard. Nevertheless, the direction is clear. Reliable judgments on complex tasks no longer require the firepower of a hundred-billion-parameter model hosted on inaccessible infrastructure.

For those designing on-premise evaluation pipelines, this changes the equation. Gemma-4 31B, with appropriate quantization, can be served on a single consumer GPU with 24 GB of VRAM, while DeepSeek-V4 Flash and GPT-OSS 120B require more memory but remain manageable on servers with 4-8 mid-range GPUs. Total Cost of Ownership plummets when you avoid the monthly bill of API calls and keep the entire workflow within your own network perimeter. In settings where data sovereignty is non-negotiable — research institutes, university departments, math competitions, companies with stringent GDPR requirements — having a local, economically sustainable, and transparent judge is no longer a luxury.

Structurally, the experiment signals an acceleration in the commoditization of AI judging. Where privileged access to top-tier models was once necessary to validate math-reasoning systems, anyone can now assemble a panel of open judges and obtain competitive results. This shifts leverage away from cloud vendors toward those who own the models and the hardware, reshaping incentives for benchmark and evaluation tooling developers.

There’s also a second-order effect on scientific reproducibility. With low-cost, self-hosted judges, replicating evaluation experiments becomes feasible for labs on limited budgets, removing the excuse of prohibitive API costs. The community can adopt uniform, verifiable, and independent judging protocols, strengthening model comparisons.

The takeaway for those architecting local stacks is stark: cost-effectiveness no longer conflicts with performance. The hardware needed to run these models is already present in many enterprise data centers, and using a metric like unanimous-vote precision provides a simple operational parameter to implement and monitor.