The Airbench benchmark, posted on Reddit by user dh7net, puts a hard number on something that until recently seemed unlikely: a single RTX 5090 with aggressive quantization and the qwen3.8-flash-next-iq3_xxs-strata model completes a coding task on opencode in 7 minutes 41 seconds with 96% success. Claude Code Opus 5.5, the cloud reference, takes 14 minutes 26 seconds for 100%. That four-point gap buys almost half the time.

The picture broadens. The same source reports that zcode on four RTX 6k cards with glm-5.3-flash-NVFP4 hits 100% in 31 minutes 13 seconds: same quality as cloud, a little more than double the time. If a small error margin is acceptable, DSHv0.2rc2 on an RTX PRO 6000 WS with qwen3.8-flash-next-NVFP4 reaches 98% in 23 minutes 30 seconds.

The hidden variable is the harness

But the most consequential result isn't about the model. On local hardware, the harness matters as much as the model. Testing the same strata quantization on the same RTX 5090 produces scores from 22% to 96% depending on the harness. That is an enormous spread, and it makes any comparison that doesn't spell out the full stack—GPU, quantization, harness, and context—unusable.

To compare fairly, the author uses three models that all five harnesses ran on the same RTX 5090: opencode averages 94%, omp 91%, pi 71%, hermes 67%, openclaw 56%. opencode and omp stay above 85% regardless of model. pi is unstable; hermes can score well but is slow, with many local runs exceeding 1h20 and some hitting the 2-hour cap.

Architecture and context choices also matter. MTP variants lose ground every time: nvfp4 averages 52% with MTP versus 70% without it; on pi, swift drops from 96% to 55% with MTP. A 65k context window also hurts, with scores falling between 45% and 61%. Gemma-4-26b is fast but tops out at 47%.

The same variability appears in the cloud. With deepseek-v4.1-flash, omp, pi and opencode all score 98%, while hermes gets 82%. So this isn't just a local hardware problem: the harness introduces a systemic variable that can distort any benchmark.

For anyone evaluating on-premise deployment, this changes the order of questions. It's no longer enough to ask whether an open-weight model can match a proprietary one; you have to ask which harness drives it, with which quantization and which context window. A company moving a coding agent to local GPUs for data sovereignty reasons isn't just buying an architecture—it inherits integration and validation work. The same GPU can be a bottleneck or a competitive asset depending on how it's orchestrated.

For those weighing on-premise deployment, AI-RADAR maps trade-offs between control, TCO and performance in its analytical frameworks on /llm-onpremise.

The Airbench project invites contributions, but login is mandatory and configuration details are left to a notes field: two limits that show how hard it is to build reliable benchmarks without systematic control of variables. The challenge is not only technical, it's also methodological.