That quantization erodes the performance of large language models is hardly news. But results freshly published by a research team at the Chinese University of Hong Kong zoom in on an effect rarely discussed: cutting precision leaves encyclopedic knowledge nearly intact, while it systematically destroys agentic capabilities. For anyone managing self-hosted infrastructure, that difference rewrites the trade-off calculus entirely.

The team, which runs a small university HPC cluster, subjected the various quantized versions of Qwen 3.6 to two canonical benchmarks: GPQA Diamond for knowledge, and Terminal‑Bench 2 for agentic use. The former measures the ability to answer specialized questions (physics, chemistry, biology); the latter assesses the capacity to complete complex tasks in simulated environments, from terminal usage to multi‑step planning. The charts posted on the lab’s website speak clearly: GPQA stays flat, with negligible variation across lower‑precision formats. Terminal‑Bench 2, in contrast, plunges when one drops below FP8 representations, marking a sharp performance regression.

It is not only a matter of absolute scores. The team observed a notable gap compared to the official figures released by the model’s vendor for FP8 precision. The hypothesis is that the discrepancy arises from the timeout setting: the Harbor benchmark used by the university applies variable timeouts ranging from 10 minutes to one hour depending on the task, while Qwen’s official tests used a flat three‑hour timeout. In agentic tasks, where the model must reason through successive steps, a tighter time limit further penalizes the quantized versions, already slowed by lower computational precision. This is a critical point: in on‑premise practice, where job schedules are often constrained to tight time windows to share resources among multiple users, timeout becomes a non‑trivial design variable.

The repeatability of the tests adds further complexity. Comparing multiple runs of the same test, the researchers found wide fluctuation in the results. A lower‑precision quant can occasionally beat a higher‑precision version, if it happens to hit a particularly lucky execution. But on average, the agentic degradation remains stark. This stochastic behavior suggests that multi‑step planning tasks amplify the noise introduced by compression, making the model less reliable precisely when it needs coherence over long horizons.

For organizations evaluating on‑premise deployment, these data rewrite the vocabulary of trade‑offs. Aggressively quantizing an LLM can look like the straight path to reduce TCO, containing hardware needs. But if the goal is to build autonomous agentic pipelines—assistants that execute actions, not just answer questions—then the tolerance for degradation is extremely low. “Crystallized” knowledge is the easiest resource to preserve, while executive planning ability breaks first. System architects for self‑hosted solutions thus face a fork: invest in more VRAM to keep higher precision, or accept that agentic tasks remain unreliable and perhaps delegate them to cloud‑hosted models, partially nullifying gains in sovereignty and control.

These findings also signal a structural trajectory. As the industry shifts attention from pure chatbots to autonomous agents, the pressure on compute infrastructure becomes more selective. Quantization techniques will remain indispensable for democratizing model access, but the knowledge‑action gap shows that the real differentiator will be less about “what a model knows” and more about “what it does.” For university and corporate data centers trying to keep pace with these developments, Hong Kong’s lab has turned on a beacon: before you quantize, ask not only what you want to know, but what you want to do.

The team is now applying the same benchmark suite to GLM‑5.2 quantizations, but the process—reads the note—is very slow. A detail that on its own tells how burdensome it is to empirically validate deployment choices, and how much remains to be dug under the surface of press‑release numbers.