How large is too large for your GPU? For years, the answer was: not very. Then came Mixture of Experts, 2-bit quantization, and speculative decoding. Now, with the latest llama.cpp update, the question gets even blurrier: because Tencent's Hy3, a behemoth with 299 billion total parameters, enters the local toolbelt.

If you know llama.cpp, you know it doesn't just take models and throw them onto a CPU. Over the years, the project has honed an arsenal of optimizations turning consumer hardware into credible inference platforms. But adding Hy3 is no routine integration: it's a proving ground for multi-token-prediction (MTP) used as a speculative decoding engine. The model, a Mixture of Experts with 80 layers plus one MTP layer, now runs with a pipeline where the MTP head acts as the draft model. Instead of predicting a single next token, it generates several future tokens at once, which are then verified in parallel by the main network. The result is higher throughput and lower perceived latency, without needing a separate draft model.

For on-premise deployments, the combination is potentially game-changing. With 299 billion total parameters but only a fraction active per token thanks to expert routing, Hy3 demands significant memory to hold the entire checkpoint (hundreds of gigabytes in GGUF format), yet the compute cost per inference is governed by the number of activated experts. On systems with large shared memory – such as Apple Silicon workstations with 192 GB of unified memory – or on multi-GPU setups, the model becomes manageable. The acceleration from MTP speculative decoding helps offset memory bandwidth limitations, making the model usable even where throughput would otherwise be prohibitive.

This signals a structural shift. The ability to run a cloud-scale LLM entirely on-premise erodes the advantage of centralized providers, not only on costs (TCO of dedicated hardware can undercut API calls at high volume) but especially on data sovereignty. For companies handling sensitive data under regimes like GDPR, having a viable path to air-gapped inference on frontier models is no longer science fiction. And the fact that this evolution is driven by an open-source project like llama.cpp – not a big tech vendor – confirms a trend: the primary aggregator for independent inference is becoming community code, not a paid service.

Pull request #25395, like countless others, is the product of a distributed effort that keeps chipping away at the moat of large AI providers. Hy3, with its architecture-embedded speculative decoding, is the latest brick in a wall rising around the idea of computational sovereignty.