Llama.cpp just received a small but significant tweak that delights anyone serving models on their own hardware. Pull request #25532 (the number appears at the end of the original post) moves MTP sampling – the multi-token speculative decoding technique – from the CPU to the GPU. In tests with a recent consumer-grade card, an RTX 5090, the measured token-per-second improvement reaches 8% on Qwen3.6:35B. On an aging Tesla P40, still widespread in homelabs and some refurbished enterprise machines, the gain drops to 4%, yet the report’s author still describes it as the largest speed bump they’ve seen in a while.

The gap between the two GPUs is instructive. The P40 (Pascal architecture, compute capability sm_61) has 580 GB/s memory bandwidth, while the 5090 reaches 1,792 GB/s. With the IQ4_NL-quantized Qwen3.6-35B-A3B model, decode time on the P40 is dominated by weight transfers from VRAM to the cores: the CPU↔GPU round-trip for sampling was only a fraction of total time. Offloading it entirely to the GPU removes that cost, but the net acceleration is smaller than on the 5090, where the far larger bandwidth made the CPU↔GPU delay proportionally heavier. The key fact: acceptance ratio stays identical, so there’s no quality regression.

The P40 test: numbers and context

The side-by-side comparison published by the user puts CPU sampling and GPU sampling through the same benchmark sessions. With CPU sampling, the P40 ran between 50 and 76 tokens/s depending on the task (Python code, concept explanations, summaries, translations, stepwise math). Moving sampling to the GPU lifts the same hardware to 52–80 tokens/s, with an observed peak of 84 tokens/s. The absolute gain is roughly 2 tokens/s, which adds up over long workloads.

For the on-premise world, this PR signals a clear direction: community inference frameworks are squeezing every millisecond by moving the final stages of the pipeline – stages that remained on the CPU for years – onto the GPU. It’s not just a free one-off optimization; it’s a structural sign that competition in pure efficiency is shifting from models to runtimes. While commercial solutions sometimes struggle to keep up with the heterogeneous GPUs found in enterprise fleets, projects like llama.cpp keep delivering incremental improvements that translate into time savings and a smoother experience for those running internal chatbots, confidential document analysis, or air-gapped environments.

Perhaps the most telling aspect isn’t the percentage gain itself, but the fact that the community is refining MTP speculative decoding even on older hardware. On a P40, the change eases friction between generation stages without requiring any hardware upgrade. For teams evaluating on-premise deployment with mixed-generation GPU machines, this means a simple runtime update can extend the useful life of existing cards and defer capital spending.

And as new GPU bandwidth continues to climb, the relative weight of control operations still left to the CPU will become ever more visible. With this change, llama.cpp previews a future where the entire inference pipeline – from weight loading to token sampling – runs as a GPU monolith, further lowering the Total Cost of Ownership for self-hosted setups.