Accelerating On-Premise LLM Inference with Speculative Decoding

Optimizing performance for Large Language Models (LLMs) in self-hosted environments represents a strategic priority for companies aiming to maintain control over their data and infrastructure. In this context, techniques like speculative decoding emerge as promising solutions to improve inference efficiency, reducing latency and increasing throughput on dedicated hardware. A recent study explored the application of this methodology to the Gemma 4 31B model, achieving significant results in terms of speed.

Speculative decoding allows a smaller, faster model (draft model) to generate a draft output, which is then verified and corrected in parallel by a larger, more accurate model (main model). This process can drastically reduce the time required to generate responses, especially when the output is predictable. The conducted tests demonstrate how this technique can unlock significant acceleration potential for on-premise LLM deployments, a crucial aspect for CTOs and infrastructure architects.

Technical Details and Benchmark Results

The test configuration utilized an RTX 5090 GPU equipped with 32GB of VRAM, a high-end hardware component that offers high computational capabilities for AI workloads. The main model used was Gemma 4 31B UD-Q4_K_XL, which occupies approximately 18.3GB of VRAM, alongside Gemma 4 E2B UD-Q4_K_XL (4.65B) as the draft model, consuming 3.0GB of VRAM. The execution environment was based on a llama.cpp fork integrating the TurboQuant KV cache (turbo3), with a 128K token context window and Flash Attention enabled.

The benchmarks revealed an average increase of 29.0% in token generation speed, moving from a baseline of 57.17 tokens per second (t/s) to 73.73 t/s with speculative decoding. The most substantial gains were observed in scenarios with highly structured and predictable output: code generation recorded a 50.5% increase (from 57.15 t/s to 86.05 t/s), while mathematical explanations saw a 49.5% increase (from 57.45 t/s to 85.86 t/s). Even in less predictable contexts, such as Korean poetry or translation, improvements were still achieved, albeit more modest, at 9.5% and 10.7% respectively.

Optimization and Deployment Challenges

A critical aspect that emerged during testing concerns model compatibility. Initially, a mismatch in tokenizer metadata (add_bos_token) between the GGUF versions of the main and draft models forced llama.cpp into token translation mode, nullifying all performance gains and, in some cases, slowing down inference. Resolving this issue, by downloading updated GGUF model versions with correct metadata, was crucial to unlock the expected performance. This highlights the importance of accurate model version and metadata management in an on-premise deployment.

System configuration also requires attention to specific parameters. The --parallel 1 setting proved mandatory; using the default value (auto=4) for this parameter resulted in a quadruple allocation of the draft model's KV cache, consuming excess VRAM and drastically reducing speed. It was also observed that a Q4 quantized draft model (3.0GB) offers similar performance to a Q8 (4.8GB), but with lower VRAM consumption, leaving more headroom for larger contexts. It is important to note that speculative decoding is not compatible with multimodal functionalities that include vision.

Implications for On-Premise Deployments

These results have direct implications for organizations considering LLM deployment in self-hosted or air-gapped environments. The ability to achieve significant throughput increases on existing hardware translates into an improved Total Cost of Ownership (TCO) and greater operational efficiency. For CTOs and infrastructure architects, speculative decoding offers a path to maximize GPU resource utilization, reducing the need for additional hardware investments to achieve specific performance levels.

However, as highlighted by vocabulary compatibility issues and parameter management, implementing these optimizations requires deep technical understanding and careful configuration. The choice of the draft-max parameter, with draft-max 8 identified as the optimal sweet spot for mixed workloads, is an example of how fine-tuning can influence overall efficiency. For those evaluating on-premise deployments, AI-RADAR offers analytical frameworks on /llm-onpremise to assess trade-offs between performance, costs, and control, providing the necessary tools to make informed decisions in a rapidly evolving technological landscape.