llama.cpp and the Evolution of On-Premise LLM Deployments

The landscape of Large Language Models (LLM) is constantly evolving, with growing interest in on-premise deployment solutions that ensure greater control, data sovereignty, and cost optimization. In this context, frameworks like llama.cpp are establishing themselves as pillars for the efficient execution of LLMs on local hardware. A recent benchmark highlighted the capabilities of llama.cpp build b9455b, demonstrating a significant performance leap on a common hardware configuration for industry professionals: two NVIDIA RTX 3090 GPUs.

These results are particularly relevant for CTOs, DevOps leads, and infrastructure architects evaluating self-hosted alternatives to cloud services. The ability to run complex models like the Qwen3.6-27B-UD-Q8_K_XL, a 27-billion-parameter model with Q8_K_XL quantization, on high-end consumer hardware, opens new perspectives for implementing sensitive AI workloads or those with specific latency and throughput requirements.

Technical Details and Impressive Performance

The test showcased the performance of llama.cpp build b9455b, which integrates advanced features such as tensor-split and flash-attn acceleration, in addition to speculative decoding (via draft-mtp). On a configuration with two NVIDIA RTX 3090s, the framework achieved a decoding speed exceeding 70 tokens/second, with peaks of 81 tokens/second. This represents a significant improvement over previous iterations of llama.cpp, which ranged between 30 and 50 tokens/second.

The tensor-split, configured with a 50,50 division, allowed for effective distribution of the model's load across the two GPUs, fully utilizing the available VRAM. Prefill performance was also notable, with speeds exceeding 1400 tokens/second in various scenarios. The model was configured to handle a large context window of 262144 tokens, an increasingly common requirement for applications that need to process large amounts of text. The adoption of a quantized KV cache (q8_0) further contributes to memory usage efficiency.

Implications for On-Premise Deployments and Trade-offs

These results position llama.cpp as an increasingly strong competitor against other LLM serving solutions, such as vLLM, which previously held an advantage in terms of throughput on multi-GPU configurations. Although vLLM had previously achieved similar performance (over 70 tokens/second), the user noted a higher quality in the code output generated by the Qwen3.6-27B-UD-Q8_K_XL model running with llama.cpp, calling it a "different beast altogether." This suggests that optimizations are not only about speed but also about the fidelity and reliability of the model's output.

However, the benchmark also highlights a crucial trade-off: prefill latency for extremely large contexts. While prefill speeds are high, processing a 100,000-token context can take approximately 60 seconds. This aspect is critical for interactive applications or those requiring rapid responses to large inputs, and must be carefully considered in deployment architecture design. For those evaluating on-premise deployments, analytical frameworks on /llm-onpremise can help assess these trade-offs in terms of TCO and specific requirements.

Future Prospects and Final Considerations

Continued innovation in frameworks like llama.cpp is vital for the widespread adoption of LLMs in on-premise environments. The ability to achieve high performance on accessible hardware, such as the RTX 3090s, democratizes access to advanced AI capabilities, reducing reliance on cloud services and strengthening data sovereignty. Companies can thus maintain full control over their models and sensitive data, a crucial aspect for regulated industries or those operating in air-gapped environments.

The balance between decoding throughput, prefill latency, and VRAM requirements remains a constant challenge. Advances in tensor-split, flash-attn, and speculative decoding demonstrate that software optimization can unlock significant potential even on existing hardware. For technical decision-makers, it is essential to monitor these evolutions to build AI infrastructures that are resilient, efficient, and compliant with their operational and business needs.