Nicknamed Tequila after its creator’s hostname, a PC built around a 16 GB NVIDIA GeForce RTX 4060 Ti, a 6-core AMD Ryzen 5 9600X, and 138 GB of DDR5 RAM is running a 98 GB model at speeds that seemed unattainable just a week ago: 7.44 tokens per second, up from 2 t/s. The model is DeepSeek V4 Flash, compressed with the most aggressive quantization (Q2_K_XL) into a single GGUF file, and the leap is courtesy of llama.cpp’s developers, who between builds b9986 and b10034 squeezed out optimizations that wring every CPU cycle.

The configuration is a blueprint for DIY on-premise deployment. The model doesn’t fit into the GPU’s 16 GB of VRAM, so generation happens almost entirely on the CPU, with model layers split between GPU and system RAM. The context window is a generous 131,072 tokens, and the inference engine uses multiple threads (16 for generation, 20 for batch processing) alongside flash attention. Prompt evaluation on 14 tokens reached 15.24 t/s, but the headline is sustained output on a 12,000-token generation: 7.44 t/s with an average latency of 134 milliseconds per token.

A 300% boost—from 2 to 7 t/s—is more than a lab curiosity. It pushes CPU inference of massive models to near-interactive levels, suitable for chatbots, summarization, translation, and coding assistants where perceived fluidity matters. And it does so without renting cloud GPU instances; a model of this size would typically require an A100 or H100, with monthly costs running into four figures.

The deeper point isn’t the model itself—extreme quantization sacrifices precision, and DeepSeek V4 Flash is still a niche family—but the technological trajectory. Llama.cpp is becoming the reference runtime for efficient inference on heterogeneous architectures, pushing consumer CPU and GPU combos far beyond their nominal limits. Recent optimizations touch cache management, computation graph reuse (nearly 12,000 graphs reused in the test), and thread scheduling on Zen 5 architectures, proving that software can multiply performance decisively.

For organizations weighing on-premise deployments with data sovereignty requirements, this is confirmation: you don’t need data centers full of accelerators. A well-specced desktop with generous RAM and a mid-range GPU can run large LLMs locally, keeping data safe and eliminating transfer costs. Trade-offs remain—latency is acceptable but not real-time, and the model quality hit from quantization must be assessed case by case—but the price-performance ratio is rapidly approaching that of far pricier solutions.

There’s a structural shift unfolding. Hardware becomes a commodity when software bridges the gap, and llama.cpp is doing exactly that: democratizing access to large-model inference and shifting competition from silicon to the quality of the runtime layer. In an ecosystem where models grow faster than IT budgets, the ability to run locally on existing company machines could become the deciding factor for many AI adoption projects.

For those navigating these choices, AI-RADAR offers analytical frameworks at /llm-onpremise to weigh trade-offs between GPU acceleration, CPU offloading, and quantization strategies—with compliance impact top of mind. Meanwhile, the llama.cpp community keeps "cooking," as the tester puts it, and each commit brings the dream of a personal, powerful, fully offline AI closer.