Sixteen gigabytes of VRAM, a graphics card anyone can install in a desktop case, and a 30-billion-parameter language model churning out text at over 50 tokens per second. That is not the distant future but the result of the latest experiment by Reddit user Azazelionide: the handcrafted engine dubbed garlic-inference managed to run Qwen3-30B-A3B on an RTX 5060 Ti with performance that beats llama.cpp by 50%. The boost comes from custom CUDA kernels and C++ code, inspired by solutions presented at conferences like NeurIPS, ICML, and EuroSys.

The model’s Mixture-of-Experts architecture, with only 3 billion active parameters per token out of a total of 30, is what makes the feat possible. Without MoE, a 30B LLM in float8 would easily eat up most of the available memory, throttling speed. Here, the actual per-token computation is constrained, and the bottleneck shifts to how well the runtime orchestrates expert routing and data transfers between VRAM and compute units. That is exactly where garlic-inference gains ground, targeting inefficiencies with highly specialized code.

The most significant aspect is not the raw number, but the signal it sends to anyone watching the hardware landscape for on-premise deployment. The RTX 5060 Ti, with its 16 GB of memory, is not an enterprise workstation card; it’s built for gaming and small-scale development. Yet paired with a non-generic inference engine, it delivers latency comparable to what we would have associated only a few months ago with data-center GPUs sporting four times the memory. This means the gap between consumer hardware and professional solutions is narrowing faster than vendors would like to admit, and that the TCO equation may shift in favor of self-hosted setups much earlier than expected.

Still, one must ask who can truly benefit from such an optimization. The garlic-inference project is a proof of concept: it lacks enterprise support, stability guarantees, and integration with production pipelines. For a company evaluating whether to bring inference in house, relying on code not battle-tested at scale is often an unacceptable risk. But the direction is unmistakable: local solutions are maturing to the point where workloads can be pulled from the cloud without sacrificing user experience. Data sovereignty, GDPR compliance, and cost predictability then become tangible arguments, not just principles.

The existence of an engine like garlic-inference also reopens the debate between generalist frameworks and vertical implementations. llama.cpp remains the default choice thanks to its flexibility and vast community, but a 50% gap achieved with custom code hints that there is still ample room to squeeze consumer architectures further. It’s not hard to imagine the coming months sparking an optimization race similar to what we saw in the mobile world after TensorFlow Lite and Core ML arrived: whoever ships a lean, high-performance runtime first wins over developers who don’t want to send their data to the cloud.

In the short term, the losers are cloud services that sell inference as a commodity at high margins. If a single person with a card costing a few hundred euros can handle a 30B model with perfectly smooth interactivity, the datacenter’s added value shrinks to edge cases: much larger models, online fine-tuning, or orchestration at scale. For the rest, local compute becomes not only an ethical option but a decision of plain economic sense. The bet, of course, is that consumer-grade hardware will keep evolving with a favorable price-performance ratio, and that open-source communities will find ways to bring these optimizations out of the garage and into tools anyone can use.