Running a cutting-edge LLM in a home lab is always a calculated gamble. This time the testbed is Kimi K3, a still sparsely documented model, deployed on a system with 768GB of DDR5 RAM and two NVIDIA 5090 GPUs. Reddit user iVoider published the first results, using a dedicated llama.cpp fork and Q2_K quantization of the model.

The most striking figure is the gap between prefill and decoding: 50-70 tokens/s for processing long prompts, versus a generation speed that hovers around 4 tokens/s. In an on-premise context with consumer hardware, these numbers signal a successful compromise: the ability to run a model designed for far more expensive infrastructure, accepting a just-barely-interactive throughput suitable for chat or non-urgent batch jobs.

But it’s the decoding behavior over time that deserves attention. The user notes that tokens/s increase progressively during generation — a detail that goes beyond anecdote and points to a precise technical dynamic: hot cache. Most likely, llama.cpp’s offloading mechanism shifts part of the KV cache between VRAM and system RAM, and the initial tokens pay a heavy swap penalty. As the cache stabilizes, memory requests become more local and DDR5 bandwidth, although slow compared to GPU HBM, starts being used more efficiently.

This phenomenon illuminates a structural issue for anyone evaluating on-premise deployment of large LLMs. Memory capacity alone isn’t enough: bandwidth determines the latency of each generated token. In hybrid setups like this one, where the combined VRAM of the two 5090s is still limited, system RAM acts as an extension but pays a penalty in effective speed. The observed acceleration is a side effect of a hierarchical memory architecture still not optimized for continuous inference.

The experiment, in its simplicity, sends a dual signal. On one hand, it confirms that the pace of development of tools like llama.cpp is chipping away at the competitive advantage of enterprise systems: with a few thousand euros’ worth of cards and plenty of DDR5, one can explore models that until yesterday seemed the exclusive domain of corporate clusters. On the other, it puts the spotlight back on data sovereignty and infrastructure control: an organization wanting to keep everything on-premise, perhaps due to GDPR constraints, must reckon with borderline speeds and the need to build serving pipelines tolerant of variable latency.

Kimi K3 in a home lab isn’t a lab benchmark but a real-world proof of concept. The next challenge won’t just be squeezing more tokens/s out of DDR5, but managing the decoding acceleration in a deterministic way, perhaps with schedulers that exploit cache warm-up to pipeline consecutive requests more effectively. Until then, those pushing on-premise deployment will have to live with a decode that starts sluggish and then picks up pace — a small technical puzzle that reveals how young the self-hosting AI ecosystem still is.

The experiment confirms that extreme quantization (Q2_K) is an indispensable ally for lowering the memory barrier, while also reminding us that model quality can suffer under complex workloads. It’s a trade-off familiar to those tracking local deployment, made tangible here by concrete numbers.