The report comes from a homelabber: two GeForce RTX 3090 cards, paired with a second-generation Intel Optane build, are enough to load and run a Kimi AI model with 2.8 trillion parameters. The reported throughput is 3.11 tokens per second. It is not a number that competes with datacenters, but the signal it sends is more interesting than the figure itself.

In a traditional inference scenario, the weights of an LLM must fit in GPU VRAM. When a model exceeds the combined capacity of the cards, the only paths are shrinking the checkpoint or spreading work across multiple nodes. Here, instead, a memory hierarchy comes into play: parameters that are not active at a given moment can live on low-latency storage such as Optane, while GPUs process only the portion needed for each token. It is an approach that sacrifices throughput but dramatically lowers the hardware threshold for experimenting with extremely large models.

The result, 3.11 tokens per second, is slow for a conversational assistant but may make sense for batch jobs, evaluations, or quality tests on local datasets. The structural point is that the entry cost for self-hosted inference is no longer dictated solely by total VRAM. It also depends on the ability to orchestrate volatile and persistent memory, and Optane has a specific role here because of its lower latency compared with traditional NAND SSDs.

This shifts the incentives for organizations that cannot or do not want to send data to a cloud API. A team with data sovereignty requirements can evaluate an on-premise system built on consumer GPUs and memory tiering, accepting high latency in exchange for full control. TCO must be calculated carefully: initial CapEx can be low, especially with used hardware, but electricity consumption, maintenance, and pipeline development time can erode the benefits.

At the market level, experiments like this signal that the frontier of the largest models is no longer monopolized by datacenter GPU vendors. Inference software developers will increasingly need to account for heterogeneous configurations where VRAM, system RAM, and persistent storage work together. This does not necessarily weaken cloud providers, but it could increase competitive pressure on pricing for sporadic workloads.

For those evaluating on-premise deployment, there are trade-offs between upfront costs, energy consumption, and operational complexity; AI-RADAR provides analytical frameworks at /llm-onpremise to help navigate them. A clear limit remains: 3.11 tokens per second does not replace cloud infrastructure for production. But as a boundary experiment, it shows that the distance between a homelab and a datacenter is now measured in memory trade-offs, not just in the number of GPUs.