The striking number isn't the token throughput as much as the fact that a machine like this exists in a basement. A user on Reddit described a local inference setup for deepseek-v4-flash-0731: an Epyc 7663 processor, 256 GB of ECC DDR4-3200 memory, and a single RTX 5090 with 32 GB of VRAM. The combination, which the author called "relatively low rent", hosts a model whose weights take up roughly 151 GB.

With Q8_K_XL quantization, generation runs between 23.8 and 24.6 tokens/s. Prompt processing starts around 60 tokens/s on the first prompt and climbs to 385 tokens/s as the cache fills on tasks using 100-128k tokens of total context. The user also reported two regressions: DFlash made the setup slower, and temporarily adding an RTX 3090 did not help.

The interesting point is that a 32 GB GPU does not hold the entire model. In a Mixture-of-Experts architecture, the system can split work between CPU and GPU; the CPU with a large amount of ECC RAM handles part of the experts or layers, while the GPU accelerates what sits in VRAM. This challenges a common assumption: for models of this scale, organizations don't necessarily need a fleet of GPUs with aggregate VRAM equal to the weight count. System memory capacity and memory bandwidth become just as important. In this specific case, DDR4-3200 on an Epyc platform is not the highest-bandwidth option, and the throughput remains workstation-class rather than production service. But it is enough for personal use or for testing very long contexts.

That has second-order implications for anyone evaluating on-premises deployment. If a 151 GB model can run in a basement without a dedicated electrical panel and without more than $10,000 in accelerators, the marginal cost of data sovereignty drops. This isn't about building a data center; it's about accepting higher latency and lower token throughput in exchange for control over data. That trade-off matters for small teams, labs, and companies subject to data residency requirements. Cloud API providers that serve intermittent workloads could lose exactly the users who previously had no local option for models this large.

A structural limit remains: memory bandwidth and PCIe transfer do not disappear. The Blackwell RTX 5090 accelerates, but it cannot fully erase the distance between system RAM and GPU. The fact that an RTX 3090 made performance worse suggests the balance between compute and memory is delicate: adding any GPU is not enough. The relevant variable is the CPU-RAM-accelerator combination rather than the individual card.

In TCO terms, the calculation goes beyond component cost. A machine like this consumes energy, requires abundant ECC RAM, and needs maintenance. Still, the entry cost for self-hosted inference of a large LLM appears lower than many deployment plans assume. For those evaluating on-premises deployments, AI-RADAR offers analytical frameworks at /llm-onpremise to compare these trade-offs: memory capacity, bandwidth, energy use, and integration complexity.