When people discuss what can run locally, parameter count tends to dominate. But anyone working with long contexts soon discovers that parameter count is not the whole story. Once an LLM fits into VRAM, the hidden cost becomes the KV cache: for every token generated, the model must keep key and value states available. At 100k or 200k tokens of context, a machine that comfortably holds the weights can start to struggle, not because the model is too large, but because working memory is saturated.

The mechanism is familiar to inference developers. The KV cache grows linearly with context: it is not a one-time overhead, but persistent state that follows each generation. GQA and MQA reduce the number of heads that keep these states, and quantization of the cache lowers the bits per element. These are useful mitigations, but they do not change the nature of the problem: more context means more memory. For local models, this means the constraint is no longer just VRAM capacity, but the ability to manage persistent memory and data movement.

This shifts the center of optimization. If the bottleneck is the state to maintain rather than the weights, it becomes rational to design models that remember more selectively. The question is no longer how many billions of parameters are needed, but how much a model must remember to sustain a long context without exhausting memory. It is a subtle but structural difference: it separates the cost of knowledge from the cost of a session.

For on-premise deployment, the consequence is immediate. Loading a model is one thing; serving multiple long-context sessions is another. Each session keeps its own KV cache, so total VRAM must hold weights, activations, and cache. With long contexts, a single request can saturate available space or force a reduction in batch size. In many cases the choice is not between larger models, but between long context, cache precision, and parallelism. For those evaluating on-premise deployment, AI-RADAR offers analytical frameworks at /llm-onpremise to weigh these trade-offs without reducing them to a question of raw VRAM.

The open-model industry may therefore shift its metrics: a model with fewer parameters but a more compact KV cache can be more useful on local hardware than one with twice the parameters and the same bottleneck. Hardware vendors will also see growing interest in bandwidth and memory capacity, not just peak compute. Anyone designing inference infrastructure would do well to treat the KV cache as a first-class operational cost, because that is where long contexts turn a sufficient GPU into a machine at its limit.