The silent bet of "outdated" hardware

Two RTX 3090s, saturated PCIe 3.0, DDR4-2133 and a pair of Xeon E5-2696 v4: on paper, a workstation many teams would decommission. Instead, the numbers reported in a recent technical post tell a different story. With llama.cpp and the UD-Q6_K_XL quantization of Qwen3.8-Flash-Next, decode moves from about 17 tokens/s to 25-29 tokens/s on short and medium contexts. At 131,000 tokens of depth, the improvement is equally visible: from 12 to about 17 tokens/s. Full context remains 261,888 tokens, with KV in f16.

The figure is not just about acceleration. It is about the fact that the jump arrives without replacing GPUs, without moving to PCIe 4.0 and without upgrading RAM. It arrives from a software change: llama.cpp PR #27861 introduces an LRU cache of experts resident in GPU. This is a perspective shift that directly touches the core of on-premise strategies.

For AI-RADAR this is a signal to isolate. Not because two 3090s suddenly become a training platform, but because MoE inference feasibility moves from pure VRAM inventory to intelligent management of the memory hierarchy. And in that game, older hardware has wider margins than single-GPU benchmarks suggested.

The starting point is an architecture with 48 expert layers pinned in system RAM. The GPU does not hold the full model: it holds a fast cache. When the runtime guesses which experts are needed, inference accelerates. When it misses, it pays the cost of re-reading them from host RAM. The difference between these two states is a new design factor.

Why expert caching changes the rules

PR #27861 does not load whole expert layers into VRAM. It caches the most recently used experts for each layer. The bet is simple: an MoE model tends to choose for a token the same experts used for previous tokens. The reported data points to a success rate of 80-85 percent on code and higher on prose. When the runtime guesses right, it avoids re-reading from system RAM. When it misses, it pays a latency cost.

This mechanism has an immediate consequence: the role of VRAM changes. It is no longer a complete archive, but a cache. The distinction is subtle and fundamental. An archive grows with model size. A cache grows with access locality. If accesses are predictable, a small cache can produce a disproportionate gain. If they are random, capacity is needed. The Qwen3.8-Flash-Next case shows that, at least on code and prose, locality exists.

A second decisive step is lowering ubatch from 2048 to 512. Compute buffers scale with ubatch. Reducing it frees about 5 GB per GPU. The cache grows from 80 to 135 slots per layer at full context. The price is slower prefill on long prompts. Short prompts remain essentially unchanged. It is an explicit trade-off: favour continuous decode over initial processing of large contexts.

Not all parameters respond the same way. The author reports that thread count, poll, CPU masks, KV q8, lazy PLE and n-gram drafts on prose produced no benefit or made things worse. Multi-token prediction at temperature 0.7 is counterproductive because speculative verification batches re-read experts from host RAM. It works only in greedy mode or at deep context. More than two cache uploads per step saturate PCIe 3.0 and make the success rate collapse.

The centre of gravity shifts: not just capacity, but scheduling

The result moves the centre of gravity of the problem. In MoE models, VRAM is no longer the only feasibility indicator. Memory hierarchy and cache policy matter just as much. A system with saturated PCIe 3.0 and slow DDR4 can still improve measurably, if the software avoids re-reading from RAM the experts already present in GPU. This is a lesson valid for many on-premise deployments.

The second-order implications are concrete for teams managing clusters with older hardware. The marginal cost of keeping two 3090s alive drops. The TCO of newer machines must be justified with workloads where prefill or continuous training dominate. If the main use case is decode on long and predictable contexts, investing in modern GPUs is not automatically the most rational choice.

There is a further nuance: expert caching rewards workloads with high temporal locality. Code and prose are two examples. Other workloads may behave differently. A generalist serving system, with prompts varying widely by domain and style, could have lower success rates. Adopting this technique is not neutral: it must be calibrated on the real nature of traffic.

The boundary between caching and scheduling becomes porous. The LRU cache implicitly decides which experts stay in GPU and which are re-read. But it plans nothing in advance. A future step could be a predictive policy using recent history to preload likely experts before the next step. That is not what PR #27861 does today, but it is a direction the case makes natural.

Who benefits and who loses

The immediate beneficiaries are teams with workstations or small servers equipped with previous-generation consumer GPUs. Two RTX 3090s on PCIe 3.0 are not only for experiments: they can sustain fluid decode on an MoE model with a large context. The entry cost for self-hosted inference drops. Data sovereignty becomes more accessible, because there is no need to rent last-generation cloud GPUs.

Teams that need to extend the useful life of existing hardware also benefit. In many enterprise contexts, hardware refresh is a slow process. Being able to improve inference through a software change reduces budget pressure. It is not a magic wand: prefill on long prompts worsens, and some advanced techniques stop working. But for a class of workloads, the gain is real.

Those who built their sizing on total VRAM alone lose. If expert caching becomes common practice, MoE models may be evaluated differently. A GPU with less VRAM but a solid interconnect and a well-managed cache can beat a GPU with more memory but less mature software. The "GB of VRAM" metric does not disappear, but it stops being the only yardstick.

Vendors pushing new hardware as the only solution for MoE inference also lose. The narrative "you need the latest GPU" weakens. It remains true for continuous training, massive prefill or very large dense models. But for MoE decode on shared hardware, software-defined inference changes the equation. TCO shifts: less hardware, more software engineering.

The broader picture: inference becomes a caching problem

The Qwen3.8-Flash-Next case signals a direction. MoE inference is becoming a caching and scheduling problem, not just a capacity problem. It is a theme crossing the whole AI infrastructure field: from KV caches to prefix caching, from scheduling queues to preemption policies. PR #27861 adds a specific piece: expert caching.

The concept is not new in systems history. Memory hierarchy has been at the core of computing architectures for decades: registers, L1, L2, L3 caches, RAM, storage. Bringing this scheme into MoE inference means applying to model weights the same logic applied to data. Frequently used experts sit close to the GPU. The others wait in RAM. Efficiency depends on the quality of access prediction.

This opens second- and third-order questions. Why not use larger caches, perhaps across multiple GPUs? Why not share the cache across concurrent requests, increasing the success rate? Why not expose the cache policy as an API, so that applications can suggest which experts to keep in GPU? These hypotheses are not present in the source, but they are logical consequences of a moving field.

There is also a reading for the open-source world. llama.cpp master and PR #27861 are sufficient to replicate the result. No fork is needed. This lowers the experimentation barrier. Teams wanting to test the technique on their own workstation can do so quickly. Iteration speed itself becomes a competitive advantage for on-premise teams.

What to watch now

The next declared test is UD-Q4_K_XL quantization, followed by a possible return to multi-token prediction. Q4 reduces the VRAM footprint and could further increase the number of cache slots per layer. But it introduces a risk: model quality loss. The balance between larger cache and weight precision is one of the variables to monitor.

A second signal to observe is cache size. The author estimates about 100 MB per slot per GPU on Q6. Calibration depends on residual VRAM after KV and compute buffers. On different configurations, the optimal size changes. There is no universal value. This makes the technique closer to a tuning art than to a standard option.

A third signal is interconnect evolution. On saturated PCIe 3.0, more than two cache uploads per step make the success rate collapse. On PCIe 4.0 or 5.0, that limit could move. But even with faster interconnects, re-reading from RAM remains a cost. The memory hierarchy will not disappear: its tolerance to cache misses will simply increase.

Finally, it is worth watching whether other frameworks adopt similar techniques. If LRU expert caching became a standard option in more runtimes, MoE inference on older hardware would normalize. In that case, the competitive advantage would shift from simple adoption to fine-tuning cache policies for specific workloads. For those evaluating on-premise deployments, AI-RADAR will continue to collect analytical tools on /llm-onpremise to weigh these trade-offs.