Two RTX 3090s on PCIe 3.0, a pair of Xeon E5-2696 v4 CPUs, and 188 GB of DDR4-2133: on paper this looks like a workstation many would have written off. Instead, using llama.cpp and the UD-Q6_K_XL quantization of Qwen3.8-Flash-Next, the author of a technical post lifted decode from about 17 tokens/s to 25-29 tokens/s on short and medium contexts, and from 12 to roughly 17 tokens/s at 131,000 tokens of depth. The full context remains 261,888 tokens with f16 KV.

The key change is PR #27861, a GPU-resident LRU expert cache. Rather than loading whole expert layers into VRAM, the runtime caches recently used experts per layer. Because the model tends to pick the same experts for a token as it did for the preceding few dozen tokens, the hit rate reaches 80-85% on code and higher on prose. The 48 expert layers stay pinned in system RAM: the GPU becomes a fast cache, not the archive.

The other decisive move was lowering ubatch from 2048 to 512. Compute buffers scale with ubatch, and this frees roughly 5 GB per GPU, taking the cache from 80 to 135 slots per layer at full context. The cost is slower prefill on long prompts; short prompts remain essentially unaffected.

This shifts the center of gravity of the problem. In MoE models, VRAM is no longer the only feasibility indicator: the memory hierarchy and the expert cache policy matter just as much. A system with saturated PCIe 3.0 and slow DDR4 can still improve measurably if the software avoids re-reading experts from RAM when they are already in GPU memory. The second-order implications are concrete for anyone managing on-prem clusters with older hardware: the marginal cost of keeping two 3090s alive drops, while the TCO of newer machines must be justified by workloads dominated by prefill or continuous training.

It is not a magic bullet. The author notes that thread count, poll, CPU masks, q8 KV, lazy PLE, and n-gram drafts on prose either did nothing or hurt. Multi-token prediction at temperature 0.7 is counterproductive because verification batches re-read experts from host RAM; it only wins in greedy mode or at deep context. More than two cache uploads per step saturate PCIe 3.0 and cause the hit rate to collapse.

To replicate, no fork is needed: just llama.cpp master plus PR 27861. The cache size should be calibrated to the VRAM left after KV and compute buffers; on Q6 the author estimates about 100 MB per slot per GPU. The next declared test is UD-Q4_K_XL quantization, followed by a possible return to MTP. More than a single trick, this case signals a direction: MoE inference is becoming a caching and scheduling problem, not only a capacity problem. In that space, the headroom for older on-prem hardware is wider than single-GPU benchmarks suggested. For those evaluating on-prem deployment, AI-RADAR collects analytical tools on /llm-onpremise to weigh these trade-offs.