From FP8 KV cache support in llama.cpp to MoE expert caching and aggressive quantization, local inference is shrinking VRAM requirements. These techniques extend context lengths and keep larger models usable on consumer hardware.
llama.cpp has added support for an FP8 key-value cache, reducing VRAM usage during local LLM inference. For teams evaluating self-hosted deployments, the immediate benefit is longer contexts on existing hardware, but the change also points to a struc...
A European startup asks for insider information about upcoming Gemma models and describes a real deployment: Gemma 4 31B with NVFP4 quantization on three RTX Pro 6000 cards, used for language reasoning in German and English. The focus is not coding b...
A benchmark across 35 weight quantizations with llama.cpp, ExLlamaV3 and vLLM measures the KL divergence introduced by quantized KV caches. At 8-bit, memory drops by about 1.9-2 GiB at 64k tokens and the effect on token distribution is nearly zero. 4...
A pull request in llama.cpp introduces llama_moe_cache, a cache for MoE experts executed on CPU, plus a tuning script. The goal is to accelerate MoE offloading for users with limited VRAM, an approach similar to tuning mechanisms seen in Strata. No b...
