llama.cpp and speculative decoding methods are reducing memory use for KV caches and MoE offloading, allowing longer contexts on limited GPUs. Reproducibility and VRAM visibility are becoming central issues for local inference.
llama.cpp has added support for an FP8 key-value cache, reducing VRAM usage during local LLM inference. For teams evaluating self-hosted deployments, the immediate benefit is longer contexts on existing hardware, but the change also points to a struc...
A Hugging Face repository attributed to Tencent circulates the hypothesis of a new speculative decoding method capable of tripling decoding speed for 27B models. The unconfirmed report refocuses attention on a technique that reduces latency without a...
A new open-source tool promises to show exactly which processes consume VRAM. For teams running LLMs locally, this visibility changes how they address bottlenecks, size hardware, and calculate real resource costs. It's not just debugging—it's the fou...
A benchmark across 35 weight quantizations with llama.cpp, ExLlamaV3 and vLLM measures the KL divergence introduced by quantized KV caches. At 8-bit, memory drops by about 1.9-2 GiB at 64k tokens and the effect on token distribution is nearly zero. 4...
A pull request in llama.cpp introduces llama_moe_cache, a cache for MoE experts executed on CPU, plus a tuning script. The goal is to accelerate MoE offloading for users with limited VRAM, an approach similar to tuning mechanisms seen in Strata. No b...
A single-model multi-agent framework shares one backbone and coordinates GPU-resident KV states through a directed acyclic graph. Zero-copy radix forking, attention-only KV mounting, and packed DAG-GRPO training cut peak KV memory by 2.8x and raise t...
An anecdotal report about a Q4 model running with llama.cpp on an RTX 4090 reopens the reproducibility problem in self-hosted deployments: without hashes and pinned versions, local control remains incomplete.
