llama.cpp 1.0, KV cache compression, MoE caching and AI-assisted ports are turning local LLM execution into a stable, lower-VRAM platform. New tools also improve GPU/VRAM visibility and distribute workloads across CPU, GPU and NPU.
AMD's HRX Loom and Ryzen AI Max experiment shows that moving 50 LLMs to client GPUs in three days is not just a hardware feat. The real signal is the shift of value toward middleware and porting pipelines, with effects on self-hosted setups, data sov...
Together AI introduces TKCC, a continuous compilation system built on agents for inference. It tackles the measurement throughput-fidelity trade-off with a tiered persistent engine and a cost model. In tests it outperforms the best open source engine...
An AMD team ported 50 trending Hugging Face models to client GPUs in about three days, using an AI agent workflow and HRX Loom v0.1 software on Ryzen AI Max 'Strix Halo' hardware. The experiment signals that the barrier to local inference is no longe...
llama.cpp releases version 1.0: the runtime for running LLMs locally becomes a stable foundation for downstream applications. Hugging Face also announces desktop apps to reduce local AI access to a few clicks. A maturity step that strengthens the sel...
A two-person team integrated SGLang with IBM Spyre in under a week using an AI-assisted plugin of about 1,500 lines. SGLang achieved roughly 3.6× the throughput of the reference PyTorch inference implementation and performed on par with vLLM on measu...
The Cyanogen co-founder introduces Kora, a Linux operating system designed to split AI model execution across CPU, GPU, and NPU. The announcement signals a structural shift: using the entire local hardware instead of concentrating inference on a sing...
llama.cpp has added support for an FP8 key-value cache, reducing VRAM usage during local LLM inference. For teams evaluating self-hosted deployments, the immediate benefit is longer contexts on existing hardware, but the change also points to a struc...
A new open-source tool promises to show exactly which processes consume VRAM. For teams running LLMs locally, this visibility changes how they address bottlenecks, size hardware, and calculate real resource costs. It's not just debugging—it's the fou...
A benchmark across 35 weight quantizations with llama.cpp, ExLlamaV3 and vLLM measures the KL divergence introduced by quantized KV caches. At 8-bit, memory drops by about 1.9-2 GiB at 64k tokens and the effect on token distribution is nearly zero. 4...
A pull request in llama.cpp introduces llama_moe_cache, a cache for MoE experts executed on CPU, plus a tuning script. The goal is to accelerate MoE offloading for users with limited VRAM, an approach similar to tuning mechanisms seen in Strata. No b...
