Recent advances in llama.cpp, including massive speedups on Intel GPUs and multi-token prediction support, together with a wave of new open-weight MoE and ternary models, are making high-quality local LLM inference more accessible and efficient.
A handful of code lines in llama.cpp multiplies Intel Battlemage GPU performance up to 2.7 times on 118K-token contexts with quantized KV cache. The gain is so sharp that it turns a gaming card into a tangible alternative for local inference of 35B m...
A llama.cpp PR switches the SYCL dispatch for quantized KV caches from VEC to TILE kernel, yielding up to 169% faster decode on Intel Battlemage GPUs at long contexts. Gains are substantial even at 32K, pointing to cost-effective on-premise deploymen...
Alibaba launched Qwen3.8-Max, a 2.4-trillion-parameter MoE model, while DeepSeek offers V4-Flash at $0.14 per million input tokens. Both release weights under open licenses, enabling on-premise deployment. Task cost hinges on token generation and int...
A user’s updated local benchmark places the MXFP4-quantized DeepSeek V4 Flash 0731 at the top for efficiency and quality, delivering 1,000 tokens/sec prefill and 90 tokens/sec generation. The result shines a spotlight on low-precision quantization an...
Maple-Preview applies ternary weights to a 20B-parameter model, shrinking memory to ~5 GB while activating only 1B parameters per token via an MoE-like design. This preview hints at local inference on consumer GPUs, but framework support remains imma...
Maple-Preview is a new open-weight model with 20 billion total parameters, only 1 billion active per token, and ternary weight quantization. The combination slashes VRAM requirements for inference, bringing on-premise deployment within reach of consu...
InclusionAI has released Ling-3.0-flash weights on Hugging Face. The model packs 127.5 billion parameters with 512 experts, 8 active per token. The real story is the official FP8 version: 128GB, enough for a single unified-memory system or a multi-GP...
InclusionAI released Ling-3.0-flash, an open-weight MoE with 124 billion total parameters but only 5 billion active per token. Announced before the Kimi K3 and DeepSeek-V4-Flash wave, its sizing could carve a niche in on-premise deployment, where eff...
A pull request eliminates the CPU-GPU round-trip for MTP sampling in llama.cpp. On an RTX 5090 the gain reaches nearly 8%, while on a Tesla P40 it’s limited to around 4% due to memory bandwidth. A pure performance uplift with zero extra cost for loca...
A Reddit question about moving to llama.cpp reveals much more than a UI switch: it’s the moment when local inference graduates from individual tinkering to enterprise-ready stacks built on control, reproducibility, and automation.
The runtime that powers millions of local LLMs adds multi-token prediction support for DeepSeek's MoE model, slashing latency and reinforcing the case for self-hosted as a mature alternative to the cloud.
A Reddit PSA warns DeepSeek-V4-Flash users: the chat template hoists every system message to the top, breaking prompt caching and driving up costs and latency for both self-hosted and hosted deployments. The fix: use the latest_reminder role, which D...
