llama.cpp updates, specialized runtimes, low-bit quantization, and serving improvements are lowering the cost of self-hosted inference.
The JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF collection highlights a compact MoE design with roughly 2.5 billion active parameters and GGUF distribution. For on-prem teams, this mix lowers compute per token, supports quantization on consumer hardw...
A title and a link are enough to mark a shift in perception. The C/C++ runtime for local LLM inference is no longer confined to GitHub repositories: its public stage legitimizes self-hosted deployment as a strategic choice for TCO and data sovereignt...
A pull request in llama.cpp proposes a GPU cache for Mixture of Experts model components kept in system RAM, aiming to speed up inference when the full model doesn't fit in VRAM. The idea targets users with limited GPU memory and could shift the bott...
A pull request in the llama.cpp repository introduces support for GLM5Next MTP; a Reddit post confirms that GLM 5 Flash MTP can be run locally. For teams evaluating self-hosted deployment, the move widens the set of models available without cloud, wi...
FluidPD addresses prefill-decode imbalance in LLM serving without adding workers. FluidToken offloads part of prefill work to decode workers when slack is available; FluidRole reassigns running workers between roles without reloading the model. On Az...
Unofficial throughput figures attributed to Azure Foundry suggest GPT-6.1 Sol would emit a token with two inference passes instead of three. Data collected on artificialanalysis.com feed the hypothesis of recurrent and nested model architectures. For...
The local inference runtime llama.cpp reaches version 0.6.0, adding MTP speculative decoding support for Qwen4Exp models. The move strengthens the self-hosted LLM path, where hardware control and pipeline ownership matter more than API convenience.
A refactoring task on an RTX 5070 Ti with Qwen3.8-27B in IQ3_XXS quantization produces a biographical digression about Lee Kuan Yew inside the reasoning. The episode shows how aggressive compression and speed can erode internal coherence, making trac...
An RTX 5070 Ti user running a Qwen3.8-27B model quantized to IQ3_XXS through Strata reports a 2-3x speed gain, but also an odd reasoning artifact: during a code cleanup task the model inserted a biography of Lee Kuan Yew. The case highlights how aggr...
A field test with IQ3_S quantization shows that the same exact model can produce a coherent answer in llama.cpp and a repetitive loop elsewhere. The report challenges the idea that choosing the right model is enough: runtime, sampling and scheduling ...
Ultra-specialized runtimes like Strata and ninfer give up the generality of llama.cpp and vLLM to optimize a few models and one hardware family. The bet: maximum output from fixed local configurations, even at the cost of disposable code. For self-ho...
A pull request in the llama.cpp repository introduces an optimization that halves the indexer score memory for Qwen Flash Next. The change, reported on Reddit, results in lower VRAM usage during inference. For on-premise and self-hosted deployments, ...
PR #29761 from am17an brings MTP support to Qwen Flash Next in llama.cpp, with GGUF quantizations already available on Hugging Face. A signal for local-stack teams: self-hosted ecosystems are shortening the path from model release to practical adopti...
A pull request reopens MTP integration for Qwen Flash Next in llama.cpp, with GGUF quantizations available on Hugging Face. The work is still WIP; for local LLM operators it becomes a test bed for reducing latency without hardware changes, but with a...
Hugging Face has open-sourced a collection of WebGPU kernels for more than 200 common machine learning operations, all running entirely in the browser. The company aims to integrate these optimizations into Transformers.js, ONNX Runtime Web, and Lite...
Pull request #27773 by timkhronos adds GLM-5.3-Flash (GLM5-Next) support to llama.cpp. The model can now run on home computers without cloud services. A signal for on-premise deployments: open source runtimes are consolidating as a distribution chann...
ISTA DASLab released quantized GGUFs for Qwen3.8-Flash-Next and a Coder variant with half the experts removed. The GSQ and RCO combination brings the resident working set to 29.6 GB, within a 32 GB accelerator, with benchmarks near or above BF16. A s...
The open-source runtime for local LLM inference brings a change that makes prompt lookup drafting 42 times faster. This is not full generation speed, but a key component of speculative decoding: for self-hosted deployments, it can cut overhead and TC...
A pull request in llama.cpp introduces a tiled matrix-vector multiplication for k-quant formats. According to the report, using VNNI instructions speeds up CPU mul_mat by 3-7x with minimal added complexity. The improvement targets prompt prefill, mak...
A sparse Reddit note points to Oh My Pi and custom model support through vLLM, llama.cpp, and SGLang. The real story is structural: self-hosted inference still depends on multiple runtimes with distinct hardware profiles, and integration—not a single...
