llama.cpp and specialized runtimes are rapidly evolving with MTP speculative decoding, MoE GPU caches, smaller index memory, new endpoints, and Decision Models. The self-hosted inference stack is becoming faster and more feature-complete beyond pure text generation.
A title and a link are enough to mark a shift in perception. The C/C++ runtime for local LLM inference is no longer confined to GitHub repositories: its public stage legitimizes self-hosted deployment as a strategic choice for TCO and data sovereignt...
A pull request in llama.cpp proposes a GPU cache for Mixture of Experts model components kept in system RAM, aiming to speed up inference when the full model doesn't fit in VRAM. The idea targets users with limited GPU memory and could shift the bott...
A pull request in the llama.cpp repository introduces support for GLM5Next MTP; a Reddit post confirms that GLM 5 Flash MTP can be run locally. For teams evaluating self-hosted deployment, the move widens the set of models available without cloud, wi...
The local inference runtime llama.cpp reaches version 0.6.0, adding MTP speculative decoding support for Qwen4Exp models. The move strengthens the self-hosted LLM path, where hardware control and pipeline ownership matter more than API convenience.
A field test with IQ3_S quantization shows that the same exact model can produce a coherent answer in llama.cpp and a repetitive loop elsewhere. The report challenges the idea that choosing the right model is enough: runtime, sampling and scheduling ...
Ultra-specialized runtimes like Strata and ninfer give up the generality of llama.cpp and vLLM to optimize a few models and one hardware family. The bet: maximum output from fixed local configurations, even at the cost of disposable code. For self-ho...
A pull request in the llama.cpp repository introduces an optimization that halves the indexer score memory for Qwen Flash Next. The change, reported on Reddit, results in lower VRAM usage during inference. For on-premise and self-hosted deployments, ...
The llama.cpp project has announced a new feature called Decision Models, signaling an expansion beyond pure text generation. For those running Large Language Models locally, the move suggests a maturation toward more actionable outputs, with implica...
Pull request #29818 by ngxson introduces /v1/systemone in the llama.cpp server and references five GGUF models: Laya, Julia-1, Lev, OpenJev, and Kev-4B. No technical details are provided, but for self-hosted deployments the expanding API surface rais...
A Pi.dev extension lets users skip the reasoning phase of a locally running Qwen 27B with llama.cpp, using the same native mechanism as the web interface. /skip-reasoning or Alt+T produces output immediately, but the author warns that skipping reason...
PR #29761 from am17an brings MTP support to Qwen Flash Next in llama.cpp, with GGUF quantizations already available on Hugging Face. A signal for local-stack teams: self-hosted ecosystems are shortening the path from model release to practical adopti...
