llama.cpp and adjacent runtimes are rapidly adding model support, quantization, MTP, decision models, and memory optimizations for local LLMs. New open model releases and specialized runtimes are extending the practical reach of self-hosted AI.
Aleph Alpha publishes the Kolibri-1 card: 78 billion total parameters, 3.46 billion active, and a 1 million token context under Apache 2.0. AI-RADAR analysis shows the constraint shifting from compute to memory and KV cache, with direct implications ...
Ultra-specialized runtimes like Strata and ninfer give up the generality of llama.cpp and vLLM to optimize a few models and one hardware family. The bet: maximum output from fixed local configurations, even at the cost of disposable code. For self-ho...
Aleph Alpha releases Kolibri-1, an LLM with 78 billion parameters, 3.46 billion active per token, and up to 1 million tokens of context under Apache 2.0. The combination touches three constraints of self-hosted deployments: memory, compute, and licen...
A pull request in the llama.cpp repository introduces an optimization that halves the indexer score memory for Qwen Flash Next. The change, reported on Reddit, results in lower VRAM usage during inference. For on-premise and self-hosted deployments, ...
The llama.cpp project has announced a new feature called Decision Models, signaling an expansion beyond pure text generation. For those running Large Language Models locally, the move suggests a maturation toward more actionable outputs, with implica...
Pull request #29818 by ngxson introduces /v1/systemone in the llama.cpp server and references five GGUF models: Laya, Julia-1, Lev, OpenJev, and Kev-4B. No technical details are provided, but for self-hosted deployments the expanding API surface rais...
A Pi.dev extension lets users skip the reasoning phase of a locally running Qwen 27B with llama.cpp, using the same native mechanism as the web interface. /skip-reasoning or Alt+T produces output immediately, but the author warns that skipping reason...
PR #29761 from am17an brings MTP support to Qwen Flash Next in llama.cpp, with GGUF quantizations already available on Hugging Face. A signal for local-stack teams: self-hosted ecosystems are shortening the path from model release to practical adopti...
A pull request reopens MTP integration for Qwen Flash Next in llama.cpp, with GGUF quantizations available on Hugging Face. The work is still WIP; for local LLM operators it becomes a test bed for reducing latency without hardware changes, but with a...
As reported, Ling-3.1-flash combines 560 billion total parameters, 25 billion active per token, and a context window up to one million tokens, with a two-week free trial and then an open-source release. The profile points to a sparse architecture: lo...
Pull request #27773 by timkhronos adds GLM-5.3-Flash (GLM5-Next) support to llama.cpp. The model can now run on home computers without cloud services. A signal for on-premise deployments: open source runtimes are consolidating as a distribution chann...
ISTA DASLab released quantized GGUFs for Qwen3.8-Flash-Next and a Coder variant with half the experts removed. The GSQ and RCO combination brings the resident working set to 29.6 GB, within a 32 GB accelerator, with benchmarks near or above BF16. A s...
A user reports that a Qwen 3.8 Flash Next model on dual RTX 3090s outperformed GPT-5.6 Luna on a real project after being connected to Codex CLI. The result reignites the debate about which layer matters most in local deployment: not just the model, ...
