Techniques such as MoE expert caching, fluid prefill-decode scheduling, algebraic compression and 1-bit quantization are reducing the hardware thresholds for local and on-prem LLMs.
Precision Numerics debuts with Boole-1.7B, a 1-bit Qwen3-1.7B model at 247 MB packed weights, 13.9 times smaller than BF16. It retains 85.9% of the BF16 average score across ten tasks, outperforms 1-bit Bonsai-1.7B, and nearly matches the 1.58-bit ve...
A new compression format for LLMs uses algebraic integers instead of the usual floating-point or fixed-point formats. The research shows competitive accuracy with the MX format at equal bit-per-parameter budgets. For self-hosted inference, the potent...
The JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF collection highlights a compact MoE design with roughly 2.5 billion active parameters and GGUF distribution. For on-prem teams, this mix lowers compute per token, supports quantization on consumer hardw...
An arXiv preprint describes LittleBit, an ultra-low-bit quantization approach based on latent factorization and quantization-aware training for very small models. The news matters for on-premise LLM operators: it is not just about saving space, but a...
A pull request in llama.cpp proposes a GPU cache for Mixture of Experts model components kept in system RAM, aiming to speed up inference when the full model doesn't fit in VRAM. The idea targets users with limited GPU memory and could shift the bott...
A pull request in the llama.cpp repository introduces support for GLM5Next MTP; a Reddit post confirms that GLM 5 Flash MTP can be run locally. For teams evaluating self-hosted deployment, the move widens the set of models available without cloud, wi...
FluidPD addresses prefill-decode imbalance in LLM serving without adding workers. FluidToken offloads part of prefill work to decode workers when slack is available; FluidRole reassigns running workers between roles without reloading the model. On Az...
