Techniques like 1-bit and 2-bit quantization, algebraic-integer compression and KV-cache SVD are shrinking LLMs to fit modest hardware while preserving accuracy.
NVIDIA's Nemotron Bolt 3.5 74B A9B arrives Oct 15 as a 74B-parameter, 9B-active MoE compressed to 24GB using 2-bit quantization with quantization-aware training. It targets speed and intelligence balance; a 4GB E4B model follows Oct 23.
OpenBMB has released MiniCPM-V 4.7, a 35B-A3B multimodal model and the first MoE in the series to run smoothly on-device. Combining Lookahead Expert Scheduling, layer-wise quantization, and speculative sampling, it claims 13+ tokens/s on iPhone 17 Pr...
Precision Numerics debuts with Boole-1.7B, a 1-bit Qwen3-1.7B model at 247 MB packed weights, 13.9 times smaller than BF16. It retains 85.9% of the BF16 average score across ten tasks, outperforms 1-bit Bonsai-1.7B, and nearly matches the 1.58-bit ve...
A new compression format for LLMs uses algebraic integers instead of the usual floating-point or fixed-point formats. The research shows competitive accuracy with the MX format at equal bit-per-parameter budgets. For self-hosted inference, the potent...
A shared learning rate can manufacture false advantages when comparing fine-tuning recipes with different parameter counts. In SVD-based KV-cache compression, healing only the encoder appeared superior, but with per-arm learning rates it reaches pari...
The JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF collection highlights a compact MoE design with roughly 2.5 billion active parameters and GGUF distribution. For on-prem teams, this mix lowers compute per token, supports quantization on consumer hardw...
An arXiv preprint describes LittleBit, an ultra-low-bit quantization approach based on latent factorization and quantization-aware training for very small models. The news matters for on-premise LLM operators: it is not just about saving space, but a...
