Ultra-low-bit quantization, 1-bit checkpoints, MoE sparsity, and tiny CPU models are shrinking AI models for local use. The focus is on reducing memory and compute while preserving accuracy for on-device and on-prem inference.
LiquidAI has introduced LFM2.5-24B-A2B, a reasoning model with 24 billion total parameters and 2.45 billion active. Built for private on-device agents, it claims to compete with Qwen3.6-35B-A3B and run up to 1.7 times faster than comparable MoEs, wit...
OpenBMB has released MiniCPM-V 4.7, a 35B-A3B multimodal model and the first MoE in the series to run smoothly on-device. Combining Lookahead Expert Scheduling, layer-wise quantization, and speculative sampling, it claims 13+ tokens/s on iPhone 17 Pr...
Precision Numerics debuts with Boole-1.7B, a 1-bit Qwen3-1.7B model at 247 MB packed weights, 13.9 times smaller than BF16. It retains 85.9% of the BF16 average score across ten tasks, outperforms 1-bit Bonsai-1.7B, and nearly matches the 1.58-bit ve...
A new compression format for LLMs uses algebraic integers instead of the usual floating-point or fixed-point formats. The research shows competitive accuracy with the MX format at equal bit-per-parameter budgets. For self-hosted inference, the potent...
The JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF collection highlights a compact MoE design with roughly 2.5 billion active parameters and GGUF distribution. For on-prem teams, this mix lowers compute per token, supports quantization on consumer hardw...
An arXiv preprint describes LittleBit, an ultra-low-bit quantization approach based on latent factorization and quantization-aware training for very small models. The news matters for on-premise LLM operators: it is not just about saving space, but a...
The LiquidAI/d1-3B page on Hugging Face, shared without technical details, places the project in the compact LLM segment. The 3B label suggests a model aimed at local inference, self-hosted setups, and data sovereignty constraints. The article analys...
A 19,969-parameter, 81 KB FP32 model generates short stories with goal, problem, and resolution while running on CPU. MacroStories lowers the bound for coherent narrative generation and questions assumptions about local inference.
