Tiny, MoE, and ultra-low-bit models are lowering hardware requirements for on-device and on-premise AI. Aggressive quantization, however, is also surfacing reliability problems such as repetitive loops and reasoning drift.
The JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF collection highlights a compact MoE design with roughly 2.5 billion active parameters and GGUF distribution. For on-prem teams, this mix lowers compute per token, supports quantization on consumer hardw...
An arXiv preprint describes LittleBit, an ultra-low-bit quantization approach based on latent factorization and quantization-aware training for very small models. The news matters for on-premise LLM operators: it is not just about saving space, but a...
The LiquidAI/d1-3B page on Hugging Face, shared without technical details, places the project in the compact LLM segment. The 3B label suggests a model aimed at local inference, self-hosted setups, and data sovereignty constraints. The article analys...
A 19,969-parameter, 81 KB FP32 model generates short stories with goal, problem, and resolution while running on CPU. MacroStories lowers the bound for coherent narrative generation and questions assumptions about local inference.
TinyDecide is a 10-million-parameter model with a roughly 6 MB footprint, smaller than every entry on the Decision Index leaderboard. It runs in the browser, Node.js, Python, Rust, and on an ESP32. For edge and on-premise deployments, the signal is n...
A refactoring task on an RTX 5070 Ti with Qwen3.8-27B in IQ3_XXS quantization produces a biographical digression about Lee Kuan Yew inside the reasoning. The episode shows how aggressive compression and speed can erode internal coherence, making trac...
A field test with IQ3_S quantization shows that the same exact model can produce a coherent answer in llama.cpp and a repetitive loop elsewhere. The report challenges the idea that choosing the right model is enough: runtime, sampling and scheduling ...
