Mixture-of-experts architectures, low-bit quantization, and reduced active parameters are making large models deployable on-device and on-premise. Releases span vision-language, coding, and long-context models, plus practical apps like receipt scanning.
A Reddit experiment shows that replacing the heavily quantized PLE in common GGUFs with a full FP8 sidecar improves the perceived quality of Qwen 3.8 Flash Next, with no speed impact. Only disk usage increases.
The release of Dory1.0-4B-BF16 shows how BF16 precision redefines local deployment. It is not just a small model: about 8 GB of weights shift attention to VRAM, TCO, and quantization. For teams with workstations and small servers, the signal is clear...
A 4-billion-parameter BF16 model from NVIDIA Nemotron Labs appears on Hugging Face. The size matters less than the precision: BF16 keeps it away from aggressive quantization and places it in the local deployment space on GPUs with enough VRAM. We loo...
The ORL (On-Policy Residual Learning) project aims to speed up diffusion LLMs without sacrificing quality. On a single RTX 5090 the team reports 2,788 tokens/s with LLaDA-2.1-mini in NVFP4; the open-source dlmrt inference engine uses CUDA graphs and ...
Reflection has released Beam, an open LLM with a 501-billion total and 23-billion active parameter MoE architecture. It supports a 1 million token context and BF16, FP8, and NVFP4 precision. The focus is token-efficient reasoning: about one third of ...
The model card for Tencent Hy-Memory-30B-A3B sparks debate over a slippery claim: an architecture with 3 billion active parameters out of 30 billion total would beat Opus 5 on an unspecified memory test. The issue is not the benchmark but the structu...
A three-day empirical test of GYRO-M, a Q2 quantization of Qwen Flash Next, reports q3-level stability at reduced size. The project comes from a single contributor and holds up on long contexts, but shows weaknesses on Cyrillic text. The case reopens...
NVIDIA positions Nemotron 3.5 Bolt as a vision-language model for everyday agentic tasks running locally: 74B total parameters, 9B active, native 2-bit quantization, and a 24GB footprint. It supports text, images, video, and up to 256K tokens of cont...
Microsoft AI has developed a local version of MAI Code 1.1 Flash, a coding-optimized mixture-of-experts model with 137 billion total and 6.8 billion active parameters. The news points to a push toward self-hosted development tools on Windows, with im...
inclusionAI's Ling-3.1-flash appears on Hugging Face with 560 billion total parameters, 25 billion active parameters per token, and a 1-million-token context window. The mixture-of-experts configuration reduces per-token compute, but shifts the focus...
NVIDIA's Nemotron Bolt 3.5 74B A9B arrives Oct 15 as a 74B-parameter, 9B-active MoE compressed to 24GB using 2-bit quantization with quantization-aware training. It targets speed and intelligence balance; a 4GB E4B model follows Oct 23.
LiquidAI has introduced LFM2.5-24B-A2B, a reasoning model with 24 billion total parameters and 2.45 billion active. Built for private on-device agents, it claims to compete with Qwen3.6-35B-A3B and run up to 1.7 times faster than comparable MoEs, wit...
OpenBMB has released MiniCPM-V 4.7, a 35B-A3B multimodal model and the first MoE in the series to run smoothly on-device. Combining Lookahead Expert Scheduling, layer-wise quantization, and speculative sampling, it claims 13+ tokens/s on iPhone 17 Pr...
Precision Numerics debuts with Boole-1.7B, a 1-bit Qwen3-1.7B model at 247 MB packed weights, 13.9 times smaller than BF16. It retains 85.9% of the BF16 average score across ten tasks, outperforms 1-bit Bonsai-1.7B, and nearly matches the 1.58-bit ve...
A 0.8B model reading receipts on the phone shows on-device inference as a concrete option for personal documents. The project combines fine-tuning on donated data, JSON output and a self-hosted release valve via an OpenAI-compatible server. For those...
A developer built a free Android app that uses a 0.8-billion-parameter vision model to read receipt photos and extract store, date, total, and line items into JSON. The model, fine-tuned on receipts, runs on the phone without sending data to third pa...
A technical report on Kairos combines linear attention, sliding-window attention, and MoE in a byte-level multimodal model. The authors check only convergence on a single 440-byte sentence, not generalization. The 200M/25M parameter target is sized a...
A new compression format for LLMs uses algebraic integers instead of the usual floating-point or fixed-point formats. The research shows competitive accuracy with the MX format at equal bit-per-parameter budgets. For self-hosted inference, the potent...
