A wave of open-weight LLMs, from Meta's Muse Glimmer and Qwen 3.8 to quantized Kimi K3, Ling and Nemotron variants, is making self-hosted AI viable. Community formats like GGUF, MLX and on-device runtimes are central to this shift.
A Reddit megathread aggregated official links and quantized variants for the new Qwen 3.8 27B on release day. GGUF, MLX, and FP8 builds were already available, highlighting the maturity of local inference ecosystems and the shift toward deployment-ce...
The appearance of the deepseek-ai/DeepSeek-V4-Pro-0813 repository on Hugging Face puts LLM distribution back in focus. Without technical details, however, an identifier does not guide on-premise decisions: VRAM, quantization, serving pipelines, and d...
Hugging Face shows an official countdown for Qwen/Qwen3.8-27B, suggesting a pre-release phase. The move signals a community-driven distribution strategy and gives on-premise teams a window to assess VRAM constraints, quantization, and TCO before avai...
The new Nemotron-3.5 Lightning 30B-A3B in BF16 arrives on Hugging Face. This move shifts the focus toward ultra-efficient MoE architectures, designed for those who run LLMs on their own hardware, cutting cloud dependency without sacrificing performan...
InclusionAI releases Ling-3.0-tiny, an 8B-parameter MoE with just 1.3B active, reaching 100-105 tokens/s on DGX Spark and 86-90 on an M4 Pro MacBook, with a peak memory of 8.34 GiB at 8K context in FP8. Performance sits between 4B and 8-12B dense mod...
Meta releases Muse Glimmer, a 30-billion-parameter model optimized for on-device agentic workflows. ExecuTorch now supports inference on NVIDIA GPUs and Apple silicon Macs, integrating GGUF, k-quant, 128K+ token context, and DFlash speculative decodi...
The arrival of Muse-Glimmer-30B in GGUF format isn't just a model release. It signals a strengthening local inference infrastructure, shifting focus from cloud behemoths to on-premises GPU practicality and sovereign data pipelines.
Meta releases Muse Glimmer, a 30-billion-parameter LLM under Apache 2.0 license, optimized for local AI agents on consumer GPUs. Quantized to 4-bit, the model requires under 20 GB, targeting on-device tasks with personal data access. Benchmarks show ...
Meta releases Muse Glimmer, a 30B parameter model under Apache 2.0, optimized for always-on agentic workflows on consumer hardware. 4-bit quantization and speculative decoding allow it to fit within 24 GB GPUs, enabling local autonomous agents with m...
The official INT4 of Ling-3.0-flash runs on a single DGX Spark, but the default configuration delivers only 20.8 tok/s. Two changes — dropping `--enforce-eager` to enable CUDA graphs and switching on MTP speculative decode with one token — push it to...
A Hugging Face user has released a quantized Kimi K3 variant that strips multilingual support and applies IQ2-XXS via Unsloth, shrinking the footprint from 711GB to 478GB. Behind the technical move lies a clear lesson for on-premise evaluators: cutti...
A user runs Kimi K3 across two clusters using llama.cpp over RPC, with partial offloading and IQ1_M quantization, aiming for a 2-3x speedup by consolidating GPUs into a single system and reaching Q2_K_XL — all while hoping that models like Qwen3.8 wi...
A power user pushes a 32GB AMD Radeon AI Pro R9700 to its limits with INT4-quantized Qwen3.6 models. The 35B MoE achieves 262,144 token context and 52 tok/s at 100k depth, while the 27B leverages speculative decoding to sustain 59 tok/s past 50k. The...
