A wave of open-weight Mixture-of-Experts models like Ling-3.0-flash, DeepSeek V4 and Qwen variants, combined with extreme quantization and ternary weights, achieve near-frontier performance on consumer GPUs and CPU systems.
A Hugging Face user has released a quantized Kimi K3 variant that strips multilingual support and applies IQ2-XXS via Unsloth, shrinking the footprint from 711GB to 478GB. Behind the technical move lies a clear lesson for on-premise evaluators: cutti...
A user runs Kimi K3 across two clusters using llama.cpp over RPC, with partial offloading and IQ1_M quantization, aiming for a 2-3x speedup by consolidating GPUs into a single system and reaching Q2_K_XL — all while hoping that models like Qwen3.8 wi...
Alibaba launched Qwen3.8-Max, a 2.4-trillion-parameter MoE model, while DeepSeek offers V4-Flash at $0.14 per million input tokens. Both release weights under open licenses, enabling on-premise deployment. Task cost hinges on token generation and int...
A user’s updated local benchmark places the MXFP4-quantized DeepSeek V4 Flash 0731 at the top for efficiency and quality, delivering 1,000 tokens/sec prefill and 90 tokens/sec generation. The result shines a spotlight on low-precision quantization an...
Maple-Preview applies ternary weights to a 20B-parameter model, shrinking memory to ~5 GB while activating only 1B parameters per token via an MoE-like design. This preview hints at local inference on consumer GPUs, but framework support remains imma...
Maple-Preview is a new open-weight model with 20 billion total parameters, only 1 billion active per token, and ternary weight quantization. The combination slashes VRAM requirements for inference, bringing on-premise deployment within reach of consu...
InclusionAI has released Ling-3.0-flash weights on Hugging Face. The model packs 127.5 billion parameters with 512 experts, 8 active per token. The real story is the official FP8 version: 128GB, enough for a single unified-memory system or a multi-GP...
InclusionAI released Ling-3.0-flash, an open-weight MoE with 124 billion total parameters but only 5 billion active per token. Announced before the Kimi K3 and DeepSeek-V4-Flash wave, its sizing could carve a niche in on-premise deployment, where eff...
Three open-weight LLMs match human agreement on IMO-GradingBench at costs up to 100X lower than frontier models. The unanimous voting rule proves most stable, opening paths for local, sovereign, low-TCO evaluations.
