New quantization formats and speculative decoding techniques promise lower memory and faster token generation for local LLMs. However, runtime and tooling integration often lags behind model releases.
ISTA-DASLab's release of GGUF quantized weights for DeepSeek-V4.1-Flash-1017 is not just a quantization milestone. It exposes the real bottleneck in local deployment: runtime integration. With llama.cpp still lacking official support and references p...
KLD is often treated as a universal metric for ranking quantized models, but it only measures the average divergence between the probability distributions of the base model and the quantized one. The number is fragile: it depends on the dataset, is e...
ISTA DASLab has released three GGUF builds of DeepSeek-V4.1-Flash-1017 with GSQ and RCO quantization, at 1.75 to 2.25 bits per weight over the 551.9 billion backbone parameters. A shared 110.6 GB Engram file stays memory-mapped on disk. The model run...
ISTA-DASLab has published GGUF quants of DeepSeek-V4.1-Flash-1017 using the GSQ-RCO quantization approach. However, llama.cpp does not yet support the architecture: the relevant pull requests are still open. The model card mentions Strata, but the or...
Specter combines quantization, speculative decoding, and expert offloading to speed up mixture-of-experts inference on a single GPU. A lightweight quantized draft model runs on-device and anticipates expert usage with high fidelity, reducing PCIe bot...
KRAFTON AI releases Raon-1 mini preview, a 30B MoE LLM with 4B active parameters, focused on Korean conversation and agentic workflows. With Apache 2.0, speculative decoding support, and NVFP4/MXFP8 quantization, inference can run on a single NVIDIA ...
The new EAGLE-4 method posts a 3.59× speedup over vanilla decoding in published benchmarks, ahead of DFlash2 (2.80×), DSpark (2.57×) and MTP (2.22×). Support is currently limited to Qwen3.8-27B, and integration with llama.cpp and vLLM is missing. For...
Migtissera has released a GGUF fine-tune of a Qwen model. Internal benchmarks suggest it produces up to a quarter fewer tokens for the same score. Fewer tokens mean faster responses, longer sessions and lower inference costs—an advantage for teams ru...
ISTA-DASLab updated the Qwen3.8-Flash-Next-GSQ-RCO-Coder GGUF model. Retained weights are unchanged from the IQ3_S base, but about 25% of the retained expert selection differs. The model remains English-only by design: with a limited expert budget, E...
A proposal for llama.cpp introduces IQ2_K–IQ6_K types: same file sizes as Q-quants, substantially better KL divergence, slightly lower performance and slower creation. For local LLM deployment, the trade-off shifts: same VRAM footprint, output closer...
