Aggressive quantization and new inference frameworks are bringing very large open models within reach of smaller GPUs. Runtime support in tools like llama.cpp and vLLM still lags behind published weights and acceleration methods.
ISTA-DASLab's release of GGUF quantized weights for DeepSeek-V4.1-Flash-1017 is not just a quantization milestone. It exposes the real bottleneck in local deployment: runtime integration. With llama.cpp still lacking official support and references p...
KLD is often treated as a universal metric for ranking quantized models, but it only measures the average divergence between the probability distributions of the base model and the quantized one. The number is fragile: it depends on the dataset, is e...
ISTA DASLab has released three GGUF builds of DeepSeek-V4.1-Flash-1017 with GSQ and RCO quantization, at 1.75 to 2.25 bits per weight over the 551.9 billion backbone parameters. A shared 110.6 GB Engram file stays memory-mapped on disk. The model run...
ISTA-DASLab has published GGUF quants of DeepSeek-V4.1-Flash-1017 using the GSQ-RCO quantization approach. However, llama.cpp does not yet support the architecture: the relevant pull requests are still open. The model card mentions Strata, but the or...
Specter combines quantization, speculative decoding, and expert offloading to speed up mixture-of-experts inference on a single GPU. A lightweight quantized draft model runs on-device and anticipates expert usage with high fidelity, reducing PCIe bot...
KRAFTON AI releases Raon-1 mini preview, a 30B MoE LLM with 4B active parameters, focused on Korean conversation and agentic workflows. With Apache 2.0, speculative decoding support, and NVFP4/MXFP8 quantization, inference can run on a single NVIDIA ...
The new EAGLE-4 method posts a 3.59× speedup over vanilla decoding in published benchmarks, ahead of DFlash2 (2.80×), DSpark (2.57×) and MTP (2.22×). Support is currently limited to Qwen3.8-27B, and integration with llama.cpp and vLLM is missing. For...
DeepSeek has released DeepInfer, an open-source framework for large-scale inference of its models. The code gives the community an official reference to study serving optimizations and improve custom engines such as gufo, halogen, strixite and Strata...
ISTA-DASLab updated the Qwen3.8-Flash-Next-GSQ-RCO-Coder GGUF model. Retained weights are unchanged from the IQ3_S base, but about 25% of the retained expert selection differs. The model remains English-only by design: with a limited expert budget, E...
