DeepSeek-V4.1 at 1.75-bit and Qwen4's one-million-token context showcase the promise of aggressive quantization and the persistent VRAM/runtime bottlenecks in self-hosted inference. The attention shifts to runtime support and memory footprint rather than raw model size.
The pull request in qwen-code signals that Qwen4 targets a one-million-token window. For self-hosted inference the cost is not theoretical: the key-value cache grows with tokens and fills VRAM along with weights. Qwen4-27B makes multi-GPU setups or a...
Alibaba brings Qwen4 to a one-million-token context window. The news emerges from a pull request in qwen-code: Qwen4-27B is expected tomorrow at the Apsara event in the Netherlands, while Qwen4-max is dated November 1st with October 1st as a likely p...
ISTA-DASLab's release of GGUF quantized weights for DeepSeek-V4.1-Flash-1017 is not just a quantization milestone. It exposes the real bottleneck in local deployment: runtime integration. With llama.cpp still lacking official support and references p...
KLD is often treated as a universal metric for ranking quantized models, but it only measures the average divergence between the probability distributions of the base model and the quantized one. The number is fragile: it depends on the dataset, is e...
ISTA DASLab has released three GGUF builds of DeepSeek-V4.1-Flash-1017 with GSQ and RCO quantization, at 1.75 to 2.25 bits per weight over the 551.9 billion backbone parameters. A shared 110.6 GB Engram file stays memory-mapped on disk. The model run...
ISTA-DASLab has published GGUF quants of DeepSeek-V4.1-Flash-1017 using the GSQ-RCO quantization approach. However, llama.cpp does not yet support the architecture: the relevant pull requests are still open. The model card mentions Strata, but the or...
