The local LLM ecosystem is standardizing on GGUF quantization, llama.cpp compatibility and consumer GPU/CPU setups, pushing 27B models onto 16GB cards and improving speed and quality with IQ/FP8 formats.
ISTA-DASLab has published GGUF quants of DeepSeek-V4.1-Flash-1017 using the GSQ-RCO quantization approach. However, llama.cpp does not yet support the architecture: the relevant pull requests are still open. The model card mentions Strata, but the or...
Specter combines quantization, speculative decoding, and expert offloading to speed up mixture-of-experts inference on a single GPU. A lightweight quantized draft model runs on-device and anticipates expert usage with high fidelity, reducing PCIe bot...
KRAFTON AI releases Raon-1 mini preview, a 30B MoE LLM with 4B active parameters, focused on Korean conversation and agentic workflows. With Apache 2.0, speculative decoding support, and NVFP4/MXFP8 quantization, inference can run on a single NVIDIA ...
The new EAGLE-4 method posts a 3.59× speedup over vanilla decoding in published benchmarks, ahead of DFlash2 (2.80×), DSpark (2.57×) and MTP (2.22×). Support is currently limited to Qwen3.8-27B, and integration with llama.cpp and vLLM is missing. For...
A user documents a jump from 9 to 48 tokens/s after switching to STRATA on CachyOS with ready NVIDIA drivers, running Qwen 3.8 Flash Next at 3 bits. The 8 GB GPU handles a 32k context and reaches 42k, staying 130 MB from crashing. The case highlights...
Migtissera has released a GGUF fine-tune of a Qwen model. Internal benchmarks suggest it produces up to a quarter fewer tokens for the same score. Fewer tokens mean faster responses, longer sessions and lower inference costs—an advantage for teams ru...
With North Micro Vision on llama.cpp and Transcribe in Unsloth, Cohere normalizes local inference and fine-tuning on private data. The move shifts TCO analysis: marginal cost of dedicated hardware can diverge from per-token API cost. VRAM and quantiz...
Cohere announced integration of the vision encoder of North Micro Vision Instruct into llama.cpp, making all its recent architectures compatible with the local runtime. Cohere Transcribe also arrives in Unsloth for fine-tuning. A concrete signal for ...
ISTA-DASLab updated the Qwen3.8-Flash-Next-GSQ-RCO-Coder GGUF model. Retained weights are unchanged from the IQ3_S base, but about 25% of the retained expert selection differs. The model remains English-only by design: with a limited expert budget, E...
A proposal for llama.cpp introduces IQ2_K–IQ6_K types: same file sizes as Q-quants, substantially better KL divergence, slightly lower performance and slower creation. For local LLM deployment, the trade-off shifts: same VRAM footprint, output closer...
A homelab, not a vendor, documented a 27-billion-parameter LLM with 262,000 tokens of context on a single Radeon RX 9060 XT with 16 GB of VRAM, using quantization, KV cache streaming, and software fixes. The signal: the entry cost for high-context se...
A homelab with a single 16GB Radeon RX 9060 XT and 24GB of DDR5 runs a 27-billion-parameter LLM at 262,000 tokens of context. The setup uses IQ4_XS quantization and KV cache streaming, reporting 42.7 tokens/s generation at 20k context.
A Reddit experiment shows that replacing the heavily quantized PLE in common GGUFs with a full FP8 sidecar improves the perceived quality of Qwen 3.8 Flash Next, with no speed impact. Only disk usage increases.
