Quantization formats, expert-routing tweaks, FP8 sidecars and token-reduction methods are being adopted to cut VRAM and latency. The goal is to preserve quality while shrinking models or accelerating decoding on local hardware.
The new EAGLE-4 method posts a 3.59× speedup over vanilla decoding in published benchmarks, ahead of DFlash2 (2.80×), DSpark (2.57×) and MTP (2.22×). Support is currently limited to Qwen3.8-27B, and integration with llama.cpp and vLLM is missing. For...
ISTA-DASLab updated the Qwen3.8-Flash-Next-GSQ-RCO-Coder GGUF model. Retained weights are unchanged from the IQ3_S base, but about 25% of the retained expert selection differs. The model remains English-only by design: with a limited expert budget, E...
A GGUF repository with no notes on Hugging Face shifts verification costs to deployers. AI-Radar analyzes quantization, context, VRAM, and provenance, showing why missing documentation blocks self-hosted pipelines, TCO calculation, and sovereignty.
A proposal for llama.cpp introduces IQ2_K–IQ6_K types: same file sizes as Q-quants, substantially better KL divergence, slightly lower performance and slower creation. For local LLM deployment, the trade-off shifts: same VRAM footprint, output closer...
A Reddit experiment shows that replacing the heavily quantized PLE in common GGUFs with a full FP8 sidecar improves the perceived quality of Qwen 3.8 Flash Next, with no speed impact. Only disk usage increases.
A 4-billion-parameter BF16 model from NVIDIA Nemotron Labs appears on Hugging Face. The size matters less than the precision: BF16 keeps it away from aggressive quantization and places it in the local deployment space on GPUs with enough VRAM. We loo...
The ORL (On-Policy Residual Learning) project aims to speed up diffusion LLMs without sacrificing quality. On a single RTX 5090 the team reports 2,788 tokens/s with LLaDA-2.1-mini in NVFP4; the open-source dlmrt inference engine uses CUDA graphs and ...
A three-day empirical test of GYRO-M, a Q2 quantization of Qwen Flash Next, reports q3-level stability at reduced size. The project comes from a single contributor and holds up on long contexts, but shows weaknesses on Cyrillic text. The case reopens...
