New quantization formats, speculative decoding, and MoE offloading are lowering latency and memory demands for open models. Techniques range from improved llama.cpp quants to single-GPU MoE systems and faster diffusion LLMs.
Specter combines quantization, speculative decoding, and expert offloading to speed up mixture-of-experts inference on a single GPU. A lightweight quantized draft model runs on-device and anticipates expert usage with high fidelity, reducing PCIe bot...
KRAFTON AI releases Raon-1 mini preview, a 30B MoE LLM with 4B active parameters, focused on Korean conversation and agentic workflows. With Apache 2.0, speculative decoding support, and NVFP4/MXFP8 quantization, inference can run on a single NVIDIA ...
The new EAGLE-4 method posts a 3.59× speedup over vanilla decoding in published benchmarks, ahead of DFlash2 (2.80×), DSpark (2.57×) and MTP (2.22×). Support is currently limited to Qwen3.8-27B, and integration with llama.cpp and vLLM is missing. For...
ISTA-DASLab updated the Qwen3.8-Flash-Next-GSQ-RCO-Coder GGUF model. Retained weights are unchanged from the IQ3_S base, but about 25% of the retained expert selection differs. The model remains English-only by design: with a limited expert budget, E...
A proposal for llama.cpp introduces IQ2_K–IQ6_K types: same file sizes as Q-quants, substantially better KL divergence, slightly lower performance and slower creation. For local LLM deployment, the trade-off shifts: same VRAM footprint, output closer...
A Reddit experiment shows that replacing the heavily quantized PLE in common GGUFs with a full FP8 sidecar improves the perceived quality of Qwen 3.8 Flash Next, with no speed impact. Only disk usage increases.
The ORL (On-Policy Residual Learning) project aims to speed up diffusion LLMs without sacrificing quality. On a single RTX 5090 the team reports 2,788 tokens/s with LLaDA-2.1-mini in NVFP4; the open-source dlmrt inference engine uses CUDA graphs and ...
Reflection has released Beam, an open LLM with a 501-billion total and 23-billion active parameter MoE architecture. It supports a 1 million token context and BF16, FP8, and NVFP4 precision. The focus is token-efficient reasoning: about one third of ...
The model card for Tencent Hy-Memory-30B-A3B sparks debate over a slippery claim: an architecture with 3 billion active parameters out of 30 billion total would beat Opus 5 on an unspecified memory test. The issue is not the benchmark but the structu...
A three-day empirical test of GYRO-M, a Q2 quantization of Qwen Flash Next, reports q3-level stability at reduced size. The project comes from a single contributor and holds up on long contexts, but shows weaknesses on Cyrillic text. The case reopens...
