NVFP4 and GGUF quantization, KV cache reuse, MoE conversion, and CPU-side tricks are bringing models up to 125B within reach of consumer and older GPUs. Community tests show large-context and multi-GPU setups becoming a practical on-premise frontier.
Replacing the dense vocabulary projection with approximate HNSW-based search accelerates autoregressive decoding on CPU. Tested on Gemma 3, Llama 3.2 and Qwen 3, batch-one throughput improves by up to 82% for Gemma 3 270M while preserving AlpacaEval ...
LLMFan46 has published GGUF variants of LongCat-Flash-Lite-Sparse, Qwen3.8-27B, and Qwen3.5-122B-A10B, plus Qwen3-Coder-Next and Laguna-S2.1 with optional vision. The need for a personal llama.cpp fork to run sparse attention and a 1M token context h...
Support for RadixArk/Qwen3.8-Flash-Next-NVFP4 in SGLang-V100 shows four V100 32GB running full 256k-token context with over 50GB offloaded to system RAM. Prefill around 4,000 tokens/s and decoding around 60 tokens/s. MTP data signal throughput gains ...
A Reddit setup shows a 27B dense model at 47-50 tokens/s with a 100,000-token context on a 16 GB RTX 4070 Ti SUPER. The signal is not the model but fine-grained runtime control: asymmetric kvarn5/kvarn4 KV cache, a 1,024-token full-precision tail, an...
A three-and-a-half-hour test on a 128GB MacBook Pro M5 Max ran a 79GB 2-bit Qwen3.8-Flash-Next with a 358,400-token context slot. Prefill fell from 1,561 to 318 tokens/s; generation from 30–35 to 11.5 tokens/s at 169K. After 100K tokens the model mix...
A documented Reddit setup runs a Qwen3.8-27B on an RTX 4070 Ti SUPER at 47-50 tokens/s with a 100,000-token context window. The credit goes to the beellama.cpp runtime and the asymmetric kvarn5/kvarn4 KV cache, which freed about 6% VRAM. A precision ...
llama.cpp has merged support for Qwen3.8-Flash-Next and a community report shows 55 tokens/s with a Q4 GGUF on four RTX 3090s. An informal test that shifts attention from software to hardware: for local inference, multi-GPU cost and complexity remain...
A Qwen3.8-27B checkpoint in NVFP4 shows that W4A4 quantization guided by distillation can bring a 27-billion-parameter LLM below 20 GB without significant benchmark loss. AI-Radar reads the result as a shift in VRAM and TCO thresholds, but also as a ...
A fully quantized NVFP4 Qwen3.8-27B checkpoint drops to 19.7 GB from 55.6 GB in BF16 while keeping near-identical scores on GPQA-Diamond and AIME26. The team used quantization-aware distillation with the QUASAR algorithm and supports vLLM on Blackwel...
ToMoE turns dense checkpoints into MoE without fine-tuning, lowering active parameters per token without reducing memory footprint. A shift for on-premise deployments: less compute, same model. Runtime, latency, and VRAM constraints remain. The signa...
KVBoost reuses key-value cache at chunk level and reduces time-to-first-token by 4.49x on Qwen2.5-3B, with no accuracy loss. Unlike prefix caching, it does not require a shared leading prefix: a dual hash separates positional and content identity. As...
A new method converts dense LLMs into Mixture-of-Experts models through differentiable dynamic structural pruning, reducing active parameters per token without removing them. It needs no fine-tuning and outperforms previous structural pruning on Phi-...
