Community tools and runtime techniques are making long-context and MoE LLMs practical on consumer GPUs, using optimized KV and expert caches, quantization, CPU offload, and multi-GPU setups.
The Qwen3.8-Flash-Next case on dual RTX 3090s shows that for MoE models VRAM is no longer the only variable: memory hierarchy and expert caching shift decode from 17 to 29 tokens/s. A concrete signal for teams evaluating on-premise deployments on old...
A Reddit post reports a drop from 15 hours of active programming to 4 hours for debugging or implementing a feature with Qwen 27B, even at 0.5 tokens per second. The data reframes expectations for asynchronous inference and the value of recovered hum...
For local LLMs, parameter count is not the whole constraint: the KV cache grows with every token and can become the real bottleneck at 100k or 200k context. GQA, MQA, and quantization reduce its footprint, but growth remains linear. Future optimizati...
A user documents a jump from 17 to 25-29 tokens/s on two RTX 3090s with a full 261,888-token context, using llama.cpp's LRU expert cache and a smaller ubatch. The setup keeps 48 expert layers in system RAM and shows meaningful headroom for on-prem in...
Perplexity has open-sourced the lily repository inside pplx-garden, a Mac inference server built around a single model, Qwen 3.6, to get the best performance on Apple Silicon. This is not general-purpose serving: vertical integration around one model...
Two compact LLMs, Spark-X2.5-4B and Spark-X2.5-1.7B, claim a native 1-million-token context and an architecture that is not a fine-tune. The Hugging Face page reports the 4B version rivaling a 9B Qwen 3.5 model. Local inference currently requires a c...
The Reddit post is thin, but the message is clear: MTP support for Qwen3.8-Flash-Next in GGUF format promises to raise tokens per second in llama.cpp runs. More than a single optimization, it signals the direction of the open-source stack for local i...
A Reddit experiment runs GLM 5.3 and Flash in Q4 quantization on 4-6 RTX PRO 6000 WS to generate a penthouse in Blender via BlenderMCP. Flash completes the scene with one-third of the tokens and similar timing, while the full model spends 22 minutes ...
Replacing the dense vocabulary projection with approximate HNSW-based search accelerates autoregressive decoding on CPU. Tested on Gemma 3, Llama 3.2 and Qwen 3, batch-one throughput improves by up to 82% for Gemma 3 270M while preserving AlpacaEval ...
LLMFan46 has published GGUF variants of LongCat-Flash-Lite-Sparse, Qwen3.8-27B, and Qwen3.5-122B-A10B, plus Qwen3-Coder-Next and Laguna-S2.1 with optional vision. The need for a personal llama.cpp fork to run sparse attention and a 1M token context h...
Support for RadixArk/Qwen3.8-Flash-Next-NVFP4 in SGLang-V100 shows four V100 32GB running full 256k-token context with over 50GB offloaded to system RAM. Prefill around 4,000 tokens/s and decoding around 60 tokens/s. MTP data signal throughput gains ...
A Reddit setup shows a 27B dense model at 47-50 tokens/s with a 100,000-token context on a 16 GB RTX 4070 Ti SUPER. The signal is not the model but fine-grained runtime control: asymmetric kvarn5/kvarn4 KV cache, a 1,024-token full-precision tail, an...
A three-and-a-half-hour test on a 128GB MacBook Pro M5 Max ran a 79GB 2-bit Qwen3.8-Flash-Next with a 358,400-token context slot. Prefill fell from 1,561 to 318 tokens/s; generation from 30–35 to 11.5 tokens/s at 169K. After 100K tokens the model mix...
A documented Reddit setup runs a Qwen3.8-27B on an RTX 4070 Ti SUPER at 47-50 tokens/s with a 100,000-token context window. The credit goes to the beellama.cpp runtime and the asymmetric kvarn5/kvarn4 KV cache, which freed about 6% VRAM. A precision ...
llama.cpp has merged support for Qwen3.8-Flash-Next and a community report shows 55 tokens/s with a Q4 GGUF on four RTX 3090s. An informal test that shifts attention from software to hardware: for local inference, multi-GPU cost and complexity remain...
A Mac Studio M4 Max with 128GB of unified memory hosted the test of Qwen3.8-Flash-Next, the first model this year to break 94% on tolitius's personal benchmark. The 27B model excels at coding but loses in general knowledge to Gemma 31B and Qwen 3.6 o...
