Benchmarks and engineering reports focus on Qwen3.8-Flash-Next and MoE models, using expert caches, quantizations and KV cache tuning on Apple Silicon and RTX GPUs to improve decode speed and lower TCO. The emphasis is shifting from raw parameters to memory hierarchy, bandwidth per dollar and runtime optimizations.
A Reddit comparison uses VRAM per dollar and rated bandwidth to navigate GPUs discussed in LocalLLaMA communities. Prices were gathered with ChatGPT, new or second-hand, with acknowledged inaccuracies. A rough method, but useful for anyone evaluating...
A custom llama.cpp branch brings expert expansion to MoE models and tests it on Apple Metal. The author says it beats their DS4 version, but cross-platform validation is missing. The case highlights how local inference for sparse models still depends...
A local comparison with Q8_K_XL quantization on two 128GB StrixHalo nodes shows that DeepSeek-V4-Flash-Vision generates tokens more slowly than Qwen3.8-Flash-Next, yet completes tasks in about half the time. The author observes fewer hallucinations a...
The Qwen3.8-Flash-Next-oQ4e-mtp variant shows local inference speeds on Apple Silicon: 45 tokens per second on M4 Max and 25 on M2 Ultra, according to llm-bench.io. The source also notes similar speed to Qwen3.8 27B on Apple Silicon. Useful for self-...
A technical post shows how Qwen3.8 27b in Q4_K_XL quantization runs on an RTX 3090 with 24 GB of VRAM and 100k token context. The author argues that small self-hosted LLMs capable of 80-90% of repetitive work erode cloud vendor revenue more than fron...
A benchmark of 21 quantized Qwen3.8 27B variants on an RTX 5080 with 16GB VRAM shows how much GGUF choice matters. The bartowski IQ4_XS variant offers the lowest mean KL divergence and highest same-top-p agreement, while more aggressive quants prove ...
An experimental modification to llama.cpp updates the Ngram PLE table of Qwen models in memory, injecting new knowledge without reloading the model. Tested only with q8 quantization, it requires memory mapping and suffers from unreliable output contr...
The Qwen3.8-Flash-Next case on dual RTX 3090s shows that for MoE models VRAM is no longer the only variable: memory hierarchy and expert caching shift decode from 17 to 29 tokens/s. A concrete signal for teams evaluating on-premise deployments on old...
For local LLMs, parameter count is not the whole constraint: the KV cache grows with every token and can become the real bottleneck at 100k or 200k context. GQA, MQA, and quantization reduce its footprint, but growth remains linear. Future optimizati...
A user documents a jump from 17 to 25-29 tokens/s on two RTX 3090s with a full 261,888-token context, using llama.cpp's LRU expert cache and a smaller ubatch. The setup keeps 48 expert layers in system RAM and shows meaningful headroom for on-prem in...
