Quantized Qwen3.8 and DeepSeek models are running on single GPUs and Mac/EpYC setups at usable speeds, making local inference more practical.
A Reddit setup shows a 27B dense model at 47-50 tokens/s with a 100,000-token context on a 16 GB RTX 4070 Ti SUPER. The signal is not the model but fine-grained runtime control: asymmetric kvarn5/kvarn4 KV cache, a 1,024-token full-precision tail, an...
A three-and-a-half-hour test on a 128GB MacBook Pro M5 Max ran a 79GB 2-bit Qwen3.8-Flash-Next with a 358,400-token context slot. Prefill fell from 1,561 to 318 tokens/s; generation from 30–35 to 11.5 tokens/s at 169K. After 100K tokens the model mix...
A documented Reddit setup runs a Qwen3.8-27B on an RTX 4070 Ti SUPER at 47-50 tokens/s with a 100,000-token context window. The credit goes to the beellama.cpp runtime and the asymmetric kvarn5/kvarn4 KV cache, which freed about 6% VRAM. A precision ...
The HP Z4 G6i workstation combines Xeon 600 series, RTX graphics and declared Linux support, but the top configuration exceeds $41,000. The signal for AI-Radar is not raw power: it is the legitimization of workstations as on-premise nodes for LLMs, l...
Tested for a month under intensive workloads, the HP Z4 G6i combines an Intel Xeon 600 "Granite Rapids WS" processor and NVIDIA RTX graphics. The top configuration costs more than $41,000. Ubuntu LTS support and a Linux-ready label make it a candidat...
llama.cpp has merged support for Qwen3.8-Flash-Next and a community report shows 55 tokens/s with a Q4 GGUF on four RTX 3090s. An informal test that shifts attention from software to hardware: for local inference, multi-GPU cost and complexity remain...
A Mac Studio M4 Max with 128GB of unified memory hosted the test of Qwen3.8-Flash-Next, the first model this year to break 94% on tolitius's personal benchmark. The 27B model excels at coding but loses in general knowledge to Gemma 31B and Qwen 3.6 o...
The Qwen team has released Qwen3.8-Flash-Next, a 125B open-weight model with 6B active parameters and a native 262,144-token context. The architecture combines Qwen Sparse Attention, Gated Residual, and a 51B n-gram embedding to reduce long-context l...
A Qwen3.8-27B checkpoint in NVFP4 shows that W4A4 quantization guided by distillation can bring a 27-billion-parameter LLM below 20 GB without significant benchmark loss. AI-Radar reads the result as a shift in VRAM and TCO thresholds, but also as a ...
A fully quantized NVFP4 Qwen3.8-27B checkpoint drops to 19.7 GB from 55.6 GB in BF16 while keeping near-identical scores on GPQA-Diamond and AIME26. The team used quantization-aware distillation with the QUASAR algorithm and supports vLLM on Blackwel...
In less than a month, Qwen3.8-27B collected more likes than the next four models combined, surpassing QwQ-32B. The HuggingFace community favours general-purpose LLMs around 27 and 9 billion parameters, sizes that fit self-hosted setups and consumer G...
An Epyc 7663, 256 GB ECC RAM and one RTX 5090 run a 151 GB MoE LLM at 24 tokens/s with Q8_K_XL quantization. The case challenges the assumption that aggregate VRAM must match model weights: balancing CPU, memory bandwidth and PCIe becomes the real co...
A user tested DeepSeek V4 Flash locally with an Epyc 7663, 256 GB of ECC RAM and an RTX 5090 32 GB. Using Q8_K_XL quantization and 100-128k token contexts, the system reached 23.8-24.6 tokens/s. The result shows an LLM with roughly 151 GB of weights ...
A Reddit user describes how community suggestions turned a frustrating setup into a working local LLM with Home Assistant and visual input. The self-hosted barrier is not hardware but configuration, and the community reduces the cognitive cost. We an...
A team created GGUF quantizations of Qwen 3.8 27B and compared them on an RTX PRO 6000 using a voxel island creation task. Q4_K_M occupies 17.1 GB and decodes at 67 tokens/s, with 95.6% top-1 agreement versus BF16. Qualitative differences are limited...
A one-shot test asks qwen3.8:27b in FP8 on vLLM and an RTX 6000 Pro 96GB to convert 2.1 MB of C into HTML/three.js. With a 262,144-token context window, the file does not fit so the agent must navigate it. Cloud Opus 5 finishes in 21 minutes with an ...
A shopper found an RTX 5080 for $702 at Walmart, saving nearly $800 compared to current retail prices. More than a lucky break, the episode reveals the gap between official pricing and real-world cost of consumer GPUs, with direct effects on TCO for ...
A user with an M1 Max is fed up: Qwen 3.8 27B in xhigh mode takes all night for a single task. They ask for a 35B A3B variant, less brilliant but faster. The request exposes the conflict between long-thinking and consumer hardware in self-hosted scen...
