A wave of community experiments shows large 27B and MoE models such as Qwen 3.8, DeepSeek V4, GLM-Air, and Kimi K3 running on consumer and prosumer GPUs through quantized GGUF, speculative decoding, and runtimes like llama.cpp, vLLM, and KTransformers.
An Epyc 7663, 256 GB ECC RAM and one RTX 5090 run a 151 GB MoE LLM at 24 tokens/s with Q8_K_XL quantization. The case challenges the assumption that aggregate VRAM must match model weights: balancing CPU, memory bandwidth and PCIe becomes the real co...
A user tested DeepSeek V4 Flash locally with an Epyc 7663, 256 GB of ECC RAM and an RTX 5090 32 GB. Using Q8_K_XL quantization and 100-128k token contexts, the system reached 23.8-24.6 tokens/s. The result shows an LLM with roughly 151 GB of weights ...
A Reddit user describes how community suggestions turned a frustrating setup into a working local LLM with Home Assistant and visual input. The self-hosted barrier is not hardware but configuration, and the community reduces the cognitive cost. We an...
llama.cpp enables MTP for GLM-4.5-Air, a 106-billion-parameter MoE with only 12 billion active parameters. The change speeds up inference on memory-rich but compute-limited machines like Strix Halo, DGX Spark, and RTX 3090, and strengthens the fine-t...
A team created GGUF quantizations of Qwen 3.8 27B and compared them on an RTX PRO 6000 using a voxel island creation task. Q4_K_M occupies 17.1 GB and decodes at 67 tokens/s, with 95.6% top-1 agreement versus BF16. Qualitative differences are limited...
A one-shot test asks qwen3.8:27b in FP8 on vLLM and an RTX 6000 Pro 96GB to convert 2.1 MB of C into HTML/three.js. With a 262,144-token context window, the file does not fit so the agent must navigate it. Cloud Opus 5 finishes in 21 minutes with an ...
A user with an M1 Max is fed up: Qwen 3.8 27B in xhigh mode takes all night for a single task. They ask for a 35B A3B variant, less brilliant but faster. The request exposes the conflict between long-thinking and consumer hardware in self-hosted scen...
A Modal hosting test with 8 B300s shows 92 tok/s in decode and $190 per million output tokens. The 1-bit variant on 8 A100s costs less per hour but triples the cost per token. The analysis reveals why hourly pricing is a misleading metric for LLM inf...
The new llama.cpp release, with source and pre-built binaries, marks a step for self-hosted inference. Less friction for developers and companies running LLMs locally, more pressure on cloud-only services.
An unverified report describes a nearly twenty-hour agentic coding session with Qwen3.8-27B Q6 on an RTX 3090 and an RTX 3060, sustaining 60–63 token/s. If replicated, the data shifts the cloud versus self-hosted boundary. We analyze trade-offs, TCO ...
A user reports nearly twenty hours of agentic coding with Qwen3.8-27B Q6 on an RTX 3090 and an RTX 3060, sustaining 60–63 tokens/s. The case shows how a mid-size quantized LLM can handle prolonged on-prem workloads on consumer hardware, shifting the ...
A test on a 16GB RTX 4060 Ti shows Qwen 3.8 27B at 30-35 tokens/s with Q3_xxs quantization, strong coding and logic, but conversational slips. The Q4 floor is not automatic for local deployments.
On a 32GB RTX 5090, a new GGUF NVFP4 quant for Qwen3.8 27B reaches 6,250 tokens/s in prefill with 2048-token prompts, 50% faster than a Q4_0 of the same memory footprint and 4-7% faster than other NVFP4 quants. It includes a quantized MTP draft head ...
The expiration of a Claude Code Pro subscription pushed a user to a local Qwen3.8-27b LLM on a 5090M GPU with 24GB of VRAM, alongside Pi. A test app for aurora forecasting showed similar timing, a better UI from Pi but better science from Claude Sonn...
A builder validated a self-hosted configuration with 16 RTX 5060 Ti 16GB cards, two Broadcom/PLX PEX88096 switches, and a Xeon Gold 6330. The system serves DeepSeek V4 Flash-0731 with up to 1 million tokens of context and, in one setup, an average ge...
A bare-metal test with two RTX 3090s, vLLM, and DFlash2 speculative decoding pushes Qwen3.8-27B to 218 tok/s on a single request, with prefill up to 1342 tok/s and a 131k context ceiling. The setup uses INT4 quantization, custom vLLM changes, and pea...
An agentic workload running for over eight hours on a single RTX PRO 6000 with DeepSeek Harness and NInfer handled 966 model calls, 131.2 million input tokens and 853.3 thousand output tokens with zero generation failures. The API price comparison es...
llama.cpp drops sequential build numbers and adopts semantic versioning with v0.1.0. For self-hosted and on-premise deployments, the move gives operators clearer signals about breaking changes, dependency pinning, and upgrade planning, even though 0....
KTransformers, a framework for heterogeneous LLMs, releases version 0.7 with expanded AVX-512 support, a targeted change for AMD EPYC servers. For self-hosted teams, the message is structural: the CPU is no longer a fallback, but an active component ...
