Self-hosted LLM inference on consumer and prosumer hardware is now practical, with models like Qwen 3.6 running at usable speeds thanks to optimized quantization and community tools like Ollama, driving adoption away from cloud APIs.
Xiaomi has uploaded the weights for MiMo-V2.5-DFlash, a 300B+ parameter LLM, along with a separate MTP model, to Hugging Face. Running on two 24 GB GPUs with DDR5 offload delivers 8-10 tokens/s; DFlash could double that. The release underscores how m...
Two mining-era NVIDIA P102-100 cards, costing around $100 combined, run Qwen 35B via llama.cpp, serving three concurrent users at 23 tokens per second. A showcase of how on-premise inference becomes affordable thanks to llama.cpp’s CUDA 12.x compatib...
A custom C++ and CUDA experiment pushes a 30-billion-parameter MoE model past 50 tok/s on a consumer GPU with 16 GB of VRAM. The garlic-inference engine beats llama.cpp by 50%, revealing untapped optimization headroom for self-hosted inference and st...
A user found that the Qwen3.6 35B-A3B model, quantized to Q8_0 and running on CPU, generated complex HTML code with unexpected quality compared to the 4-bit GPU version. A test that raises questions about trade-offs between precision, hardware, and c...
A user demonstrates how a Strix Halo APU runs a 35 billion parameter LLM locally at under 150W, with negligible energy costs. The comparison with discrete GPUs highlights new evaluation criteria for on-premise deployment.
The $65M round backed by Benchmark marks a coming of age for the open source tool that lets developers run AI models on their own PCs. The milestone reflects a structural shift: local inference is no longer a hobby but a real bet on sovereignty, cont...
After buying a 32GB VRAM GPU and running a 5-bit quantized 31B model, a Reddit user finds it blows away the standard free ChatGPT model. The episode exposes a potential downgrading of OpenAI's free tier and strengthens the case for self-hosting when ...
A user reports that the Qwen 3.6 35B A3B model, used for coding and technical planning, works flawlessly as long as a disciplined workflow is in place. It's a sign that on-premises LLMs are now mature enough, and the real challenge has shifted from m...
AMD has started shipping the Ryzen AI Halo, a mini PC built on the Strix Halo platform with a fully open-source software stack. A concrete move for those seeking on-premise LLM deployments without proprietary lock-in.
A Reddit thread asks users to share their favorite locally run Vision-Language Models. Detailed hardware, inference engine, and usage reports emerge, showing how the community works around unreliable benchmarks with hands-on accounts. A valuable snap...
