Enterprises and developers are increasingly running large language models on local hardware to reduce costs, improve data sovereignty, and avoid cloud lock-in. Tools like Ollama, llama.cpp, and quantization techniques are making local inference competitive with cloud services.
Two mining-era NVIDIA P102-100 cards, costing around $100 combined, run Qwen 35B via llama.cpp, serving three concurrent users at 23 tokens per second. A showcase of how on-premise inference becomes affordable thanks to llama.cpp’s CUDA 12.x compatib...
A custom C++ and CUDA experiment pushes a 30-billion-parameter MoE model past 50 tok/s on a consumer GPU with 16 GB of VRAM. The garlic-inference engine beats llama.cpp by 50%, revealing untapped optimization headroom for self-hosted inference and st...
Hugging Face CEO Clem Delangue describes a market where enterprises are moving away from consumption-based API services to self-hosting models. With roughly half the Fortune 500 already on the platform, self-hosting becomes the strategic choice for d...
A user demonstrates how a Strix Halo APU runs a 35 billion parameter LLM locally at under 150W, with negligible energy costs. The comparison with discrete GPUs highlights new evaluation criteria for on-premise deployment.
The $65M round backed by Benchmark marks a coming of age for the open source tool that lets developers run AI models on their own PCs. The milestone reflects a structural shift: local inference is no longer a hobby but a real bet on sovereignty, cont...
After buying a 32GB VRAM GPU and running a 5-bit quantized 31B model, a Reddit user finds it blows away the standard free ChatGPT model. The episode exposes a potential downgrading of OpenAI's free tier and strengthens the case for self-hosting when ...
AMD has updated ZenDNN, its open-source library for accelerating inference on Zen CPUs. Version 6.0 adds optimizations and extends quantized model support, strengthening the role of EPYC and Ryzen CPUs for those handling AI workloads locally, with da...
A developer tested whether local language models can accurately answer technical questions. Without RAG, performance drops sharply; with a knowledge base they become reliable. Thinking mode barely helps. Apple Intelligence, constrained to a 4K contex...
A user reports that the Qwen 3.6 35B A3B model, used for coding and technical planning, works flawlessly as long as a disciplined workflow is in place. It's a sign that on-premises LLMs are now mature enough, and the real challenge has shifted from m...
NVIDIA releases Audex-30B-A3B, a mixture-of-experts LLM with 30 billion parameters that extends Nemotron-Cascade-2 to audio without regressing on text capabilities. It handles recognition, translation, text-to-speech, and audio generation, with up to...
The LocalLLaMA community spawned HuggingBay, a tool that makes the distribution of Large Language Models outside official platforms a reality. The project signals a shift toward decentralized infrastructure, highlighting the constraints of centralize...
A user experiment demonstrates that a 27B LLM quantized to Q8 can serve contexts up to 115,000 tokens on a consumer GPU with 32 GB of VRAM, by selectively quantizing the key-value cache. The trade-offs between memory, speed, and perceived quality ope...
