Local AI is moving from experiments to practical deployments on consumer GPUs, phones, consoles and edge devices, with benchmarks, apps and optimizations for on-premise use.
A 24-billion-parameter LLM tested on an AMD W7900 with a 128,000-token context and sustained speed around 95 tokens per second signals that on-premise inference is maturing beyond NVIDIA's ecosystem. AI-Radar explores trade-offs, data sovereignty, an...
A user reported a positive experience with LiquidAI's LFM2.5-24-A2B model (24B parameters), highlighting high performance (~95 tps) on an AMD W7900 GPU. The model, requiring 30GB of VRAM for Q8 quantization with a 128k token context, proves reliable ...
Mesa extends support to NPUs and demonstrates YOLOX via PyTorch, moving accelerator access outside proprietary toolchains. For edge and on-premise, the open driver reduces lock-in and porting costs, changing the TCO calculation. AI-Radar reads the st...
Mesa's open-source driver stack extends NPU support: Tomeu Vizoso's work brings YOLOX via PyTorch to neural accelerators without proprietary SDKs. The structural signal is a shared driver that reduces lock-in, simplifies local deployment, and shifts ...
A jailbroken PS5 performs inference with Qwen 27B at around 20 tokens per second on its GPU. The Reddit report shows how consumer hardware with unified memory can support self-hosted inference for a 27B-parameter model, lowering the barrier for on-pr...
LiquidAI has introduced LFM2.5-24B-A2B, a reasoning model with 24 billion total parameters and 2.45 billion active. Built for private on-device agents, it claims to compete with Qwen3.6-35B-A3B and run up to 1.7 times faster than comparable MoEs, wit...
A half-serious Reddit post imagines that in 2030 owning a home GPU cluster will be seen like owning a gun for home defense. The sign 'we don't call tech support' with a stylized V100 becomes a symbol of a posture: local hardware control and technical...
OpenBMB has released MiniCPM-V 4.7, a 35B-A3B multimodal model and the first MoE in the series to run smoothly on-device. Combining Lookahead Expert Scheduling, layer-wise quantization, and speculative sampling, it claims 13+ tokens/s on iPhone 17 Pr...
A 0.8B model reading receipts on the phone shows on-device inference as a concrete option for personal documents. The project combines fine-tuning on donated data, JSON output and a self-hosted release valve via an OpenAI-compatible server. For those...
A developer built a free Android app that uses a 0.8-billion-parameter vision model to read receipt photos and extract store, date, total, and line items into JSON. The model, fine-tuned on receipts, runs on the phone without sending data to third pa...
A community ranking evaluates LLMs that can run on consumer GPUs with 10-16GB of VRAM and 26-32GB of RAM. The criteria include minimum 10 tokens/s, narrative coherence, handling of typos and nuance, and frequency of slop patterns. The results highlig...
DigUp is a free, open-source Mac app that runs EmbeddingGemma 2 locally to search files by content. The 865 MB Q8_0 GGUF model runs on Apple Silicon through llama.cpp and Metal; search loads only the 250 MB text encoder and responds in about a tenth ...
Qwen-Image-2.1-Turbo's presence on Hugging Face adds another option for local image generation. The Turbo variant targets lower latency, a critical factor for self-hosted deployments with limited VRAM. The repository becomes a testbed for evaluating ...
LemonSeed Studio runs local Qwen3.8-27B inference on iPad Pro with an external AMD GPU in a Thunderbolt enclosure. The LSE engine, identical across iPadOS, macOS, and Linux, hits 158.5 tokens/s on R9700 with DFlash2 speculative decoding and 64.4 on S...
Four open decision models tested on a rented RTX 4090: Laya, d1, Clef-Flash and Lev read 9,534 words and flagged centipede names. Laya is fastest at 3.9 ms per word on llama.cpp; Lev is most accurate on the target but about 13x slower. The test shows...
The JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF collection highlights a compact MoE design with roughly 2.5 billion active parameters and GGUF distribution. For on-prem teams, this mix lowers compute per token, supports quantization on consumer hardw...
