Innovations in quantization, hardware-specific optimizations (Intel Battlemage, AMD GPUs, CPUs) and on-device engines enable LLMs to run smoothly on smartphones, Raspberry Pi and desktops, expanding self-hosted AI.
The Israeli startup closes a Series A+ round to expand its edge AI platform from grocery retail to fast food, logistics, and industry. The goal: turn existing hardware into an intelligent network that learns locally, without sending raw data to the c...
The official INT4 of Ling-3.0-flash runs on a single DGX Spark, but the default configuration delivers only 20.8 tok/s. Two changes — dropping `--enforce-eager` to enable CUDA graphs and switching on MTP speculative decode with one token — push it to...
A free, open-source alternative to paid transcription services: Meetily records and transcribes meetings locally, without sending sensitive data to the cloud. A choice that impacts privacy, costs, and enterprise data control.
A C99 inference engine with zero dependencies drives the 2-billion-parameter ternary BitNet model to 36.25 tokens per second on an Intel Xeon CPU, using custom SIMD and VNNI instructions. But the real ceiling isn't compute: at batch size 1, memory ba...
A power user pushes a 32GB AMD Radeon AI Pro R9700 to its limits with INT4-quantized Qwen3.6 models. The 35B MoE achieves 262,144 token context and 52 tok/s at 100k depth, while the 27B leverages speculative decoding to sustain 59 tok/s past 50k. The...
A handful of code lines in llama.cpp multiplies Intel Battlemage GPU performance up to 2.7 times on 118K-token contexts with quantized KV cache. The gain is so sharp that it turns a gaming card into a tangible alternative for local inference of 35B m...
A local test on AMD hardware compared Qwen 35B-A3B (MoE, 3B active) against a dense 27B model. The MoE managed ~116 tok/s vs ~30 tok/s for the dense variant, yet the coding-quality gap remained narrow except on subtle edge cases. A real-world case th...
A llama.cpp PR switches the SYCL dispatch for quantized KV caches from VEC to TILE kernel, yielding up to 169% faster decode on Intel Battlemage GPUs at long contexts. Gains are substantial even at 32K, pointing to cost-effective on-premise deploymen...
An independent report analyzes LiquidAI's LFM2.5-2.6B model on Raspberry Pi, testing dozens of GGUF quantization and KV cache combinations. The results show negligible degradation with 8 GB of RAM, but certain quants hide sudden quality drops. The an...
A developer ran LFM2.5-2.6B, a 2.69-billion-parameter LLM with a 128K context window, on a OnePlus 13 smartphone using CPU alone and Q4_K_M quantization. The custom inference engine, just 450 KB, hits 17 tok/s and supports other model architectures. ...
