Local and on-device execution is becoming a practical option for sensitive workloads such as health data, receipts, coding and private search. New compact models and apps let users keep data on their own devices instead of relying on cloud APIs.
Microsoft AI has developed a local version of MAI Code 1.1 Flash, a coding-optimized mixture-of-experts model with 137 billion total and 6.8 billion active parameters. The news points to a push toward self-hosted development tools on Windows, with im...
A jailbroken PS5 performs inference with Qwen 27B at around 20 tokens per second on its GPU. The Reddit report shows how consumer hardware with unified memory can support self-hosted inference for a 27B-parameter model, lowering the barrier for on-pr...
LiquidAI has introduced LFM2.5-24B-A2B, a reasoning model with 24 billion total parameters and 2.45 billion active. Built for private on-device agents, it claims to compete with Qwen3.6-35B-A3B and run up to 1.7 times faster than comparable MoEs, wit...
OpenBMB has released MiniCPM-V 4.7, a 35B-A3B multimodal model and the first MoE in the series to run smoothly on-device. Combining Lookahead Expert Scheduling, layer-wise quantization, and speculative sampling, it claims 13+ tokens/s on iPhone 17 Pr...
A 0.8B model reading receipts on the phone shows on-device inference as a concrete option for personal documents. The project combines fine-tuning on donated data, JSON output and a self-hosted release valve via an OpenAI-compatible server. For those...
A developer built a free Android app that uses a 0.8-billion-parameter vision model to read receipt photos and extract store, date, total, and line items into JSON. The model, fine-tuned on receipts, runs on the phone without sending data to third pa...
OpenMed 3.0 is an Apache-2.0 open-source AI toolkit for the medical sector, designed to operate exclusively on-premise. Its core philosophy is to keep patient data on local machines, without any reliance on cloud services or external APIs, ensuring p...
A community ranking evaluates LLMs that can run on consumer GPUs with 10-16GB of VRAM and 26-32GB of RAM. The criteria include minimum 10 tokens/s, narrative coherence, handling of typos and nuance, and frequency of slop patterns. The results highlig...
DigUp is a free, open-source Mac app that runs EmbeddingGemma 2 locally to search files by content. The 865 MB Q8_0 GGUF model runs on Apple Silicon through llama.cpp and Metal; search loads only the 250 MB text encoder and responds in about a tenth ...
