Enterprises and individuals increasingly demand on-premise and on-device AI inference to retain data control, privacy, and independence from cloud lock-in.
A custom llama.cpp build with ROCmFPX kernels runs the 122-billion-parameter Qwen3.5 model on AMD GPUs at 28.50 tokens per second, cutting memory usage by 18% and boosting inference speed by 37%. A proof of concept that large MoE models can be self-h...
A new Obsidian plugin lets you chat with your vault using local AI, with no data sent to the cloud. Released under MIT license, it runs the model on your Mac through the QVAC SDK. It provides clickable citations, semantic link creation, and personali...
A new European open-source language model has appeared in online forums: Soofi S 30B-A3B. With 3 billion active parameters out of 30 billion total, it promises low-VRAM local execution, alongside reasoning-oriented preview versions. Early comparisons...
OpenLLM-France releases Luciole-23B-Instruct-1.1, a multilingual causal model under Apache 2.0 license, also available in 8B and 1B sizes. Trained on the Jean Zay supercomputer in three stages, it covers math, code, RAG, and translation. The real sig...
A system with an RTX 4060 Ti and a Ryzen 5 achieved 7 tokens per second on a 98 GB LLM using only CPU and RAM. In a week, recent llama.cpp commits tripled the speed, signaling a shift for cost-conscious on-premise inference with full data control.
Integrating Qwen into Apple Intelligence marks a turning point for data sovereignty: even global giants must adapt to local stacks. A move that reshapes AI deployment constraints on an international scale.
Thinking Machines Lab, founded by Mira Murati, releases Inkling: an open-weight LLM that doesn’t aim for benchmark dominance. A political choice that puts customization, control, and data sovereignty back at the center.
PrismML released Bonsai 27B, a dense LLM using 1-bit quantization to shrink from 54 GB to 3.8 GB and run locally in the browser via custom WebGPU kernels, retaining 90% of intelligence. A major step for on-device AI and data sovereignty.
The debut of Bonsai 27B, the first 27B-class model to run on a smartphone, redraws the line between cloud and device. The implications for data sovereignty and on-premise deployment are significant, even as technical trade-offs remain the core challe...
Six months after a $60M Series B, fintech Flex raises another $70M in a B1 round led by Halo, Ryan Smith's investment fund. The money will take its AI-powered private bank global. But the real issue behind the funding is infrastructure: for a bank ta...
A community thank-you for a symbolic milestone in llama.cpp tells a deeper story: local inference on commodity hardware is now a production reality, reshaping deployment strategies, data sovereignty, and cost calculus for enterprises.
Caltech spin-off PrismML, backed by Khosla, has compressed Alibaba’s open-source Qwen 3.6 from 54GB to under 4GB, running all 27 billion parameters actively on an iPhone 17 Pro. The model will be available for download next Tuesday. CEO Babak Hassibi...
A local interface for sd.cpp makes image generation accessible on Windows and Linux, marking a step toward creative sovereignty and hardware control.
Hiring a product manager for households puts ChatGPT into everyday family life with the elderly and caregivers. This strategic shift reignites the tension between cloud convenience and local control of sensitive data, prompting many families to consi...
A custom C++ and CUDA experiment pushes a 30-billion-parameter MoE model past 50 tok/s on a consumer GPU with 16 GB of VRAM. The garlic-inference engine beats llama.cpp by 50%, revealing untapped optimization headroom for self-hosted inference and st...
