The local deployment of LLMs is expanding across consumer GPUs, CPU-only inference, on-device voice front-ends and browser-based agents. The shift moves total cost of ownership from cloud per-token pricing toward self-hosted hardware and control.
The Swedish startup Strawberry exits beta with an enterprise browser that embeds AI assistants. Agents run on the user's device, not on cloud servers, which can address data sovereignty demands. The company has raised $6 million and plans to scale ac...
An independent project has shared an early 4-billion-parameter model for coding and 3D scene generation, fine-tuned from Qwen3.5-4B. GGUF checkpoints Q4_K_M and Q8_0 are available on Hugging Face while training continues. The author plans to update t...
The subscription economics of large LLMs is under pressure, while specialized models with a few billion parameters solve tasks like OCR and segmentation. The thesis: the future is local, with smaller models, easier training, and no data center power ...
Kitten-TTS 2 brings 1.58-bit ternary quantization to a 1.7 billion parameter model that runs in real time on CPU. Self-hosting becomes realistic without GPUs, shifting TCO from VRAM to software execution quality. For organizations under data sovereig...
KittenML has released code and weights for Kitten-TTS 2, a TTS model with 1.58-bit ternary quantization and 1.7 billion parameters. It runs in real time on CPU, performs zero-shot voice cloning from under five seconds of audio, supports more than 20 ...
An Apache-2.0 open source project offers an on-device voice front-end for local assistants: five models for text-based wake word, voiceprint, voice separation and personal VAD. The Rust engine runs in the browser, Python and Android; models take 7.5 ...
A user documents a jump from 9 to 48 tokens/s after switching to STRATA on CachyOS with ready NVIDIA drivers, running Qwen 3.8 Flash Next at 3 bits. The 8 GB GPU handles a 32k context and reaches 42k, staying 130 MB from crashing. The case highlights...
Migtissera has released a GGUF fine-tune of a Qwen model. Internal benchmarks suggest it produces up to a quarter fewer tokens for the same score. Fewer tokens mean faster responses, longer sessions and lower inference costs—an advantage for teams ru...
With North Micro Vision on llama.cpp and Transcribe in Unsloth, Cohere normalizes local inference and fine-tuning on private data. The move shifts TCO analysis: marginal cost of dedicated hardware can diverge from per-token API cost. VRAM and quantiz...
Cohere announced integration of the vision encoder of North Micro Vision Instruct into llama.cpp, making all its recent architectures compatible with the local runtime. Cohere Transcribe also arrives in Unsloth for fine-tuning. A concrete signal for ...
