Very small models, aggressive quantization and algebraic compression are pushing inference to CPUs, browsers, phones and microcontrollers like the ESP32.
A 0.8B model reading receipts on the phone shows on-device inference as a concrete option for personal documents. The project combines fine-tuning on donated data, JSON output and a self-hosted release valve via an OpenAI-compatible server. For those...
A developer built a free Android app that uses a 0.8-billion-parameter vision model to read receipt photos and extract store, date, total, and line items into JSON. The model, fine-tuned on receipts, runs on the phone without sending data to third pa...
A new compression format for LLMs uses algebraic integers instead of the usual floating-point or fixed-point formats. The research shows competitive accuracy with the MX format at equal bit-per-parameter budgets. For self-hosted inference, the potent...
A shared learning rate can manufacture false advantages when comparing fine-tuning recipes with different parameter counts. In SVD-based KV-cache compression, healing only the encoder appeared superior, but with per-arm learning rates it reaches pari...
An arXiv preprint describes LittleBit, an ultra-low-bit quantization approach based on latent factorization and quantization-aware training for very small models. The news matters for on-premise LLM operators: it is not just about saving space, but a...
A 19,969-parameter, 81 KB FP32 model generates short stories with goal, problem, and resolution while running on CPU. MacroStories lowers the bound for coherent narrative generation and questions assumptions about local inference.
EmbeddingGemma 2 maps text and images into a 768-dimensional space. Ported to ruNNtime, a TypeScript WebGPU inference library, it enables photo search by description with inference entirely on the browser GPU. No upload, data stays on device: a concr...
TinyDecide is a 10-million-parameter model with a roughly 6 MB footprint, smaller than every entry on the Decision Index leaderboard. It runs in the browser, Node.js, Python, Rust, and on an ESP32. For edge and on-premise deployments, the signal is n...
