A wave of compact, quantized and CPU-friendly models such as Bonsai 2, K2 Horizon 7B and Von is bringing LLM inference to consumer GPUs, browsers and even CPU-only systems. The focus is shifting from raw parameters to token efficiency, VRAM footprint and runtime behavior.
Von, an open source LLM with 395 million parameters, runs entirely on CPU with 1-2 GB of memory and claimed latency between 25 and 300 ms. Announced on Reddit as a drop-in replacement for JEV and not yet optimized, it reopens the discussion about GPU...
A post on r/ROCm highlights the problem of FP4-optimized inference engines: fast in benchmarks, but often unreliable on small dense models. Compression can work on Large Language Models with many parameters, but on small models logical errors become ...
Von, an open-source model with 395 million parameters, is introduced as a drop-in replacement for TypeSafe's JEV. It runs entirely on CPU with 1–2 GB of memory and latency of 25–300 ms; the author says it beats JEV in all benchmarks. Code is on GitHu...
UkisAI's question about Bonsai 2 Swift is not a simple update request. It highlights how the TCO of local LLMs increasingly depends on runtime behavior: wasted tokens, overthinking loops and quantization choices are redefining on-premise efficiency.
Prism-LM released Bonsai 2 models based on Qwen3.8, evaluated with the same methodology used for the rest of the comparison. The result, roughly 91.5% on a composite benchmark, is a relative reference point, not an absolute figure. Using Prism's fork...
A one-shot RTX 5090 test compares Bonsai 2 27B, Gemma 4 12B and Qwen 3.5 9B on a voxel Japanese pagoda scene in three.js. Bonsai, in 7.2 GB PQ2_0 quantization, generates 106,396 tokens in 17.5 minutes at 101 tokens/s and looks qualitatively best. 90%...
After Swift Qwen3.8 27B downloads jumped from 100,000 to 150,000 overnight, UkisAI is asking whether to apply the same token-efficiency treatment to Bonsai 2, which their tests show suffers from overthinking loops and high token usage. The team wants...
Bonsai's own documents show Ternary Bonsai 2 27B scoring 52.8 on Terminal-Bench 2.1 and 60.8 on SWE-bench Verified, roughly three quarters of the full-precision model used as comparison. Yet the competitor name appears wrong, and the 98.2% retained-c...
From the AI lab comes a signal: a small LLM could shift attention from brute-force scale to efficiency. Without public technical data, the news matters as a directional indicator: compact models suit local deployment, data control, and cost-conscious...
Ternary Bonsai 2 is available on Hugging Face: it derives from Qwen3.8-27B, keeps the hybrid-attention architecture, and uses ternary weights to shrink below 6GB. According to the model card, it is 9x smaller than FP16 while retaining 98.2% of the in...
UkisAI's open-source model passes 100k downloads and ranks first among fine-tunes on Hugging Face. It penalizes overthinking patterns in small LLMs, cutting token usage by 58.3% and boosting speed 1.95x without accuracy loss. Swift1.5 and Swift Qwen3...
A Hugging Face page and online demo present a 230-million-parameter model fine-tuned for chess. The news is small, but it opens a practical discussion on specialized compact models for local deployments.
A Reddit post condenses the entire open-source LLM lexicon into one file name: Qwen base, 27 billion parameters, 'Uncensored' labels, and GGUF format. The analysis shows how community naming rewards remixing but undermines the reproducibility that on...
Voodoo Quant, a method aimed at aggressive quantization on small GGUF models, is now MIT licensed. Instead of static analysis, it uses gradient descent to choose per-tensor quantization levels. People working with local LLMs and limited VRAM can repl...
A 7B model in GGUF lands between 27B and 35B parameter models. For self-hosted deployment, K2 Horizon 7B shifts TCO calculations and makes VRAM efficiency the central metric, not parameter count. Yet deeper verification is needed.
IFM's K2 Horizon 7B lands on the Artificial Analysis Intelligence Index between Qwen 3.6 27B and Qwen 3.6 35BA3b. In early local testing, it compiles the latest llama.cpp for CUDA without trouble. If the result holds, a 7B performing like much larger...
A pull request in llama.cpp adds support for Maple 20B-A1B, a ternary MoE LLM aimed at CPU inference. The open question is whether it can lower VRAM requirements for self-hosted deployments.
