A wave of quantized, distilled, and ternary open models (Qwen3.8, Bonsai 2, K2 Horizon) is making capable local inference viable on consumer GPUs and in browsers. The focus is shifting from raw parameter counts to runtime efficiency, VRAM footprint, and GGUF quantization details.
UkisAI's question about Bonsai 2 Swift is not a simple update request. It highlights how the TCO of local LLMs increasingly depends on runtime behavior: wasted tokens, overthinking loops and quantization choices are redefining on-premise efficiency.
Prism-LM released Bonsai 2 models based on Qwen3.8, evaluated with the same methodology used for the rest of the comparison. The result, roughly 91.5% on a composite benchmark, is a relative reference point, not an absolute figure. Using Prism's fork...
A one-shot RTX 5090 test compares Bonsai 2 27B, Gemma 4 12B and Qwen 3.5 9B on a voxel Japanese pagoda scene in three.js. Bonsai, in 7.2 GB PQ2_0 quantization, generates 106,396 tokens in 17.5 minutes at 101 tokens/s and looks qualitatively best. 90%...
After Swift Qwen3.8 27B downloads jumped from 100,000 to 150,000 overnight, UkisAI is asking whether to apply the same token-efficiency treatment to Bonsai 2, which their tests show suffers from overthinking loops and high token usage. The team wants...
Bonsai's own documents show Ternary Bonsai 2 27B scoring 52.8 on Terminal-Bench 2.1 and 60.8 on SWE-bench Verified, roughly three quarters of the full-precision model used as comparison. Yet the competitor name appears wrong, and the 98.2% retained-c...
From the AI lab comes a signal: a small LLM could shift attention from brute-force scale to efficiency. Without public technical data, the news matters as a directional indicator: compact models suit local deployment, data control, and cost-conscious...
Ternary Bonsai 2 is available on Hugging Face: it derives from Qwen3.8-27B, keeps the hybrid-attention architecture, and uses ternary weights to shrink below 6GB. According to the model card, it is 9x smaller than FP16 while retaining 98.2% of the in...
UkisAI's open-source model passes 100k downloads and ranks first among fine-tunes on Hugging Face. It penalizes overthinking patterns in small LLMs, cutting token usage by 58.3% and boosting speed 1.95x without accuracy loss. Swift1.5 and Swift Qwen3...
New optimizations for single AMD Radeon R9700 cards double performance with Qwen3.8 27B in NVFP4 format: up to 153.1 tokens/s in decode, 3,619 tokens/s in prefill, and 471 aggregate tokens/s with eight concurrent requests. The updated vLLM MXFP4 repo...
A Reddit post condenses the entire open-source LLM lexicon into one file name: Qwen base, 27 billion parameters, 'Uncensored' labels, and GGUF format. The analysis shows how community naming rewards remixing but undermines the reproducibility that on...
Voodoo Quant, a method aimed at aggressive quantization on small GGUF models, is now MIT licensed. Instead of static analysis, it uses gradient descent to choose per-tensor quantization levels. People working with local LLMs and limited VRAM can repl...
A 7B model in GGUF lands between 27B and 35B parameter models. For self-hosted deployment, K2 Horizon 7B shifts TCO calculations and makes VRAM efficiency the central metric, not parameter count. Yet deeper verification is needed.
The periodic r/LocalLLaMA thread invites submissions of open-weight, open-source, locally hostable software, with clear descriptions, comparisons, and validation evidence. Commercial projects are excluded. For AI-RADAR, this is a structural signal: t...
IFM's K2 Horizon 7B lands on the Artificial Analysis Intelligence Index between Qwen 3.6 27B and Qwen 3.6 35BA3b. In early local testing, it compiles the latest llama.cpp for CUDA without trouble. If the result holds, a 7B performing like much larger...
A pull request in llama.cpp adds support for Maple 20B-A1B, a ternary MoE LLM aimed at CPU inference. The open question is whether it can lower VRAM requirements for self-hosted deployments.
A 150-million-parameter model with performance similar to GPT-2 Small has been trained on a single RTX Pro 6000 using 7 billion tokens. Published benchmarks: PIQA 62.24%, Hellaswag 32.20%, Arc-Easy 44.91%, Arc-Challenge 25.00%, Arithmark 3.0 33.90%, ...
A self-hosted test with Qwen3.8-Flash-Next-IQ4_XS on llama.cpp produces a complex SVG from a creative prompt. The output impresses, but previews diverge: palms and music notes disappear in the browser. The case shows why vector code generation must b...
The model card for bartowski/Qwen3.8-27B-GGUF introduces per-tensor layout maps for GGUF compression. This makes the quantization scheme explicit tensor by tensor, helping self-hosted teams estimate VRAM usage and fidelity before downloading.
