Quantized, ternary, and CPU-friendly LLMs are bringing 27B-class capabilities to consumer hardware, browsers, and edge deployments. Attention is shifting from parameter count to VRAM, token waste, and runtime efficiency.
Von, an open source LLM with 395 million parameters, runs entirely on CPU with 1-2 GB of memory and claimed latency between 25 and 300 ms. Announced on Reddit as a drop-in replacement for JEV and not yet optimized, it reopens the discussion about GPU...
A post on r/ROCm highlights the problem of FP4-optimized inference engines: fast in benchmarks, but often unreliable on small dense models. Compression can work on Large Language Models with many parameters, but on small models logical errors become ...
halogen 0.12.0 resolves deep-context degradation: on a Ryzen AI Max+ 395 with 128 GB, Qwen3.8-Flash-Next decode at one million tokens rises from 27.3 to 38.3 tok/s, and cold prefill drops from 21.2 to 17.9 minutes. For self-hosted LLM evaluation, pro...
Von, an open-source model with 395 million parameters, is introduced as a drop-in replacement for TypeSafe's JEV. It runs entirely on CPU with 1–2 GB of memory and latency of 25–300 ms; the author says it beats JEV in all benchmarks. Code is on GitHu...
UkisAI's question about Bonsai 2 Swift is not a simple update request. It highlights how the TCO of local LLMs increasingly depends on runtime behavior: wasted tokens, overthinking loops and quantization choices are redefining on-premise efficiency.
Prism-LM released Bonsai 2 models based on Qwen3.8, evaluated with the same methodology used for the rest of the comparison. The result, roughly 91.5% on a composite benchmark, is a relative reference point, not an absolute figure. Using Prism's fork...
A one-shot RTX 5090 test compares Bonsai 2 27B, Gemma 4 12B and Qwen 3.5 9B on a voxel Japanese pagoda scene in three.js. Bonsai, in 7.2 GB PQ2_0 quantization, generates 106,396 tokens in 17.5 minutes at 101 tokens/s and looks qualitatively best. 90%...
After Swift Qwen3.8 27B downloads jumped from 100,000 to 150,000 overnight, UkisAI is asking whether to apply the same token-efficiency treatment to Bonsai 2, which their tests show suffers from overthinking loops and high token usage. The team wants...
Bonsai's own documents show Ternary Bonsai 2 27B scoring 52.8 on Terminal-Bench 2.1 and 60.8 on SWE-bench Verified, roughly three quarters of the full-precision model used as comparison. Yet the competitor name appears wrong, and the 98.2% retained-c...
A layer-wise curriculum learning method for LLM compression starts with easy optimization tasks and moves to harder ones, stabilizing knowledge transfer. It reduces GPU memory usage and training hours by more than 50% on BERT and GPT-2, and outperfor...
From the AI lab comes a signal: a small LLM could shift attention from brute-force scale to efficiency. Without public technical data, the news matters as a directional indicator: compact models suit local deployment, data control, and cost-conscious...
Ternary Bonsai 2 is available on Hugging Face: it derives from Qwen3.8-27B, keeps the hybrid-attention architecture, and uses ternary weights to shrink below 6GB. According to the model card, it is 9x smaller than FP16 while retaining 98.2% of the in...
UkisAI's open-source model passes 100k downloads and ranks first among fine-tunes on Hugging Face. It penalizes overthinking patterns in small LLMs, cutting token usage by 58.3% and boosting speed 1.95x without accuracy loss. Swift1.5 and Swift Qwen3...
New optimizations for single AMD Radeon R9700 cards double performance with Qwen3.8 27B in NVFP4 format: up to 153.1 tokens/s in decode, 3,619 tokens/s in prefill, and 471 aggregate tokens/s with eight concurrent requests. The updated vLLM MXFP4 repo...
A Hugging Face page and online demo present a 230-million-parameter model fine-tuned for chess. The news is small, but it opens a practical discussion on specialized compact models for local deployments.
A 34-million-parameter module corrects 53.3% of base-model errors in Gemma 4 E2B without degrading tested benchmarks. A LoRA baseline reaches 83.3% correction but loses up to 75% capability. The KL anchoring term and logit-level correction are centra...
Voodoo Quant, a method aimed at aggressive quantization on small GGUF models, is now MIT licensed. Instead of static analysis, it uses gradient descent to choose per-tensor quantization levels. People working with local LLMs and limited VRAM can repl...
A 7B model in GGUF lands between 27B and 35B parameter models. For self-hosted deployment, K2 Horizon 7B shifts TCO calculations and makes VRAM efficiency the central metric, not parameter count. Yet deeper verification is needed.
A systematic evaluation across the Whisper family shows token merging boosts computational efficiency with almost no accuracy loss in multilingual transcription, even after DoRA fine-tuning. The method shortens token sequences during inference withou...
IFM's K2 Horizon 7B lands on the Artificial Analysis Intelligence Index between Qwen 3.6 27B and Qwen 3.6 35BA3b. In early local testing, it compiles the latest llama.cpp for CUDA without trouble. If the result holds, a 7B performing like much larger...
A pull request in llama.cpp adds support for Maple 20B-A1B, a ternary MoE LLM aimed at CPU inference. The open question is whether it can lower VRAM requirements for self-hosted deployments.
A 150-million-parameter model with performance similar to GPT-2 Small has been trained on a single RTX Pro 6000 using 7 billion tokens. Published benchmarks: PIQA 62.24%, Hellaswag 32.20%, Arc-Easy 44.91%, Arc-Challenge 25.00%, Arithmark 3.0 33.90%, ...
A self-hosted test with Qwen3.8-Flash-Next-IQ4_XS on llama.cpp produces a complex SVG from a creative prompt. The output impresses, but previews diverge: palms and music notes disappear in the browser. The case shows why vector code generation must b...
