New quantization, sparse attention, KV cache and prompt-format methods aim to reduce latency, memory and token waste, especially for long contexts and small dense models.
A llama.cpp Q8_0 benchmark shows both fine-tunes cut median completion tokens by about 40% compared with Qwen3.8-27B. Pass rates remain similar, but Swift delivers a more uniform reduction, while ThinkingCap has a longer tail and different language-l...
PRQuant combines channel reorganization and static residual compensation to reduce inference overhead in low-bit quantization. The method is training-free, builds contiguous tail blocks offline, and avoids dynamic online gathering by turning scattere...
A system called phantom-kv injects a small bank of key/value tensors into a model's KV cache. The result is a per-request capability mode: load it, use it, unload it. The weights remain identical. The audit, however, highlights semantic limits and a ...
An end-to-end trainable architecture replaces rigid heuristics with dynamic contextual thresholds, reducing KV cache traffic without sacrificing quality. The 1.45 billion parameter model achieves up to 2.5x faster decoding than FlashAttention-2 on 51...
A training-free method reduces prefill cost in long-context LLMs by targeting 'mean dilution', where a block centroid hides relevant tokens. RBS-Attention combines a centroid-based branch with a rescue branch based on maximum key-block radius. On H10...
A post on r/ROCm highlights the problem of FP4-optimized inference engines: fast in benchmarks, but often unreliable on small dense models. Compression can work on Large Language Models with many parameters, but on small models logical errors become ...
UkisAI's question about Bonsai 2 Swift is not a simple update request. It highlights how the TCO of local LLMs increasingly depends on runtime behavior: wasted tokens, overthinking loops and quantization choices are redefining on-premise efficiency.
After Swift Qwen3.8 27B downloads jumped from 100,000 to 150,000 overnight, UkisAI is asking whether to apply the same token-efficiency treatment to Bonsai 2, which their tests show suffers from overthinking loops and high token usage. The team wants...
