New techniques are making local LLM serving faster and cheaper, including low-bit quantization, GGUF-native loading, CPU and GPU kernel improvements, Vulkan, and memory-augmented model variants. These advances target real-world VRAM and compute constraints.
A new approach replaces attention in Transformers with bottleneck autoencoder modules plus iterative refinement at masked positions. In tests with BERT and C4 it reaches a significant share of attention performance with about 1.9× fewer FLOPs and mat...
A test on 50 MATH-500 benchmark questions shows that penalizing hesitation tokens via logit bias improves accuracy for Qwen3.5-4B in GGUF format across all quantization levels tested. BF16 goes from 74% to 84%, while Q2_K doubles from 12% to 24%. The...
The open-source runtime for local LLM inference brings a change that makes prompt lookup drafting 42 times faster. This is not full generation speed, but a key component of speculative decoding: for self-hosted deployments, it can cut overhead and TC...
A pull request in llama.cpp introduces a tiled matrix-vector multiplication for k-quant formats. According to the report, using VNNI instructions speeds up CPU mul_mat by 3-7x with minimal added complexity. The improvement targets prompt prefill, mak...
Qwengram-0.8B freezes Qwen3.5-0.8B and adds a 51-billion-parameter n-gram memory outside the GGUF. Perplexity drops by 5.05%, but the relevant signal is architectural: late injection controlled by a per-token gate, Q8_0 quantized sidecar, and no back...
A sparse Reddit note points to Oh My Pi and custom model support through vLLM, llama.cpp, and SGLang. The real story is structural: self-hosted inference still depends on multiple runtimes with distinct hardware profiles, and integration—not a single...
An experiment keeps Qwen3.5-0.8B frozen, adds a 51-billion-parameter PLE memory and an R=1 reader: validation perplexity drops by 5.05%. The author warns this is not benchmark accuracy gain and releases GGUFs, a modified llama.cpp runtime, and a quan...
An open source contribution adds OpenVINO support to Laya: response time drops to 40 milliseconds per question on CPU, 3.4 times faster than PyTorch. The result shows how runtime optimization can reduce latency and cost for lightweight inference work...
A llama.cpp Q8_0 benchmark shows both fine-tunes cut median completion tokens by about 40% compared with Qwen3.8-27B. Pass rates remain similar, but Swift delivers a more uniform reduction, while ThinkingCap has a longer tail and different language-l...
The local AI server project Lemonade has released 2026.39.1 and a 2026.40 release candidate. Along with a new versioning scheme, it fixes AMD APU model streaming and removes OpenMOSS support on ROCm, reported as about 40 times slower than Vulkan. A r...
Hugging Face has added native GGUF support to Transformers. Users can load a quantized model by specifying the file path and use standard APIs. On supported Apple Silicon, ggml kernels are reused, with M2 Max benchmarks close to llama.cpp. The goal i...
PRQuant combines channel reorganization and static residual compensation to reduce inference overhead in low-bit quantization. The method is training-free, builds contiguous tail blocks offline, and avoids dynamic online gathering by turning scattere...
A system called phantom-kv injects a small bank of key/value tensors into a model's KV cache. The result is a per-request capability mode: load it, use it, unload it. The weights remain identical. The audit, however, highlights semantic limits and a ...
SupraLabs has released Supra2-IMG, a 100M parameter DiT text-to-image model trained from scratch in under 10 hours on a single H100. It generates 256×256 images with competitive quality and runs locally: about 20 seconds on CPU and 2 on GPU. A concre...
