New techniques such as KV cache reuse, dynamic MoE conversion, stochastic quantization, MTP speculative decoding and depth pruning are reducing compute and memory requirements for local LLM inference.
ToMoE turns dense checkpoints into MoE without fine-tuning, lowering active parameters per token without reducing memory footprint. A shift for on-premise deployments: less compute, same model. Runtime, latency, and VRAM constraints remain. The signa...
KVBoost reuses key-value cache at chunk level and reduces time-to-first-token by 4.49x on Qwen2.5-3B, with no accuracy loss. Unlike prefix caching, it does not require a shared leading prefix: a dual hash separates positional and content identity. As...
A new method converts dense LLMs into Mixture-of-Experts models through differentiable dynamic structural pruning, reducing active parameters per token without removing them. It needs no fine-tuning and outperforms previous structural pruning on Phi-...
llama.cpp enables MTP for GLM-4.5-Air, a 106-billion-parameter MoE with only 12 billion active parameters. The change speeds up inference on memory-rich but compute-limited machines like Strix Halo, DGX Spark, and RTX 3090, and strengthens the fine-t...
A single developer reduced Qwen3.8-27B to 22.7 billion parameters with depth pruning, no fine-tuning. Distributed only as MLX for Apple Silicon, the model shows trade-offs: lower memory and compute pressure, but losses on edge cases. For on-premise d...
ECASQ jointly optimizes adaptive quantization and lossless compression by minimizing MSE under an entropy budget and an unbiasedness constraint. The optimal dynamic program runs in O(sd^2) time and O(d^2) space. A GPU-friendly approximate version red...
A developer applied depth pruning to Qwen3.8-27B, bringing it to roughly 22.7 billion parameters without fine-tuning. The model, available in bf16, q8, and q4 on MLX, handles coding, agentic use, and multi-turn conversations with limited degradation,...
