Methods like ToMoE, KVBoost and MTP reduce active parameters or reuse key-value cache chunks, cutting compute per token without full retraining.
ToMoE turns dense checkpoints into MoE without fine-tuning, lowering active parameters per token without reducing memory footprint. A shift for on-premise deployments: less compute, same model. Runtime, latency, and VRAM constraints remain. The signa...
KVBoost reuses key-value cache at chunk level and reduces time-to-first-token by 4.49x on Qwen2.5-3B, with no accuracy loss. Unlike prefix caching, it does not require a shared leading prefix: a dual hash separates positional and content identity. As...
A new method converts dense LLMs into Mixture-of-Experts models through differentiable dynamic structural pruning, reducing active parameters per token without removing them. It needs no fine-tuning and outperforms previous structural pruning on Phi-...
llama.cpp enables MTP for GLM-4.5-Air, a 106-billion-parameter MoE with only 12 billion active parameters. The change speeds up inference on memory-rich but compute-limited machines like Strix Halo, DGX Spark, and RTX 3090, and strengthens the fine-t...
