New methods such as chunk-level KV cache reuse, dense-to-MoE conversion, and entropy-constrained quantization are reducing prefill time and active compute without fine-tuning. These techniques target cheaper local and on-premise LLM serving.
ToMoE turns dense checkpoints into MoE without fine-tuning, lowering active parameters per token without reducing memory footprint. A shift for on-premise deployments: less compute, same model. Runtime, latency, and VRAM constraints remain. The signa...
KVBoost reuses key-value cache at chunk level and reduces time-to-first-token by 4.49x on Qwen2.5-3B, with no accuracy loss. Unlike prefix caching, it does not require a shared leading prefix: a dual hash separates positional and content identity. As...
A new method converts dense LLMs into Mixture-of-Experts models through differentiable dynamic structural pruning, reducing active parameters per token without removing them. It needs no fine-tuning and outperforms previous structural pruning on Phi-...
ECASQ jointly optimizes adaptive quantization and lossless compression by minimizing MSE under an entropy budget and an unbiasedness constraint. The optimal dynamic program runs in O(sd^2) time and O(d^2) space. A GPU-friendly approximate version red...
