ToMoE starts from a straightforward observation: many LLM compression methods remove less important structures, but in doing so they permanently discard capacity that cannot be recovered. The result is a smaller model that is also more brittle. The paper described on arXiv — and linked to an ICML 2026 poster — flips this logic: instead of deleting parameters, it keeps them all and reorganizes them. Dense MLPs become a Mixture of Experts (MoE), and a differentiable dynamic pruning mechanism activates only a fixed number of parameters per token.

That distinction matters. In deployments on resource-constrained machines or on-premise serving, the cost per token depends not only on how much VRAM the model occupies but on how many parameters are actually crossed during computation. ToMoE targets the second factor: it reduces active parameters without touching the original checkpoint. It also does so without fine-tuning, a detail that lowers the barrier for teams already running dense models in production and unable to afford additional training cycles.

The paper reports that the method outperforms previous structural pruning techniques across several model families: Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5. The code is available on GitHub, so the conversion is not just a theoretical formula but a usable tool.

A shift in logic: dynamic sparsity, not amputation

Here is the structural change. Permanent pruning has always had a limit: deciding what to remove is irreversible, and mistakes are paid for in quality. ToMoE introduces a conditional sparsity instead: the full model remains available, but each input triggers only a subset of experts. That is close to the logic of natively trained MoE models, but applied to already existing dense networks.

For teams evaluating on-premise deployment, this changes incentives. Instead of retraining or pruning a network to fit a specific board, one can start from a dense checkpoint and obtain a variant with reduced active computation. AI-RADAR offers analytical frameworks at /llm-onpremise for teams weighing these trade-offs in local environments. The trade-off is that total memory does not shrink proportionally: all parameters remain in the checkpoint. ToMoE is not a replacement for quantization or offloading techniques, but a different way to cut computational cost per token.

The unresolved knot: runtime and memory

The second-order effect concerns serving runtimes. A converted MoE model requires inference engines able to handle expert routing without adding latency. MoE inference frameworks exist, but not every self-hosted environment is set up for this execution pattern. Moreover, teams with little VRAM do not fully solve their problem: if the complete checkpoint does not fit in memory, activating fewer parameters is not enough. The gain is in throughput and active computation, not in the static size of the artifact.

This also explains the request that appeared in the original discussion: convert recent dense models such as Qwen3.8-27B or Muse-Glimmer-30B. The community sees ToMoE as a way to unlock local LLMs without waiting for official MoE versions. That is a signal of how much the demand for on-premise efficiency is pushing toward post-hoc conversion tools rather than toward training already optimized models.

In short, ToMoE does not solve every hardware constraint, but it shifts the boundary: it makes it possible to recover computational efficiency from existing dense models without paying the price of permanent capacity loss. For teams managing local workloads, the message is clear: dynamic sparsity can become a deployment lever, provided the runtime and memory profile match the new execution model.