Sparse models and mixture-of-experts designs are lowering memory and compute costs, while new training techniques improve efficiency further.
A Mixture-of-Experts training system cuts cumulative MoE-layer training time by 21.2 to 44.8 percent against SYMI by reducing memory demand in backpropagation and rebalancing retained work. LRMoE plans expert placement and retention quotas, selects t...
At EMNLP 2026, T-Tech researchers presented LIMe, an attention extension that reuses key-value buffers from earlier layers. In tests with models around one billion parameters, training reached the same loss as the LLaMA baseline using 15% less comput...
DiffuSpace has released Clover-35B-A3B-Preview, an open-source diffusion language model with a 35B MoE backbone, about 3B active parameters per token, and a 128K context window. The per-token active count, not total size, is the key metric for self-h...
The Ling-3.1-flash-1019 checkpoint is now on Hugging Face along with a BF16 base model for custom post-training. Total parameters remain 560B with 25B active and a 1M token context window. Reported scores move from 41 to 44 on AA Intelligence Index v...
inclusionAI updates Ling-3.1-flash: 560B total parameters, 25B active per token, and a 1M token context window. Agentic and scientific benchmarks improve. The base checkpoint is opened for research. The article analyzes implications for self-hosting ...
