## MoEBlaze: Efficiency in Training MoE Models The training of large-scale Mixture-of-Experts (MoE) models is often limited by the memory wall bottleneck. The sparse architecture of MoEs introduces significant overheads, related to the management of token routing buffers and the need to materialize intermediate tensors. This limits the maximum batch size and sequence length manageable by GPUs, negatively impacting performance and model scaling. MoEBlaze is a framework designed to address these challenges. It optimizes memory usage during MoE model training through a co-design approach that includes: * An end-to-end method for token dispatch and MoE training, with optimized data structures to eliminate intermediate buffers and activation materialization. * Co-designed kernels with activation checkpointing techniques to reduce the memory footprint, while simultaneously improving performance. Preliminary results indicate that MoEBlaze can achieve a speed increase of over 4x and memory savings of over 50% compared to existing MoE frameworks. This represents a significant step forward towards more efficient and scalable training of MoE models on modern hardware.