## MoE Inference Engine: cuda-nn A new inference engine, named cuda-nn, has been developed using Rust, Go, and CUDA. This engine is specifically designed for the inference of MoE (Mixture of Experts) models and stands out for its ability to operate without relying on PyTorch. ## Key Features * **Languages:** Implemented in Rust, Go, with Python bindings to the same shared CUDA kernels. * **Architecture:** Supports MoE (Mixture of Experts) and MQA. * **Performance:** Optimized CUDA kernels (GEMM, RoPE, SwiGLU) written by hand to maximize efficiency. * **Parameters:** Handles models with up to 6.9 billion parameters. This project represents an interesting alternative for those looking to optimize the inference of large models, leveraging the power of CUDA and the flexibility of Rust and Go. The approach of manually developing CUDA kernels allows for more precise control over performance, potentially surpassing the performance achievable with more generic frameworks.