A single GEMM kernel that covers three algorithmic variants, plus an autotuner that picks the best configuration shape by shape. That is the bet behind Helion's linear backend for vLLM, the result of joint work by Red Hat's OCTO group and Meta's Helion team. The integration aims to reduce kernel development complexity and improve LLM inference performance on NVIDIA Hopper GPUs, focusing on the 8-bit quantization formats most common for efficient inference: FP8_Dynamic, W8A8_INT8, and Block_FP8.

Normally, good performance on 8-bit GEMMs requires multiple specialized implementations. Split-K partitions the K dimension across multiple thread blocks when M or N are too small; Swap-AB rewrites A@B as ([email protected]).T to improve shapes with a small M dimension. In vLLM, for example, the Block_FP8 backend uses Swap-AB when M is below 32 on Hopper. These heuristics work on average but are not optimal for every model or workload. With Helion, a single unified GEMM implementation covers Standard GEMM, Split-K, and Swap-AB: the algorithmic choices are exposed as tunable parameters, and the ahead-of-time autotuner selects the best variant and configuration for each input shape.

Helion is a PyTorch-native, hardware-agnostic kernel DSL based on a tile-programming model. It lets developers write a kernel once in concise Python code and generate specialized code for different platforms. The AOT autotuner systematically explores a search space spanning memory layout, scheduling, and higher-level algorithmic choices. In this work, split_k and swap_ab are registered as tunable parameters, with split_k constrained to power-of-two values up to 256. The autotuner compares combinations and picks the best one for each workload.

The integration uses a hybrid dispatch strategy based on runtime token count and CUDA Graph coverage. For small shapes up to max_helion_size, the linear backend uses Helion under CUDA Graph replay; beyond that threshold, it falls back to the default CUTLASS or DeepGEMM kernels. This work sets the threshold to 32. That eliminates the CPU overhead of Helion kernel dispatch and launch outside CUDA Graphs, reduces the number of shapes that need tuning, and makes pre-tuned configs easier to maintain.

On performance, the reported results are solid. On an NVIDIA H100 80GB HBM3 GPU, Helion kernels achieve geometric mean speedups of 1.110× over CUTLASS for FP8_Dynamic, 1.178× over CUTLASS for W8A8_INT8, 1.149× over FlashInfer, and 1.177× over DeepGEMM for Block_FP8. The speedup distribution varies by shape, confirming the value of per-shape tuning over a single kernel config or generic dispatch heuristic. In end-to-end benchmarks using vLLM serve and the ShareGPT dataset, the Helion backend showed consistent throughput gains across several Qwen3 model sizes, with improvements above 10% in some workloads.

The structural point is not just the percentage gain. Helion shifts kernel optimization from a manual craft reserved for a few low-level GPU programming specialists to an automated numerical search problem. That changes incentives: a team serving a specific model on self-hosted infrastructure can obtain configurations tailored to its own workload without maintaining deep CUDA expertise in-house. The cost moves instead to ahead-of-time tuning, which can take hours, and to managing configuration files for popular models.

Here the performance-usability-maintainability trade-off emerges. Finer tuning improves performance but increases client-side autotuning effort or the burden of shipping pre-tuned configs. For the GEMM kernels studied, the team is evaluating a model in which Helion kernels and the integration framework are maintained upstream with a default config for functional testing and CI, while workload-specific autotuning is delegated to end users. The automated tooling lets users generate optimized configs for their models and hardware before deployment. This favors performance and maintainability over out-of-the-box usability, a realistic trade-off for production serving teams willing to invest in offline optimization.

It is not the only path. For auxiliary kernels such as quantization, activation, and normalization, initial experiments suggest meaningful improvements can be achieved with a small number of configs, lowering maintenance cost and opening another route toward upstream adoption.

The work is available today in a vLLM fork and described by the authors as ready for production use. On hardware, the Helion team is extending support to NVIDIA Blackwell via the CuteDSL backend, with ongoing work on AMD GPUs and TPUs. Extending coverage to MoE models, where the MoE backend becomes the main optimization target, is another stated goal. For teams running LLM inference in-house, the signal is clear: default framework kernels are not a fixed ceiling, but workload-specific optimization carries a process cost that needs to be accounted for in deployments and model updates.