At PyTorch Conference 2025, AMD demonstrated linear scaling beyond 1,000 Instinct GPUs using Primus-Turbo, an optimization library for training frameworks such as TorchTitan. Those optimizations have now flowed into mainline PyTorch: TorchTitan supports Instinct GPUs directly and delivers competitive FP8 performance out of the box. This is not a single benchmark story. It is a maturity signal for ROCm inside the most widely used open research and production framework.

The numbers describe two different scenarios. On dense models, FP8 training yields a 13.4% throughput gain over BF16 on Llama3-8B with 8×MI300X, at nearly identical peak memory. The advantage comes from faster FP8 matrix cores, not memory savings. On MoE architectures like DeepSeek-V3 671B, FP8 quantization initially added significant overhead: activations need per-row scales, expert weights need per-column scales, and an offset tensor routes tokens. The Triton quantization pipeline then recovered 89% of that overhead, taking the forward pass from about 19 ms to about 7 ms.

Format correctness comes before tuning

Instinct GPUs use an FP8 variant called e4m3fnuz, with a maximum value of 240 and no NaN or Inf encodings. TorchAO initially computed scales against a different maximum. On AMD hardware this produced silently wrong results: tensors were scaled beyond representable values, clipping activations and corrupting gradients. Because the format has no NaN/Inf encodings, the overflow raised no error. AMD therefore added hardware auto-detection in TorchAO: choosing the correct format is a correctness requirement, not a performance preference.

The real gain is in data movement

The FP8 quantization pipeline in TorchAO was a chain of separate steps: compute per-row or per-column absolute max, derive the scale, clamp and cast to FP8. Each step launched a kernel and materialized intermediate tensors in HBM. For MoE models with dozens of expert weight tensors per layer, those round-trips dominated the overhead. The AMD optimizations attack data movement at three levels.

The first reduces the number of kernels. In the backward pass, a .t().contiguous().t() pattern forced a full copy through HBM and was removed; the scale-and-cast chain was then fused into single Triton kernels. On DeepSeek-MoE-16B, backward throughput improved by 4.2×. In the forward pass, the fusion replaced five generic kernels per call with a single specialized kernel parallelized over experts and output-dimension blocks. On 8×MI325X with DeepSeek-V3 671B, end-to-end throughput rises from 5,996 to 7,027 tokens/s, recovering 89% of the gap versus the BF16 baseline.

The second level makes memory access efficient. The column-wise scales kernel had non-coalesced writes: consecutive SIMD lanes wrote to distant addresses, each triggering a separate transaction. By transposing the output tile through Local Data Share before storing, the time per MoE layer dropped from 7,290 to 1,170 microseconds, a 6.2× speedup.

The third level removes unnecessary synchronization. Triton atomic operations defaulted to acquire-release ordering, which on AMD GPUs inserts memory fences before and after every atomic. Switching to relaxed ordering removed expensive synchronization points that were unnecessary for commutative reductions.

What this means for teams running local clusters

There was also an experiment that did not work. Expanding the autotuning search space from 1 to 8-16 candidate configurations produced no measurable improvement on Llama 4 shapes with MI300X and increased first-iteration compile time; AMD reverted the change. The lesson is precise: the search space should be shaped by hardware constraints such as wavefront size, LDS capacity, and register pressure, not expanded arbitrarily.

The fact that these gains are in mainline PyTorch, with no AMD-specific components to install, lowers adoption and maintenance costs for teams using Instinct GPUs in their own datacenters. No framework forks or vendor tools are needed: updating TorchAO and TorchTitan is enough. For organizations that keep data in their own environments, an alternative to NVIDIA becomes more credible on the software side as well. AMD's collaboration with Meta/PyTorch engineers indicates that ROCm is entering the upstream development cycle, not just receiving ports. The stated next step is MXFP8 for MI355X GPUs: the competition is moving toward fine-grained quantization formats. One point remains open: the FP8 gain on dense models is solid, but efficiency on MoE models requires deep fusion work, not simply enabling the format.