LLM Inference on Existing Hardware: The Case of AMD MI50s

The adoption of Large Language Models (LLM) in enterprise environments poses significant infrastructure and cost challenges. While cloud solutions offer immediate scalability, data control, sovereignty, and Total Cost of Ownership (TCO) drive many organizations to consider on-premise deployments. In this context, optimizing existing hardware or older solutions becomes a key factor. A recent benchmark explored precisely this possibility, testing the performance of the Qwen 3.6 27B model on AMD MI50s GPUs, cards released in 2018.

The results are remarkable: the system achieved a throughput of 52.8 tokens per second (tps) for text generation (Token Generation, TG) and a significant 1569 tps for prompt processing (Prompt Processing, PP). These figures, obtained with a full-precision model and without resorting to Quantization techniques, open new perspectives for those evaluating LLM implementations in self-hosted environments, demonstrating that even older hardware can offer competitive Inference capabilities.

Technical Details and Benchmark Methodology

The benchmark was conducted using a specific configuration, based on a vLLM fork (version 0.20.1) optimized for ROCm 7.2.1 and the gfx906 architecture of the MI50s, all containerized via Docker. This choice underscores the importance of an efficient Inference Framework and a software stack well-aligned with the underlying hardware to maximize performance.

The test utilized the Huggingface Qwen 3.6 27B model, performing a single Inference with two different prompt sizes: one of 1,000 and one of 15,000 tokens. It is important to note that the model was run at full precision (float16), without the use of Quantization, a technique that reduces memory footprint and accelerates Inference at the expense of potential accuracy loss. The configuration used Tensor Parallelism (TP8), although it was observed that the unquantized model also fits with TP2, still offering a throughput of around 34 tps TG. The decision not to use optimizations like MTP (Multi-Token Pre-fill) or DFlash for large prompts highlights an approach aimed at evaluating baseline performance in specific scenarios.

Implications for On-Premise Deployments and TCO

For CTOs, DevOps leads, and infrastructure architects, the results of this benchmark are particularly relevant. The ability of GPUs like the AMD MI50s, an older generation of cards, to handle complex LLM workloads at full precision has direct implications for the TCO of on-premise deployments. Utilizing existing hardware or hardware that can be acquired at lower costs compared to the latest high-end GPUs can significantly reduce initial investment (CapEx).

This approach reinforces the feasibility of self-hosted solutions that prioritize data sovereignty and compliance, crucial aspects for regulated industries or companies with stringent security requirements. The ability to keep data and models within one's own infrastructure perimeter, even with non-cutting-edge hardware, offers unparalleled control compared to cloud-based alternatives. Furthermore, there is room for improvement, as suggested by the developers, through updating the software and hardware stacks (e.g., using low-latency PCIe switches or more aggressive DFlash/MTP optimizations), indicating a clear path to further refine performance.

Future Prospects and Final Considerations

The tests demonstrate that the achieved performance is fully usable for applications requiring conversational agents or other agentic Frameworks. This opens the door to a wide range of enterprise use cases, from code generation to internal process automation, all while maintaining the benefits of a controlled and secure environment. The continuous evolution of Open Source Inference Frameworks like vLLM, along with support for diverse hardware architectures, is fundamental to democratizing access to LLM capabilities.

In conclusion, the AMD MI50s benchmark with Qwen 3.6 27B provides a clear indication that on-premise LLM Inference is no longer the exclusive domain of the latest generation hardware. For organizations seeking a balance between performance, TCO, data sovereignty, and control, optimizing local stacks and carefully selecting hardware represents a winning strategy. AI-RADAR will continue to monitor these developments, providing in-depth analyses of the trade-offs and constraints that guide deployment decisions in the artificial intelligence landscape.