Streamlining LLM Deployment on aarch64 Architectures

The landscape of Large Language Models (LLM) is constantly evolving, with growing interest in on-premise deployment solutions that ensure data sovereignty and cost control. However, implementing these frameworks on specific hardware architectures can present unexpected challenges. A significant example has been the deployment of LLMs like vLLM on aarch64 Linux systems equipped with GPUs, such as NVIDIA GH200, GB200, and GB300 platforms. For years, developers faced hurdles in installing CUDA-enabled PyTorch, a fundamental component for hardware acceleration.

The problem lay in the distribution of PyTorch wheels. Until version 2.10, a simple pip install torch command on aarch64 systems would automatically download the CPU-only version of PyTorch from the default PyPI repository. This forced users to resort to complex workarounds, significantly slowing down the deployment process and increasing indirect Total Cost of Ownership (TCO) due to time spent on debugging and configuration.

The Technical Challenge and Interim Workarounds

The issue was not limited to needing a custom index-url to obtain CUDA-enabled PyTorch wheels. The true complexity arose with transitive dependencies. If any package in vLLM's pipeline declared a specific torch version requirement, pip would revert to the default PyPI index, silently uninstalling the previously installed CUDA-enabled build and replacing it with the CPU-only version. This led to frustrating debugging sessions where the system failed to detect the GPU, despite an seemingly correct installation.

To mitigate this inconvenience, the vLLM team developed temporary solutions. One such solution was the use_existing_torch.py script, introduced in September 2024. This script modified vLLM's dependency files, removing torch, torchvision, and torchaudio requirements to prevent pip from overwriting the existing CUDA-enabled installation. Later, as uv matured, a more elegant option was adopted: adding [tool.uv] no-build-isolation-package = ["torch"] to pyproject.toml. This configuration instructed uv to reuse the torch installation already present in the environment, avoiding unwanted re-installations. While effective, these solutions represented an additional burden for developers and an improvisation around a gap in the packaging standard.

Collaboration and the Definitive Resolution

The breakthrough came through collaboration within the PyTorch Foundation. Kaichao You from Inferact, representing vLLM on the Foundation's Technical Advisory Committee (TAC), formally raised the issue in January 2026, after tracking it in a GitHub issue since August 2025. The request was clear: publish aarch64 CUDA-enabled wheels directly to the default PyPI index, replicating the deployment experience already established on x86 architectures.

The NVIDIA engineering team played a crucial role, pushing for the publication of CUDA SBSA wheels to PyPI and leading the "small wheel" approach that dynamically links to libraries like NCCL and cuBLAS. This allowed wheel sizes to remain manageable, a critical aspect for PyPI maintainers. The PyTorch Foundation demonstrated its effectiveness in coordinating infrastructure-level issues across different projects, transforming a technical obstacle into an opportunity for systemic improvement.

Implications for On-Premise Deployments and Developer Experience

The resolution of the problem was confirmed in April 2026 with the release of PyTorch 2.11.0. Now, a simple pip install torch on aarch64 Linux systems correctly installs the CUDA-enabled version, eliminating the need for custom index-urls or complex dependency management logic. This seemingly minor change has a significant impact on developer experience and on-premise deployment strategies.

For organizations investing in self-hosted infrastructures based on aarch64 silicon, such as NVIDIA Grace Hopper and Grace Blackwell systems, this means faster and less error-prone deployment for LLM frameworks. The reduction in time spent on configuration and debugging directly translates into a lower TCO and greater agility in developing and releasing AI applications. While vLLM's workarounds remain useful for advanced scenarios (like custom PyTorch builds), the default path is now significantly simplified, making the adoption of these powerful platforms more accessible. For those evaluating on-premise LLM deployment, AI-RADAR offers analytical frameworks on /llm-onpremise to assess the trade-offs between control, performance, and costs.