When running large language models locally, video memory remains the most common bottleneck. Tensor parallelism is one of the most effective answers: it splits a model across multiple GPUs, allowing LLMs that would never fit on a single card to be loaded. Until now, however, this technique in llama.cpp was almost exclusively reserved for NVIDIA GPUs through CUDA. Pull request #25051 changes the landscape.

Piotr ‘pwilkin’ has worked to make tensor parallelism on the Vulkan backend “somewhat usable.” Vulkan is the low-level graphics and compute API natively supported by AMD, Intel, and even NVIDIA – as well as on mobile and integrated platforms. Now, models split across multiple AMD or Intel Arc GPUs can cooperate in a single inference session, lowering a barrier that relegated many accelerators to secondary roles or kept them out entirely.

Why Vulkan and tensor parallelism are a strategic match

The immediate benefit is the ability to pool VRAM from heterogeneous cards, even without NVLink or proprietary interconnects. In on-premise scenarios where organizations often repurpose workstations or servers containing GPUs from different generations and vendors, a unified compute backend avoids the need to buy homogeneous and costly hardware. The entire stack remains managed by llama.cpp, the lightweight C++ framework with no cloud dependencies that brought LLM inference to consumer CPUs and GPUs.

While CUDA is still the dominant choice in the enterprise, a functional tensor parallelism on Vulkan signals a maturing cross-vendor offering. It is no longer just about “also running on AMD,” but about distributed GPU compute becoming a standard building block for those building self-hosted inference pipelines. For teams bound by data sovereignty requirements and unable to rely on cloud services, hardware diversification translates into lower TCO and a more resilient supply chain.

What PR #25051 brings – and the remaining trade-offs

The pull request does not introduce new models or metrics, but cleans up and stabilizes the communication paths between multiple Vulkan devices, improving synchronization and tensor splitting. Followers of the repository know that Vulkan implementation often suffered from instability or crashes in multi-GPU configurations; pwilkin’s work directly addresses reliability, paving the way for deeper community testing.

Naturally, Vulkan tensor parallelism does not yet match the performance of CUDA libraries optimized with cuBLAS and NCCL. But for workloads where the priority is simply to run a model – perhaps with aggressive quantization to save VRAM – rather than to achieve maximum tokens-per-second, the gain in accessibility is enormous. It’s a classic trade-off between hardware flexibility and absolute throughput, part of a broader debate on AI democratization.

The context that matters: real on-premise, not lab setups

AI-RADAR closely follows the evolution of local stacks, and this PR is an example of how open-source software is progressively lowering the barrier to on-premise deployment. Allowing an organization to exploit GPUs already in inventory – or to mix NVIDIA and AMD within the same node – changes the economic equation of inference. Instead of investing in expensive homogeneous solutions, a modular approach becomes feasible, where each component is chosen based on cost-performance ratio, with Vulkan as the compatibility layer.

It is no coincidence that projects like llama.cpp are becoming the heart of many self-hosted stacks: from the small business serving an internal chatbot to research centers unable to send sensitive data to a public cloud. Extending tensor parallelism to Vulkan adds a crucial piece for scalability, bringing these scenarios closer to what was possible only with CUDA until yesterday. And as the community tests the path opened by pwilkin, one can expect heterogeneous multi-GPU support to become an increasingly solid and well-documented feature.