Bringing a new accelerator into an LLM serving stack is usually a project measured in months. The IBM team working on Spyre chose a different path: in less than a week, two developers and their AI coding agents connected SGLang to the enterprise accelerator with a plugin of about 1,500 lines. The result is not a lab experiment. SGLang on Spyre achieved roughly 3.6× the throughput of the reference PyTorch inference implementation based on torch-spyre and performance on par with vLLM on the measured workloads for Granite 3.3 8B and Gemma 4 26B A4B.

Underlying this is the hardware support already available in PyTorch. Torch-spyre registers Spyre as a PyTorch device and connects it to torch.compile, exposing accelerator operations through standard APIs. Instead of rewriting kernels or allocators, the team reused existing code. That is not a minor detail: the integration work became mostly a matter of connecting two existing abstractions. The AI agents used the PyTorch implementations of Spyre operations already present in the vLLM backend as reference and adapted the code to SGLang’s interfaces.

Line count is not neutral

The SGLang plugin is about 1,500 lines, compared with roughly 3,500 for the existing vLLM backend for Spyre. The difference should not be read as a contest of brevity. The team explains that about two-thirds of the gap comes from the fact that SGLang’s attention base class supplies much of the surrounding logic: the Spyre backend only has to override the inner scaled dot-product attention operation. In vLLM, that logic had to be implemented more extensively. For anyone maintaining accelerator integrations, this is a signal of how the internal structure of a serving framework can reduce or increase maintenance cost.

SGLang, however, was not ready for a non-CUDA device without changes. The work required four small patches: two made existing execution paths available without CUDA-specific dependencies, and two registered Spyre as a supported device and attention backend that does not use Triton. The first two are reusable by other accelerators; the latter are specific to Spyre. Prefix sharing through RadixCache, by contrast, worked without any Spyre-specific changes: prefix sharing is resolved in the scheduler before execution reaches the accelerator. That separation of tasks allowed new code to focus on compute primitives rather than memory planning. The published usage example sets max_running_requests=1 and disable_cuda_graph=True, a sign that the initial integration prioritizes correctness over hardware saturation.

The signal for on-premise inference

This integration says something structural about the enterprise accelerator market. For years, bringing a serving framework to a non-NVIDIA processor has meant rebuilding significant parts of the stack, often tied to CUDA. Here the path was different: the hardware vendor invests in PyTorch and torch.compile support, while SGLang’s extension points define where to connect the code. The result is that a two-person team, with the help of AI-powered coding agents, can reach a working backend in days. This lowers the entry barrier for non-mainstream accelerators and shifts the incentive: differentiation is no longer only about compute power, but about the quality and maintainability of software support.

It is also an indicator for those evaluating self-hosted deployments. A serving framework like SGLang is not a monolithic component tied to a single platform. Extension points, the device-agnostic KV pool, and operator dispatch make it possible to connect specialized hardware without rewriting the scheduler, request handling, and serving loop. The Spyre case does not guarantee that every accelerator will follow the same trajectory, but it shows that rapid integration is possible when hardware support is expressed in PyTorch. For those watching on-premise deployment trade-offs, AI-RADAR publishes analytical frameworks at /llm-onpremise to evaluate these scenarios without commercial shortcuts.

The next stated step is support for dynamic shapes and, later, Triton and RadixAttention, with the goal of opening further kernel optimizations. If those pieces arrive, the boundary between specialized hardware and open-source serving stacks will become even thinner.