When LLM prefill and decode run on separate GPU pools, each request carries a KV cache that must be transferred: 2.6 GB for a 70-billion-parameter model, exceeding 100 GB/s at production scale. The problem is that today’s systems—DistServe, Splitwise, Mooncake—treat the interconnect as a homogeneous fabric, issuing uniform RDMA without regard for the physical hierarchy.

Yet the bandwidth between two GPUs varies by a factor of 72 depending on their location: 900 GB/s via NVLink within a domain, 50 GB/s via InfiniBand across nodes, 12.5 GB/s over TCP across data centers. Ignoring this scale leaves performance on the table, performance that can make the difference between a fluid service and a jittery one.

A research group has designed a topology‑aware orchestrator that, at startup, discovers the interconnect hierarchy and selects the optimal transport for each transfer. Three mechanisms work together. First, a pipelined layer‑by‑layer transfer that overlaps transmission with ongoing prefill, hiding 60 to 85 percent of latency behind computation. Second, NVLink domain‑aware placement for Mixture‑of‑Experts models, co‑optimizing expert dispatch with KV cache locality. Third, the use of CXL 3.0 memory expanders as a shared overflow tier, delivering 6x the capacity and 86x lower latency than NVMe.

Full evaluation would require multi‑node clusters with heterogeneous interconnects and CXL 3.0 hardware—still absent from cloud offerings and beyond academic resources. The researchers present analytical bandwidth models, component implementations, and projected analysis across three architectures, showing a 3 to 18‑fold reduction in transfer latency compared to uniform RDMA.

The signal for anyone building on‑premise clusters

For those managing local AI infrastructure, this is not an academic curiosity. Disaggregated serving is almost unavoidable if you want to keep the cost of the most powerful GPUs under control, but until now it was seen as a prerogative of large clouds. The arrival of a topology‑aware orchestrator and, crucially, integration with CXL 3.0 shifts the center of gravity: the ability to extend effective memory without multiplying expensive GPUs is a concrete argument for anyone who wants data to stay under their own control.

The competitive advantage here is not only about latency. CXL shared memory eliminates the need to move the KV cache to disk, a step that with NVMe introduces latencies incompatible with interactivity. Those setting up an on‑premise cluster can now design the NVLink domain as a scheduling unit, reducing off‑domain transfers to the unavoidable minimum and letting CXL act as a fast buffer. No cloud vendor currently offers this combination: early adopters gain an efficiency edge that will be hard to close.

On a structural level, this research marks the end of the illusion that the data center is a flat cloud. The most demanding AI applications require scheduling to become aware of the electrical distance between components. Anyone building AI operating systems will have to abandon the undifferentiated network model that dominated the first cycle of distributed inference, or be overtaken by more specialized architectures. TCO, for large‑scale workloads, will depend less on the cost of a single GPU and more on the efficiency of data transfer between compute pools. It’s a shift that rewards those with direct access to hardware—and the ability to experiment with heterogeneous topologies—and penalizes those who merely consume opaque cloud services, where the topology remains hidden and efficiency cannot be customized.