Teams that combine a small on-device model with a larger LLM on an edge server know that acceleration is not free: during speculative token verification, the device and the server must exchange vocabulary information. Existing methods assume that both models share the same token set; when they do not, communication load grows considerably.

Collaborative speculative decoding works like this: the local model drafts a sequence of tokens, and the server verifies it with the larger model. But correcting the sampling requires communicating the probability distribution over tokens, and this exchange becomes heavier as the vocabularies diverge.

X-CoSD introduces hybrid resampling. It splits residual resampling into two parts: the common-token region is handled on the device, while the region exclusive to the LLM is handled on the server. Distribution transmission is therefore needed only for the shared region, not for the entire vocabulary. This architectural simplification reduces communication load without altering the server model's distribution.

The X-CoSD-E variant shifts the center of gravity further toward the server. With server resampling and device verification, the server sends only replacement candidates sampled from its own LLM and their corresponding probabilities. The device performs local verification instead of receiving or sending complete distributions. According to the authors, the server LLM distribution is preserved in this case as well.

The experiments reported in the paper indicate a significant improvement in token generation speed while maintaining quality comparable to that of the server LLM. The abstract does not provide quantitative detail, but the mechanism is presented as lossless.

The structural point is that X-CoSD attacks an often underestimated bottleneck in hybrid deployments: the tight coupling between the local model and the remote model. Until now, teams wanting to use a small model to reduce latency for a service based on a larger LLM had to align vocabularies or accept communication overhead. With a method that natively handles heterogeneous vocabularies, the choice of the local model becomes more independent. This changes incentives: on-device models can be selected for efficiency, task fit, or memory constraints without having to replicate the tokenization head of the server model.

There is also an effect on data sovereignty. Reducing the volume and granularity of exchanged distributions shrinks the information surface exposed between device and server. The E variant, in particular, sends only the candidates deemed necessary and their probabilities, not the entire distribution over the vocabulary. This is not a privacy guarantee, but it is an architectural shift that can make scenarios more acceptable in which generation must remain partly anchored to the device.

For teams designing edge or hybrid infrastructure, the message is that communication cost depends not only on model size but also on how distributions are managed during sampling. Optimizing the speculative verification protocol can be cheaper than increasing bandwidth or moving the entire model on-premises. It signals that the next round of efficiency in distributed inference will also come from framework-level choices, not only from more powerful hardware. In TCO terms, reducing distribution traffic touches a recurring operational cost, especially when inference is distributed across many devices.

For those evaluating these edge-versus-server trade-offs, AI-RADAR offers analytical frameworks at /llm-onpremise to compare costs and constraints across deployment architectures.

The aspect to watch will be behavior when the common vocabulary region is very small compared with the server region: the principle of region-based resampling is the contribution on which concrete implementations will be most interesting to see.