Faster LLM Inference with Speculative Verification

Large language models (LLMs) based on Chain-of-Thought reasoning offer high performance on complex tasks, but generating long sequences leads to high latency. Step-level speculative reasoning aims to reduce this cost, but has so far faced a trade-off between accuracy, inference speed, and resource efficiency.

ConfSpec: Confidence-Gated Cascaded Verification

ConfSpec is a cascaded verification framework that overcomes this trade-off. The key idea is that verifying a single reasoning step is a simpler discriminative task than generation. ConfSpec uses smaller models for verification, directly accepting high-confidence decisions and escalating uncertain cases to the larger target model.

Results and Benefits

Evaluations show that ConfSpec achieves speedups of up to 2.24x while maintaining the accuracy of the target model. The method requires no external judge models and is compatible with token-level speculative decoding, allowing for further acceleration. This approach can lead to a significant reduction in inference costs, especially in on-premise scenarios where resource optimization is critical.

Implications for Deployment

The efficiency of ConfSpec makes it particularly interesting for deployment scenarios where latency and TCO are critical factors. The ability to use smaller verification models reduces hardware requirements, making it possible to run LLMs even on infrastructures with limited resources.