Scalable Sampling for LLMs: Training-Free Reasoning

A new study introduces a method to improve the reasoning capabilities of large language models (LLMs) without the need for costly post-training reinforcement learning processes. The technique, called Scalable Power Sampling, focuses on sharpening the model's distribution to achieve superior performance.

Solution Details

The proposed method eliminates the need for Markov chain Monte Carlo (MCMC) iterations, often associated with high computational costs. The key innovation lies in a novel formulation that approximates the global power distribution through a token-level scaled low-temperature version, where the scaling factor captures the quality of the future trajectory. This allows the base model's generative distribution to be sharpened autoregressively and without training.

Experimental results

Empirical evaluations on math, question answering, and code tasks, using different LLMs, demonstrate that the method matches or surpasses the performance of GRPO (a reinforcement learning method) without the use of external rewards. Furthermore, a reduction in inference latency of over 10x is observed compared to MCMC-based methods.

For those evaluating on-premise deployments, there are trade-offs to consider. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.