When External Guidance Helps LLM Reasoning: The Bias-Variance Theory of GA-GRPO

Reinforcement learning with verifiable rewards (RLVR) has become the dominant way to elicit multi-step reasoning in large language models. In practice, several recent methods add an external signal: expert traces, self-explanations, or retrieved thought patterns. LUFFY, ExPO, PAPO, and TAPO belong to this family and report empirical gains, but until now none has supplied convergence rates, bias bounds, or an optimal weighting rule for that signal.

The new Guidance-Augmented GRPO framework, GA-GRPO, closes this gap. It models external guidance as a stochastic operator G that rewrites the question distribution. The resulting policy-gradient estimator is a biased on-policy estimator, with bias bounded by the total-variation divergence δ_G between the guidance-augmented sampling distribution and the policy's own distribution. The framework subsumes vanilla GRPO, LUFFY, ExPO, PAPO, and TAPO as special cases.

The theoretical results are not abstract. Under smoothness and bounded-divergence assumptions, GA-GRPO converges at rate O(1/√T) to an O(δ√T)-neighbourhood of the GRPO stationary point. In other words, if the guidance drifts too far from the policy, bias becomes the dominant cost. The paper also derives the closed-form optimal weight: λ* = σ₀²/(σ₀² + R_max² δ² T). Because T appears in the denominator, the ideal weight for external guidance decreases as training progresses. This suggests guidance is most useful early on and should be progressively reduced once the policy stabilises.

That conclusion has direct implications for teams training models on local hardware. The experiments used Qwen2.5-Math-7B-Base across nine math and out-of-distribution benchmarks. Optimal-weight GA-GRPO matched or surpassed TAPO, LUFFY, ExPO, and vanilla GRPO while requiring 31 percent fewer GPU-hours. For on-premise or self-hosted training, where TCO is a critical variable, knowing when to reduce or switch off external guidance avoids paying for compute that adds bias rather than reducing variance.

There is a structural point worth noting. The minimax lower bound proved in the study shows that the Ω(δ² T) bias term is unavoidable: no procedure can eliminate it entirely if the guidance diverges from the policy. As a result, the value of an external trace dataset depends not only on content quality but also on its distributional proximity to the behaviour the model is learning. For benchmarks, this shifts the comparison from simple success rates to estimator stability and computational cost.

Eight analysis experiments validated each theoretical prediction, reinforcing the view that the theory is a practical guide rather than a formal exercise. For those fine-tuning reasoning LLMs with limited GPU budgets, the lesson is clear: external guidance should be treated as a control signal to calibrate over time, not as a plugin that is always active. The measured GPU-hour saving does not guarantee the same result on every workload, but it signals that optimal weighting can lower the marginal cost of training without sacrificing quality.