In asynchronous LLM post-training, updating weights as soon as a rollout arrives has become the practical default: it trades batch regularity for training efficiency. The price is instability while the value model is still cold. The Li²SAO paper starts from this point and proposes a variant that closes the early gap without a pretrained value model.

The comparison begins with GRPO and SAO. The group-wise sampling in GRPO, the authors argue, does not fit asynchronous training well because rollout lengths follow a long-tailed distribution. SAO was introduced to improve stability and off-policy behavior, especially on agentic tasks. But it relies on a closed-source value model built through scaled value pretraining. LiSAO is the lightweight reproduction without that external model.

In tests on single-turn reasoning and multi-turn agentic tasks, the ranking flips. GRPO outperforms LiSAO on reasoning, while the opposite holds on agentic tasks. The real issue is early training: LiSAO's value model is not yet well trained, so it lags behind GRPO substantially at the start.

Li²SAO addresses this with group-relative actor-critic bootstrapping. While the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic. Once the critic is ready, training switches to single-rollout GAE. In the reported experiments, Li²SAO catches up with or surpasses GRPO in early training and raises the later performance ceiling.

The industrial weight of a cold critic

The deeper implication is that dependence on a closed-source value model is an operational constraint. In a self-hosted or on-premise setting, each external component adds integration work, a data exposure surface, and licensing or service costs. An algorithm that learns the value internally reduces that dependency and makes fine-tuning more reproducible and controllable.

There is also a structural shift. When the cold start of the critic dominates early phases, the incentive moves from designing the most elegant steady-state method to designing the transition. Teams running local infrastructure benefit from algorithms that do not assume auxiliary external models. Providers that built an edge around proprietary scaled value models lose part of that differentiation.

The code is available on GitHub. The larger signal is that the next front in asynchronous post-training may be less about raw model scale and more about reducing technical debt in the warm-up phase.