Harnessing the huge context window of transformers to make better decisions without touching the model’s weights: a dream that seemed within reach thanks to the resurgence of in-context reinforcement learning (ICRL). Decision-pretrained transformers, algorithm distillation, long-context meta-RL, retrieval-augmented agents — all avenues aimed at producing immediate, instantaneous learning from the stream of interactions alone. But the ground is never still, and that very non-stationarity risks turning context into a trap.

A newly circulated survey — "In-Context Reinforcement Learning under Non-Stationarity" — brings order to this still neglected field. It does not merely list pre-training objectives or architectures. It puts its finger on the wound: when the rules of the game change, accumulated memory is no longer an innocent resource. It can become obsolete, misleading, only to become useful again if the old regime returns. The model must decide what to discard, what to keep, and what to trust, all with frozen parameters.

Inference without a gym

At the heart of ICRL lies the idea of delegating adaptation to pure in-context computation, without fine-tuning sessions. The model, pre-trained or fine-tuned on distributions of tasks, observes rewards, transitions, demonstrations, feedback, and retrieves past experiences. From this crucible it extracts the latent rule of the current task and uses it to improve subsequent choices. No parameter updates: the policy remains fixed.

This approach opens concrete perspectives for those operating in environments where data cannot leave the company perimeter. An on-premise decision assistant could assimilate rounds of interactions with an operator without ever sending logs to the cloud, adapting to seasonal dynamics or regulatory changes simply by looking at recent history. Data sovereignty is enhanced, TCO lightened by forgoing costly retraining cycles.

But the survey warns: non-stationarity shatters this elegant construction. If the reward signal, the transition model, the observation channel, or even the action interface change, the past ceases to be a reliable teacher. And context, which in ICRL acts as the sole archive and oracle, can suggest lethal strategies.

The three questions nobody was asking

The authors organize the literature around three questions every on-premise implementation should pose: what changes in the environment, how the change unfolds, and how observable it is to the agent. Not all drift is equal: a sudden customer turnover requires different tools from a slow sensor degradation, and the ability to perceive the shift determines whether the model can recognize that the old advice has expired.

This has second-order consequences for infrastructure. Those pushing ever longer contexts, chasing million-token windows, risk exacerbating the problem: the more history you accumulate, the higher the computational cost and the harder it becomes to separate fresh signals from polluting residues. Adding more VRAM or increasing memory bandwidth is not enough; what’s needed is context intelligence — attention mechanisms capable of deprioritizing segments that are no longer trustworthy, architectures that partition the window into regimes, perhaps explicit retrieval-augmented memory.

For model and platform vendors, the survey marks a watershed. So far the market has rewarded whoever offered the longest context. From now on, what will also matter is who can handle the temporal quality of information. Legal and compliance teams at banks, insurers, and manufacturers, already leaning toward self-hosted setups for fear of data leakage, will find in these analyses an argument to demand not just larger models but models more aware of conceptual drift.

The direction is clear: non-stationary ICRL upends the perspective on lazy learning. Learning instantly from context is powerful, but only if the context knows how to forget.