Accumulating memory isn't learning. True learning distills transferable knowledge. This human insight has reached Microsoft Research with EvoLib, a framework that transforms how LLMs leverage the past. Instead of a static archive of reasoning traces, EvoLib builds a library of skills and insights that continuously evolve, extracted from raw experience and refined over time—all without touching model weights.

The core of the system relies on two mechanisms: consolidation and dynamic weighting. Each new experience is distilled into a knowledge unit—a reusable strategy or a lesson from a mistake—and compared to existing library entries. When similarity surfaces, the units merge into a more general form, shedding their tie to a single task and becoming applicable across domains. Meanwhile, a weighting mechanism re-evaluates each unit's importance not just by immediate utility but by how much it contributes to generating useful knowledge for future tasks. This yields a self-organizing ecosystem of competencies where long-term impact dictates prominence.

Tests are clear: on benchmarks spanning mathematical reasoning, efficiency-constrained code generation, and long-horizon decision tasks, EvoLib consistently outperforms top retrieval-based memory systems and other abstract mechanisms, using fewer tokens. More tellingly, the framework converts test-time compute into performance gains far more effectively than methods that scale compute per task in isolation. It does so with robustness to random task order—crucial for real-world streams of heterogeneous, interleaved requests.

For teams evaluating on-premise infrastructure and TCO, EvoLib carries a disruptive message. It shifts the value axis from expensive, data-hungry training to continuous inference. A deployed model—on-prem or behind APIs—can keep improving simply by accumulating experience, without any centralized fine-tuning or external data transfers. Consider a legal assistant processing sensitive documents: with EvoLib, winning strategies and corrected errors stay confined to the client's environment, compliant with data sovereignty rules, and become reusable assets for future users—nothing leaves the premises. Knowledge evolves locally, the model stays frozen, and re-training costs drop to zero.

This hints at a potential reconfiguration of incentives. Cloud model vendors pushing fine-tuning services may see part of their revenue base erode, while organizations running self-hosted deployments gain a tool to amortize hardware investment over a longer horizon. Moreover, EvoLib's ability to extract and refine knowledge from heterogeneous experience streams opens edge and air-gapped scenarios—factories, offshore platforms, military sites—where models must adapt to shifting conditions without external connectivity. The vision is not distant: the code is on GitHub, and the approach works with any black-box LLM, sliding neatly into established on-premise stacks.