The landscape of Large Language Models (LLMs) continues to evolve rapidly, with increasing focus on solutions that balance performance, cost, and data control. In this context, a user's experience with LiquidAI's new LFM2.5-24-A2B model offers interesting insights, particularly for those evaluating on-premise deployments.

The model, featuring 24 billion parameters, was tested on an AMD W7900 GPU, leveraging llama.cpp and ROCM acceleration. Preliminary results indicate notable performance, with a sustained speed of approximately 95 tokens per second (tps). This figure is significant, especially considering that the model, in its 8-bit quantized (Q8) version, requires 30GB of VRAM to handle a large context window of 128,000 tokens. The ability to maintain such an extensive context on local hardware, with VRAM requirements manageable for professional or high-end consumer cards, represents a competitive advantage for scenarios where data sovereignty and latency are critical.

The user compared LFM2.5-24-A2B with the previous Ornith 1.5-35B, finding that the new model performs exceptionally well, proving to be a reliable "workhorse" for mechanical and recursive tasks, even long-running ones. While analysis for tasks requiring deeper "insight" is still ongoing, its current speed and stability make it an ideal candidate to run alongside larger or more complex models, delegating routine operations and freeing up valuable resources.

Implications for the On-Premise Ecosystem

This experience underscores a structural trend in the industry: the increasing feasibility of running performant LLMs on local infrastructures, even outside the NVIDIA ecosystem. Compatibility with ROCM and llama.cpp on AMD W7900 hardware is not just a technical note, but a strong signal for companies looking to diversify their silicon suppliers and reduce reliance on a single vendor. For CTOs and infrastructure architects, this means more options in designing local AI stacks, with potential benefits in terms of Total Cost of Ownership (TCO) and flexibility.

The ability to manage a 24B model with a 128k context on 30GB of VRAM opens up interesting scenarios for applications requiring the processing of large volumes of text or code in air-gapped environments or those with stringent compliance requirements. This is not merely about saving cloud costs, but about gaining granular control over the entire inference pipeline, from hardware selection to software management. This approach fosters internal innovation and the creation of customized AI solutions, keeping sensitive data within the corporate perimeter.

The emergence of models like LFM2.5-24-A2B, optimized for local inference on diverse hardware, indicates that the LLM market is maturing beyond the initial phase dominated by a few players and monolithic infrastructures. The competence of these "smaller" models for specific tasks, combined with their efficiency on alternative platforms, drives towards hybrid and distributed architectures, where value lies not just in model size, but in its ability to integrate effectively into an existing IT ecosystem.