Reducing the cost of LLM inference is today’s central challenge for anyone running models in production. Parisian startup ZML has just launched ZML/LLMD, a free software that promises to accelerate inference across a wide range of AI chips, from GPUs to new accelerators. Backed by Turing Award winner Yann LeCun, the company aims to make the runtime layer a commodity untethered from specific hardware: a simple idea with potentially disruptive consequences.

Until now, optimizing inference often meant locking into a proprietary ecosystem. NVIDIA dominates with CUDA, and anyone wanting to squeeze every token per second out of an LLM had to invest in the company’s cards. But the landscape is changing fast: accelerators like AMD Instinct, Intel Gaudi, and new specialized ASICs promise lower cost per teraflop – provided you can use them without rewriting the entire pipeline. That’s where tools like LLMD come in.

ZML’s software, available for free, doesn’t just support multiple chips: its stated goal is to unify model execution while hiding hardware differences, so an organization can mix what it already owns or buy the best price/performance without vendor lock-in. For teams evaluating on-premise deployments, this could mean a sharply lower Total Cost of Ownership (TCO), because they are no longer forced to purchase expensive homogeneous clusters. The ability to run LLMs on heterogeneous machines, perhaps leveraging GPUs from different generations or even CPUs for light workloads, rewrites the economic calculus.

Behind the news there is also a question of technological sovereignty. In many regulated scenarios (GDPR, healthcare data, defense), models must run locally, without going through the cloud. Making inference economically viable on commodity and mixed hardware lowers the barrier for those who want or need to keep data in-house. ZML, with the backing of a figure like LeCun, signals that the future of inference may not be a monolith, but a federation of silicon.

Of course, real-world performance will need to be assessed when independent benchmarks become available: the software is young and the promise of cost reduction must be quantified. But the signal is strong: the race to optimize inference is expanding from models to hardware-agnostic infrastructure. And for Italian enterprises that are eyeing LLMs but remain held back by the compute bill, initiatives like this open a window worth following.