A user built an inference server with 192 GB of VRAM across multiple cards, explicitly to run a single model: MiMo v2.5. This isn't hardware excess for its own sake, but a response to a gap that anyone working with on-premise LLMs knows: the shortage of truly competitive models between 30 and 400 billion parameters. The surprise, according to the account posted on r/LocalLLaMA, is that MiMo v2.5 fills that gap better than any other contender, and does so with a rawness that says a lot about the future of local inference: at times, tokens feel like they are “pouring out of the screen” because it's so fast.

Before diving in, it’s worth clarifying the context. Anyone running self-hosted workloads knows that models below 30 billion parameters often lack competence, while those above 400 billion require infrastructure out of reach for most organizations. MiMo v2.5 sits in the middle, with a size and architecture that allow it to run – with aggressive quantization levels – on the kind of servers that are starting to populate enterprise data centers. The developer’s test compared four quantization variants: the Bartowski IQ4_XS and IQ4_NL, the Unsloth UD-Q4_K_S, and a fourth “unfused” IQ4_XS on ik_llama. The verdict, without mincing words, gives the edge to IQ4_NL for the best balance of perceived quality and throughput.

But the experience isn't without rough edges. The looping problem – the model repeating chunks of reasoning forever until it reaches max tokens – is real and requires careful tuning of generation parameters. The user notes that settings such as --repeat-penalty 1.2 and --repeat-last-n 128, combined with a temperature of 1.0 and top-p of 0.95, work well. Warning: more aggressive penalties (repeat-penalty 1.5, presence-penalty 0.4, frequency-penalty 0.2) made the model “incapable of using tools or thinking clearly.” This is a far from theoretical caveat, because anyone designing on-premise inference pipelines must reckon with unpredictable behavior when you move the penalty sliders.

On the optimization front, not everything is ready. In llama.cpp – the backbone of many local servers – Multi-Token Prediction (MTP) doesn't work, nor does --split-mode tensor, while --split-mode graph behaves well only with specific ik_llama builds. Multimodal vision and automatic speech recognition (ASR) also aren't operational yet, though open issues exist on the repository, and the prospect of being able to converse with the model while capturing tone, inflection, and background noise would increase the value of on-premise deployment, which today still depends on external services like Qwen3 for ASR.

Behind this chronicle of experiments, however, one can read a structural signal that concerns anyone deciding where to run AI. MiMo v2.5, running faster than any cloud provider the user had access to, flips the narrative that high performance necessarily means abandoning local infrastructure. Those who invested in a multi-GPU server with 192 GB of VRAM find a model that not only fits in memory, but responds with latency that rivals (and exceeds) public APIs. It's a thermometer measuring how much the gap between on-premise and cloud is narrowing, at least for a specific class of models.

The implications for data sovereignty are immediate. If an organization can run an LLM of this size without sending prompts to external services, full control over its information perimeter becomes concrete. It's not just about privacy: it’s the ability to build inference and fine-tuning pipelines where every bit stays inside the company boundaries, with direct benefits for GDPR compliance and TCO management.

Of course, knots remain to be untied. The absence of MTP means you can't yet squeeze all the potential out of the hardware, and looping forces a more careful tuning than one would expect from a “ready to use” model. But the takeaway for those writing budget checks for AI infrastructure is different: today there is a model in that parametric middle ground that justifies the expense of a dedicated server, delivering performance that the user simply describes as “faster than any fast cloud provider.” Those who were rushing to migrate everything to the cloud might want to re-examine their calculations.