The developer community is debating the merits of “vibe coding,” the practice of letting an LLM write most of the code. Amid the criticism and the defenses, one fact stands out: major vendors continue to bet on small-sized models to support software development. The latest signal comes from Google, which celebrated record inference performance for Gemma 4 31B by running hackathons focused on this compact model.

Cloud numbers vs. local reality

In Google’s cloud, Gemma 4 31B achieves 1500 tokens per second. That is 50 to 100 times what one typically measures on local machines with similarly sized models. The gap is not surprising: provider datacenters rely on optimized interconnects, high-bandwidth memory, and cooling systems that squeeze every watt. Yet the difference is striking and touches a nerve for those managing on-premise deployments.

For an organization that has decided to keep data within its own perimeter, local inference is a matter of sovereignty and compliance. Models like Gemma 4 31B, with a modest memory footprint, are natural candidates to run on consumer GPUs or mid-range servers, if one accepts a trade-off in speed. The question is not whether small models work — Google’s bet confirms they do — but whether the performance penalty is acceptable for workflows that demand near-instant responses.

Why compact models keep winning support

The push toward smaller models is not new. Running an LLM locally eliminates network latency, recurring API costs, and the risks of sharing code with external services. With parameter counts in the tens of billions, effective coding assistance becomes feasible without managing GPU clusters, reducing TCO and simplifying infrastructure. Fine-tuning on proprietary knowledge becomes easier, and quantization to INT8 or FP16 can further lower VRAM requirements.

The case of Gemma 4 31B shows that cloud providers are not abandoning this segment; on the contrary, they promote it with hackathons. It is an implicit acknowledgment that giant models are not the only path for AI-assisted software engineering. Yet the speed demonstrated in the cloud sets a bar that is hard to ignore for those building local stacks.

Closing the gap: the on-premise perspective

For anyone evaluating on-premise deployment, the performance ratio between cloud and local speeds is a design parameter. Software optimizations — runtimes like vLLM, continuous batching techniques, dedicated kernels — can yield a 2–5× boost, but the 50–100× leap requires a hardware upgrade that only the most expensive enterprise GPUs can approach today. Unsurprisingly, the conversation is shifting toward hybrid architectures: local inference for less critical tasks and cloud bursting when latency is tolerable.

The momentum around compact models remains good news for the on-premise ecosystem. Each new model, each hackathon, each public benchmark adds data points to guide investment decisions. And if Google is betting on Gemma, it signals that research into efficient architectures is not a dead end but a concrete direction for the years ahead.

An observatory for mindful deployment

The Gemma 4 31B episode offers an additional takeaway: speed is not everything. In many industrial settings, the priorities are system predictability, codebase security, and regulatory compliance. A small, self-hosted model, open to audit and control, holds value beyond raw tokens per second. The challenge for on-premise deployment is to find the right balance among these factors, accepting that the pure-performance crown remains — for now — with the hyperscalers.