When Danmoreng started experimenting with his own inference engine on an RTX 5080 16GB laptop, llama.cpp was the benchmark to beat. A few months ago, around the time Ninfer came out, it was fairly easy to get something running faster than llama.cpp: llama.cpp was considered slow compared to systems like vLLM, but vLLM wasted a lot of VRAM. Today the scenario has changed. A new benchmark by the same author shows that llama.cpp has caught up and even surpassed his "vibe-coded" engine.

The test makes no scientific claims: fixed 16k token prefill and 1k token output over Wikipedia text, with optimal settings for each engine. For the 12B model the checkpoint is identical, while the 26B uses different quantization levels across engines, so the comparison is not entirely apples-to-apples. Yet, despite this disparity, llama.cpp wins. The author welcomes this: he prefers using a stable engine with proper features over a series of vibe-coded projects, including his own.

The story goes beyond personal anecdote. It signals a structural shift in the landscape of self-hosted LLM runtimes. For months, anyone wanting to run models locally on consumer GPUs faced a trade-off: vLLM offered high throughput but used VRAM inefficiently, making it ill-suited for 16GB hardware; llama.cpp was more frugal but often too slow for sustained workloads. Incremental improvements in llama.cpp are narrowing this gap, making it less attractive to invest time in custom engine development.

There is a second-order implication: if a mature open-source framework like llama.cpp offers competitive performance, developers can focus on application logic instead of runtime optimization. This reduces fragmentation and accelerates adoption of stable self-hosted solutions. For those evaluating on-premise deployment, the TCO calculation changes: fewer development hours maintaining a custom engine, less need for hardware with abundant VRAM, and the possibility of using consumer GPUs for 12B or 26B models with adequate quantization.

Moreover, the author's stated preference for "stable engines with proper features" reflects a broader industry trend: after the exploratory phase of experimental runtimes, the community is converging on tools that are maintained, documented, and updated regularly. It is not just about raw performance, but about long-term reliability and security for those managing sensitive data locally.

The signal for the on-premise inference market is twofold. On one hand, vLLM remains a reference for data center environments with high-memory GPUs and parallelized workloads; on the other, llama.cpp is becoming the natural choice for edge, laptops, and workstations with VRAM constraints. Data sovereignty comes out stronger: quantized models runnable locally without cloud dependencies become more practical when the runtime is no longer the bottleneck.

Danmoreng's benchmark is imperfect, but the direction is clear. Incremental gains from a mature open-source project are worth more than sporadic optimizations from homemade engines. For those watching local LLM deployment, this is a sign that it pays to invest in solid foundations instead of chasing every new performance shortcut.