Optimizing LLM Performance on Local Hardware: The Role of Micro-Batching

Large Language Model (LLM) inference on local hardware presents an ongoing challenge for DevOps specialists and infrastructure architects. The pursuit of configurations that maximize throughput and minimize latency is critical, especially when managing workloads requiring data sovereignty or air-gapped environments. In this context, optimizing execution parameters can uncover significant improvement margins, transforming the efficiency of existing systems.

A recent informal study conducted with llama.cpp has highlighted an interesting opportunity to accelerate prompt processing, a critical aspect for many LLM-based applications. The discovery focuses on the impact of the ubatch parameter (physical micro-batch size) on prefill speed, demonstrating how its correct calibration can unlock unexpected performance on consumer GPUs.

Technical Details and the Throughput Discovery

The analysis was performed on a gpt-oss-120b-F16.gguf model using an RTX 3090 GPU with 24 GB of VRAM, a common setup for those experimenting with self-hosted LLMs. Tests, conducted with llama-bench, explored various configurations of ubatch and n-cpu-moe, the latter parameter determining how many Mixture-of-Experts (MoE) layers are offloaded to the CPU to free up VRAM.

The results were remarkable: by increasing ubatch from the default value of 512 to 8192, prompt prefill speed rose from approximately 380 tokens per second to over 2090 tokens per second. This represents an increase of about 5.5 times. This improvement was achieved by moving a slightly larger number of MoE layers to the CPU (from 26 to 28), a trade-off that allowed managing the increased computational workspace required by the larger micro-batch. It is important to note that, despite this gain in prefill, token generation speed experienced a slight drop, going from approximately 32.3 to 30.1 tokens per second, a reduction of about 7%.

Implications for On-Premise Deployments and TCO

This discovery has direct implications for organizations considering or managing on-premise LLM deployments. The ability to achieve such significant improvements in prefill performance using existing hardware, such as an RTX 3090, can greatly influence the overall Total Cost of Ownership (TCO). Optimizing the use of available hardware resources means extending their useful life and delaying, or even avoiding, investments in more expensive new infrastructure.

For CTOs, DevOps leads, and infrastructure architects, calibrating parameters like ubatch and n-cpu-moe becomes a strategic lever to balance performance needs with budget constraints and data sovereignty policies. While cloud solutions offer immediate scalability, self-hosted deployments require careful resource management and deep optimization to compete in terms of efficiency and operational costs. AI-RADAR provides analytical frameworks on /llm-onpremise to evaluate these trade-offs, supporting informed decisions on on-premise deployments.

Future Prospects and Final Considerations

The experiment, though informal, underscores the importance of precise tuning of runtime parameters for Large Language Models, especially in environments with limited or specific resources. The choice between prioritizing prefill speed (essential for applications processing long inputs) or generation speed (crucial for rapid responses) depends on the specific workload and application goals.

This type of optimization demonstrates how innovation in software and configuration can "close the gap" in performance with higher-end hardware, such as a DGX Spark, even if not fully matching it in all scenarios. The continuous evolution of frameworks like llama.cpp and the community of developers exploring these optimizations are fundamental to making LLM inference increasingly accessible and efficient in on-premise contexts, offering concrete alternatives to cloud-based solutions.