Efficient LLM Inference On-Premise: Qwen 3.6 on Nvidia RTX A4000

The adoption of Large Language Models (LLM) in enterprise environments raises crucial questions related to data sovereignty, infrastructure control, and Total Cost of Ownership (TCO). In this context, the self-hosted approach emerges as a valid alternative to cloud solutions, especially when it comes to optimizing existing hardware. A recent use case demonstrated how significant performance for LLM inference can be achieved on an on-premise infrastructure, leveraging graphics cards that are not the latest generation.

The adopted configuration is heterogeneous, centered around a Lenovo ThinkStation P3 Tower Gen 2 server, originally intended for OpenShift/K8s clusters. The user progressively integrated four Nvidia RTX A4000 GPUs, each equipped with 16GB of VRAM. Although the RTX A4000s do not represent the cutting edge of technology, their energy efficiency (140W per card, later limited to 125W to optimize stability and performance) and the requirement of a single PCIe slot per unit make them suitable for servers with limited space, offering a concrete example of how legacy hardware can be leveraged for AI workloads.

Technical Details and Field Performance

For model execution, the implementation utilized Llama.cpp, an Open Source framework known for its efficiency in LLM inference across various hardware architectures. Crucial was the enablement of Multi-GPU Tensor Parallelism (MTP) with the --spec-draft-n-max 4 option, which allowed the workload to be distributed among the four GPUs. The operating system used is Fedora 43, with the necessary CUDA drivers for hardware acceleration.

The main model tested was Qwen 3.6 27B Q8, an 8-bit quantized variant of the Qwen 3.6 model with 27 billion parameters, in GGUF format. Recorded performance was approximately 45 tokens per second for reasoning tasks and about 60 tokens per second for coding tasks. These metrics were achieved while maintaining the full context and without applying KV cache quantization, indicating good workload management capability. The user also experimented with Qwen 3.6 35B A3B Q8 MoE (Mixture of Experts), which achieved approximately 80 tokens per second for reasoning and 90 tokens per second for coding, albeit with a --split-mode layer configuration instead of tensor.

Implications for On-Premise Deployment and TCO

This use case offers significant insights for companies considering on-premise LLM deployment. The choice of "older" hardware like the RTX A4000s, originally purchased for about $865 each (now valued between $1,300 and $1,500 on the used/new market), demonstrates how a judicious initial investment can translate into a favorable TCO in the long term. Optimizing energy consumption by limiting the cards to 125W further contributes to reducing operational costs.

The ability to reuse and optimize existing hardware for AI workloads is a key factor for organizations aiming to maintain control over their data and comply with stringent privacy regulations, such as GDPR, avoiding reliance on external cloud services. The user's experience, who felt "redemeed" after initially doubting the investment, underscores the importance of exploring local solutions and pushing the market to get the most out of available hardware, even older models.

Outlook and Trade-offs in the Local LLM Landscape

The experiment highlights the inherent trade-offs between performance, cost, and model quality. Although the Qwen 3.6 35B A3B Q8 MoE model showed higher throughput in terms of tokens per second, the user noted that the "dense" Qwen 3.6 27B tended to produce more accurate coding solutions on the first attempt. This suggests that pure inference speed is not the only parameter to consider; the quality and reliability of the model's responses are equally crucial, especially in business contexts.

For CTOs, DevOps leads, and infrastructure architects evaluating self-hosted vs. cloud alternatives for AI/LLM workloads, this example reinforces the idea that careful planning and optimization can unlock significant value from on-premise hardware. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate the trade-offs between different architectures and deployment strategies, helping to make informed decisions that balance performance, TCO, and data sovereignty requirements.