The Energy of Local Large Language Models

The interest in deploying Large Language Models (LLMs) in on-premise environments continues to grow, driven by the need for data sovereignty, cost control, and reduced latency. However, managing these infrastructures involves significant practical considerations, including power consumption, a factor often underestimated during the planning phase. A recent contribution from the "LocalLLaMA" community has provided concrete and relevant data for those evaluating these solutions.

According to the report, a system equipped with two NVIDIA GeForce RTX 3090 GPUs, configured for LLM inference, draws approximately 760W of power measured directly at the wall during operation under load. This value decreases to about 90W in an idle state, highlighting a significant difference between the two operational scenarios. The user specified that no power limit tweaks were applied to the cards, suggesting that the data reflects an "out-of-the-box" consumption for this configuration.

Technical and Operational Implications

NVIDIA GeForce RTX 3090 GPUs, with their 24GB of VRAM each, represent a popular choice for medium-sized LLM inference in local contexts, offering a good balance between memory capacity and cost compared to enterprise solutions. However, the 760W consumption for a dual-GPU setup is not negligible. This data has direct implications for the choice of power supply unit (PSU), which must be adequately sized to support peak loads, and for the cooling system of the infrastructure.

Such high power draw generates a significant amount of heat, requiring efficient dissipation solutions to maintain operating temperatures within acceptable limits and ensure long-term system stability. For data centers or server rooms, this translates into increased requirements for air conditioning and ventilation systems, directly impacting operational costs and the overall energy efficiency of the infrastructure.

TCO and Cloud Comparison

Power consumption is a fundamental component in calculating the Total Cost of Ownership (TCO) for on-premise deployments. While the initial investment (CapEx) for hardware can be high, operational costs (OpEx) related to electricity and cooling represent an ongoing expense that must be carefully monitored. A consumption of 760W, while not extreme for an AI workload, translates into an annual energy cost that varies significantly based on local electricity prices.

This contrasts with the cloud model, where costs are often based on actual consumption (pay-per-use) and implicitly include energy and cooling. The choice between on-premise and cloud for LLM workloads is never trivial and requires a thorough analysis of the trade-offs between control, data sovereignty, and TCO. For those evaluating these decisions, AI-RADAR offers analytical frameworks on /llm-onpremise to compare different approaches and their constraints.

Prospects for Local AI Infrastructure

The data on the power consumption of a dual RTX 3090 setup underscores the importance of meticulous planning for any self-hosted AI infrastructure. It's not just about acquiring hardware, but about considering the entire ecosystem: power, cooling, physical space, and noise management. With the evolution of Large Language Models and the growing demand for inference capabilities, energy efficiency will become an increasingly critical factor.

Future innovations in silicio, Quantization techniques, and software Framework optimization will aim to reduce power consumption per token, making on-premise deployments even more sustainable. For companies prioritizing data sovereignty and complete control over their technology stack, understanding and managing these operational constraints is essential for the long-term success of their AI strategies.