Table of Contents
Measure the energy use of on-premise AI at three levels: the GPUs, read from NVIDIA's power and energy counters; the whole server, read from its management controller or a metered power strip; and the facility, by multiplying by your site's PUE. Then divide by useful work, such as questions answered, over a full day including idle hours.
Key takeaways
- A GPU's rated power is a ceiling, not its consumption. The ML.ENERGY benchmark found estimates based on rated power "nearly always an overestimation", by up to 4.1 times in its worst case (Chung et al., 2025).
- GPU power readings have gaps: on A100 and H100 cards, nvidia-smi samples only about 25% of the runtime (Yang et al., 2023). Check your readings once against an external meter.
- The GPU is not the whole bill. Add processors, memory and fans at server level, then cooling and power losses through PUE, which averaged 1.54 in Uptime Institute's 2025 survey (Uptime Institute).
- Report energy per unit of useful work, such as watt-hours per question, measured over a whole day. Idle hours can be a large share of the total.
- Batch size, model choice and clock settings change energy per answer. Measure before and after each change.
Why measure the energy of an on-premise AI system?
When you run AI on your own servers, the electricity, the cooling and the backup power are yours. Three practical reasons to measure:
- Capacity. Server rooms, UPS units and, at remote sites, generators have fixed limits. You need real peak and average figures before adding a second GPU server.
- Cost. In our three-year cost comparison, electricity was about 2% of the total for a two-GPU server, using a pessimistic constant 1.5 kW. A measured figure replaces that assumption.
- Reporting. Energy and sustainability teams increasingly ask IT for consumption by system. A measured kWh figure is easier to defend than a nameplate estimate.
For the wider deployment picture, see our guide to sovereign, on-premise AI.
Why is the GPU's rated power the wrong number?
The maximum power on a GPU datasheet (often called TDP) is what the card may draw at most. It is the right number for sizing circuits, power supplies and UPS units, which must handle the peak. It is the wrong number for energy accounting.
The authors of the ML.ENERGY benchmark measured inference energy on 40 model architectures. They found that estimates using TDP are "nearly always an overestimation", because a GPU rarely draws its maximum power continuously, with a worst case of 4.1 times on H100 GPUs (Chung et al., 2025).
Language models make the gap larger. Generating text one token at a time is limited mainly by memory bandwidth, as our GPU sizing guide explains. On an NVIDIA H200, one 2026 study found that during this generation phase the GPU draws far less than its power cap, so the cap never activates (Ma et al., 2026).
Where can you measure power on a GPU server?
Each level includes more of the system and tells you less about any single workload.
| Level | Where to read it | What it covers | Watch out for |
|---|---|---|---|
| GPU | nvidia-smi, or NVIDIA DCGM with dcgm-exporter for Prometheus | The graphics card only | Sampling gaps; misses processors, memory, fans |
| Processor | Linux powercap (Intel RAPL) energy counters | CPU packages, split into core and uncore | Counter has a maximum range and wraps; vendor-specific |
| Server | Management controller (BMC) via IPMI or Redfish | Whole server at the power supply | Resolution and accuracy vary by vendor |
| Rack | Metered PDU or a clamp meter on the feed | Everything plugged in | No split between workloads |
| Facility | IT energy multiplied by PUE | Adds cooling and power distribution losses | PUE varies by season and by room |
GPU. nvidia-smi reports power.draw.average, "the average power draw for the entire board for the last second", on Ampere (except GA100) and newer GPUs, and power.draw.instant for the last reading (NVIDIA). A command such as nvidia-smi --query-gpu=index,power.draw.average --format=csv,noheader -l 1 logs it every second. DCGM also exposes a counter of "total energy consumption for the GPU in mJ since the driver was last reloaded" (NVIDIA DCGM). The difference between two readings of that counter gives the energy over a test.
Processor. On Linux, the powercap framework exposes energy_uj, a "current energy counter in micro joules", for each Intel CPU package and its core and uncore parts (Linux kernel).
Server. Most rack servers report total power through their management controller. With ipmitool, dcmi power reading gets "power related readings from the system" (ipmitool manual). Controllers that support Redfish can publish power in watts and energy in kWh in the standard EnvironmentMetrics resource (DMTF Redfish).
Facility. Uptime Institute describes PUE as total facility power divided by the power consumption of the IT equipment (Uptime Institute).
How accurate are GPU power readings?
Less accurate than most teams assume. Yang, Adamek and Armour tested more than 70 GPUs from every NVIDIA architecture since Fermi. On A100 and H100 cards, they found that nvidia-smi samples only 25% of the runtime for power, so the GPU can draw very different power during the remaining 75% without being seen. This can cause large over- or underestimates of energy. Following their recommended practices reduced the error against an external power meter by 35% on average, and by up to 65% in some tests (Yang et al., 2023).
Practical consequences:
- Run each test for several minutes, not a few seconds.
- Log averaged readings or the cumulative energy counter rather than occasional instant readings.
- Once per server model, compare the GPU and BMC figures with a clamp meter or metered PDU on the same load.
How do you run an energy test, step by step?
- Fix the boundary. Decide whether you report GPU, server or facility energy, and say so in every report.
- Record the idle baseline. Server on, model loaded, no requests, for at least 30 minutes. This is what the server costs while it waits.
- Replay a realistic load. Use real questions from your test set, at the concurrency you expect at peak. Our guide to evaluating an LLM on your documents explains how to build that set.
- Log power and work on the same clock. Collect GPU, CPU and BMC readings alongside the serving engine's counters. vLLM exposes
vllm:prompt_tokensandvllm:generation_tokenscounters andvllm:num_requests_runningon its/metricsendpoint (vLLM). The open-source Zeus library, used by the ML.ENERGY benchmark, can also measure energy from Python code. - Compute energy. Use counter differences, or average power multiplied by duration, for idle and loaded periods separately.
- Normalise. Divide by questions answered and by thousands of output tokens.
- Scale to the facility. Multiply by your room's PUE.
- Repeat after every change of model, precision, batch settings or software version, and keep the results with the answer-quality tests.
How do you turn watts into energy per question and dinars?
A worked example for the 300-user document assistant used in our cost comparison, with 20 questions per user per working day. The power figures are assumptions for illustration, not measurements.
| Item | Value | Basis |
|---|---|---|
| Idle power, 14 hours a day | 0.5 kW | Assumption; measure at the BMC |
| Average power in working hours, 10 hours a day | 1.3 kW | Assumption; measure at the BMC |
| IT energy per day | 7.0 + 13.0 = 20.0 kWh | Calculated |
| Facility energy per day at PUE 1.54 | 30.8 kWh | Calculated |
| Questions per day | 6,000 | 300 users × 20 questions |
| IT energy per question | about 3.3 Wh | 20,000 Wh ÷ 6,000 |
| Facility energy per question | about 5.1 Wh | 30,800 Wh ÷ 6,000 |
| Idle share of IT energy | 35% | 7.0 ÷ 20.0 |
To convert to cost, the CREG tariff sheet for high voltage type A gives a single-rate price (tariff 44) of 375.62 centimes per kWh, excluding taxes. At that rate, 30.8 kWh costs about DZD 116 per day, or about DZD 42,200 a year for 11,242 kWh. The sheet refers to a CREG decision of 29 December 2015, so confirm the rate on your current Sonelgaz invoice.
Two lessons follow. First, in this example a third of the energy is spent waiting, so energy per question falls as usage grows; a lightly used server looks expensive per question. Second, timing matters on a time-of-use tariff: under tariff 42 on the same sheet, the peak rate from 17:00 to 21:00 is 872.02 centimes per kWh against 180.64 centimes for the rest of the day. Batch work such as re-indexing documents can be scheduled outside the peak.
What changes the energy per answer?
Batch size. Serving engines process several requests together, which spreads the cost of each pass over more answers. In the ML.ENERGY measurements on H100 GPUs, energy per generation for Qwen 3 32B fell by a factor of about 5.8 when the maximum batch size rose from 4 to 64. Reasoning models with long outputs cannot run such large batches, which limits this saving (Chung et al., 2025). The same paper reports that its automated optimisation recommendations sometimes saved more than 40% of energy without changing what the model computes.
Model and task. Luccioni, Jernite and Strubell measured the energy of 1,000 inferences on an A100 GPU across many models and tasks. Text generation used about 0.047 kWh on average, against 0.002 kWh for text classification, and multi-purpose generative models were "orders of magnitude more expensive" than task-specific ones for several tasks (Luccioni et al., 2024). If a task only needs a label, a classifier or a small language model may do it for a fraction of the energy.
Power limits and clocks. On LLaMA 65B across four A100 GPUs, lowering the power cap from 250 W to 175 W increased inference time by 6.7% on average and reduced total energy by 23.21%; a 150 W cap increased time by 19.49% (Samsi et al., 2023). On an H200, a later study found power capping ineffective during text generation and reported that locking the GPU clock recovered up to 32% of generation energy with minimal throughput loss (Ma et al., 2026). The two results come from different hardware and settings, so test any limit on your own server and check response times.
Idle time. Consolidating light workloads on one server, or putting test environments to sleep outside working hours, reduces the idle share shown in the example above.
What should you report, and to whom?
| Audience | Figure | Why |
|---|---|---|
| Facilities and IT infrastructure | Peak kW per server and per rack | Checks against room, UPS and generator capacity |
| Energy manager | kWh per month at facility level | Fits the site's energy accounts |
| Product owner | Wh per question and per 1,000 output tokens | Compares models and settings |
| Management | Annual cost in DZD | Puts energy beside hardware and staff costs |
Always state the conditions: model, precision, batch settings, power or clock limits, software versions and the measurement boundary. For carbon reporting, apply the grid emission factor your company already uses for its other electricity.
What we have seen at KABAS AI
DALILE runs on the client's own servers, so the power, cooling and backup supply of the server room are part of planning each deployment. The figures in this article come from the public sources cited or are labelled assumptions; they are not measurements from client sites.
The practical guidance is to measure during the pilot, on the actual server, before ordering more hardware: an idle baseline, a realistic peak, and energy per question, recorded alongside the answer-quality tests so that a model change is judged on both. Ask bidders for power and cooling figures as well, as our on-premise AI procurement checklist suggests, and for sites running on generators see AI at remote sites. Our consulting team can set up this measurement with your IT and facilities teams.
Sources
- Chung et al., The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization, arXiv 2505.06371, 2025
- Yang, Adamek and Armour, Part-time Power Measurements: nvidia-smi's Lack of Attention, arXiv 2312.02741, 2023
- Uptime Institute Global Data Center Survey 2025 (PDF)
- NVIDIA: nvidia-smi documentation
- NVIDIA DCGM: field identifiers
- NVIDIA dcgm-exporter on GitHub
- Linux kernel documentation: power capping framework
- ipmitool manual page
- DMTF Redfish EnvironmentMetrics schema
- vLLM: production metrics
- Zeus on GitHub (ML.ENERGY)
- Luccioni, Jernite and Strubell, Power Hungry Processing: Watts Driving the Cost of AI Deployment?, ACM FAccT 2024 (arXiv 2311.16863)
- Samsi et al., From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference, arXiv 2310.03003, 2023
- Ma et al., The Illusion of Power Capping in LLM Decode, arXiv 2605.11999, 2026
- CREG: Tarification de l'électricité haute tension type A (PDF)
Frequently asked questions
How much electricity does an on-premise LLM server use?
There is no single figure: it depends on the GPUs, the model, the load and how many hours the server sits idle. The rated power of the GPUs is an upper bound, useful for sizing circuits and backup power, but measured consumption is usually lower. Measure your own server over a full day, including nights and weekends.
Is nvidia-smi accurate enough to measure GPU energy?
It is a good starting point but not a reference meter. A 2023 study of more than 70 GPUs found that on A100 and H100 cards only about 25% of the runtime is sampled for power, which can over- or underestimate energy. Run tests for several minutes, log averaged readings or the cumulative energy counter, and check the result once against an external meter.
What is PUE and which value should I use?
Power usage effectiveness is total facility power divided by the power of the IT equipment, so it adds cooling and power distribution losses to the server's own consumption. Use the value your facilities team measures for the room where the server sits. If you have none, the weighted average in Uptime Institute's 2025 survey was 1.54, but small server rooms can differ widely.
Does a smaller model use less energy per answer?
Usually, but measure it on your own workload. In a study of 1,000 inferences per task, text generation used about 0.047 kWh on average against about 0.002 kWh for text classification, and task-specific models were far cheaper than multi-purpose generative ones. Repeat the same energy test with each candidate model and compare watt-hours per question.
Should we run batch AI jobs at night to save money?
If your site is on a time-of-use tariff, it can help. On the CREG high-voltage tariff sheet, the peak rate between 17:00 and 21:00 under tariff 42 is 872.02 centimes per kWh against 180.64 centimes off-peak. Jobs such as re-indexing documents can often wait until off-peak hours. Confirm the tariff that applies to your site on your Sonelgaz invoice.
Is electricity a large part of the cost of on-premise AI?
Usually not. In our three-year cost model for a two-GPU server, electricity at the official high-voltage tariff was about 2% of the total, far behind hardware and staff. Power and cooling capacity in the server room, the UPS and the generator are more often the real constraint.



