A card with a 300 W limit does not constantly draw 300 W. Waiting for a request, processing a long prompt and generating a response are different workloads. Understanding energy use therefore starts with measuring a particular task, not just reading the GPU specification.
The second distinction is equally important: GPU board power is not whole-server power. The CPU, memory, storage, fans and power supplies contribute to the rest of the system's energy use.
Our Qwen3-8B measurement
During two short Qwen3-8B Q4_K_M scenarios, GPU power readings clustered around 300 W. Before starting the model, we recorded a sample mean of 14.55 W; during a brief idle period immediately after loading, it was 68.55 W.
| Phase | Sample mean | Recorded range | Samples |
|---|---|---|---|
| No model process, 10 s | 14.55 W | 14.30–15.15 W | 5 |
| Model ready, 10 s without requests immediately after loading | 68.55 W | 66.60–74.30 W | 5 |
| 2048 input tokens, 1 concurrent request | 300.04 W | 299.93–300.24 W | 3 |
| 8192 input tokens, 4 concurrent requests | 299.97 W | 299.77–300.33 W | 10 |
These are GPU board readings within the stated windows. 68.55 W is not a steady-state idle figure: it covers ten seconds immediately after model loading. The short request samples show operation near the limit, but do not establish total task energy or monthly server consumption.
Hardware and method
| Component | Measurement configuration |
|---|---|
| GPU | RTX PRO 6000 Blackwell Max-Q Workstation Edition, 96 GB VRAM, 300 W limit |
| CPU | AMD EPYC 4565P, 16 cores / 32 threads |
| System RAM | 96 GB |
| Operating system and driver | Rocky Linux 10.2, NVIDIA 595.91.07 |
| Model and engine | Qwen3-8B Q4_K_M, llama.cpp b11146, F16 KV cache, Flash Attention |
Each scenario had a warmup and three measured rounds; every response generated 256 new tokens. The model ran on the GPU through a local API with four 8704-token slots and prompt caching disabled. Before each scenario, we waited for a GPU temperature no higher than 40°C. The power limit and driver were unchanged.
Telemetry was sampled roughly every two seconds. Request results include only samples within completed measured rounds, excluding warmup, cooldown and gaps. The operating system cache was warm after verifying the model file's checksum. JSON contains the settings and window boundaries; CSV summarises the readings. This is a separate resource study: the additional telemetry has overhead, so its timings are not directly compared with the earlier API throughput test.
What nvidia-smi reports
GPU power readings cover the entire card. NVIDIA distinguishes an instantaneous reading from an average over a defined window; field availability depends on the hardware. The power limit is a control boundary, not a constant draw. See the NVIDIA-SMI documentation.
When comparing measurements, record the field name, sampling interval and task boundaries. A maximum from samples taken every two seconds is the highest recorded reading, not a guarantee that every brief peak was captured.
Watts, kilowatt-hours and task cost
Watts describe power; kilowatt-hours describe energy consumed over time. For average power across the complete period:
Energy [kWh] = average power [W] × time [h] / 1000.
For a varying workload, add the energy from successive measurement intervals. Gaps between requests also matter when the application and GPU remain running. Multiplying energy by the applicable rate gives the energy cost.
To calculate whole-server consumption, measure the whole machine, ideally at its power input with a known measurement boundary. GPU telemetry alone cannot reveal CPU draw or power-supply losses. Identify a BMC sensor before using it too: it may describe one supply, its input or its output, rather than the same point as an energy meter.
How to compare two LLM workloads
Keep the model, quantisation, input and output lengths, and concurrent request count consistent. Then examine these together:
- total task duration;
- recorded GPU board power;
- completed useful responses or generated tokens;
- time spent waiting between tasks.
Lower power does not necessarily mean less energy per completed task if the task takes longer. Similarly, higher token throughput does not establish whether the answer is useful: model quality remains a separate criterion.
What to monitor in your own application
Separate model startup, ready-idle operation and request handling. Align power readings with time to first token, total response time and temperature. This shows whether a heavier workload produces the expected number of responses, rather than simply using more resources.
For a persistent API, consider a typical working day, not just the heaviest prompt. A short test can characterise the card's behaviour; the service's energy budget requires observing its actual traffic and idle periods.
Related reading: CPU measurements during GPU inference and API latency and concurrency tests.
View the server configuration or describe your model and workload pattern.