CDMS | Watts Behind the Weights: where the energy of AI training actually goes
Every AI model is built on electricity. According to the International Energy Agency, data centres consumed around 415 TWh in 2024, roughly 1.5% of global electricity use. By 2030, that figure is projected to more than double, to around 945 TWh, slightly more than Japan consumes today, with AI as the main driver of the increase (IEA, 2025).
In CDMS (Claim Denials Management Solution), AI is the core of the system: models predict which healthcare claims are likely to be denied, map how procedure codes interact, and estimate which interventions actually change the outcome.
As the project enters in its final stage, a second question sits next to what these models can explain: in terms of energy consumption, what it the cost to train them, and where exactly is that cost incurred?
Most studies answer the first half with a single number, the energy of a full training run. That number hides most of the story. Training a model on a supercomputer is a lifecycle, not an event: datasets and models are downloaded, staged, loaded into memory, processed on GPUs, checkpointed, validated and, at the end, shipped out. Each step stresses different hardware and draws power in a different way.
This is the gap addressed by “Watts Behind the Weights: Characterizing Energy Consumption of the AI Training Lifecycle in HPC”, a study by INESC TEC and University of Minho researchers, developed in the scope of CDMS and run on Deucalion, the EuroHPC supercomputer hosted in Portugal. Instead of measuring training as one block, the team attributed energy to each individual phase and event of the lifecycle. The paper, by Alexandre Fernandes, Alícia Oliveira, Beatriz Cepa, Cláudia Brito and Ricardo Macedo, will be presented at the SC26 Workshops in Chicago (15–20 November 2026). To the best of the authors' knowledge, it is the first study to quantify the energy of individual phases across the entire AI training lifecycle in an HPC setting, rather than execution alone.
So, where does the energy of AI training actually go, and which part of it can be avoided?
Training is not one block
This may sound abstract, so let's follow a training job from start to finish.
Before anything runs, the user prepares the setup. Datasets and models are downloaded from public repositories such as Hugging Face or Kaggle onto the supercomputer's shared storage. This happens on a login node, and its speed depends mostly on the network, not on the computational capabilities of the machine.
Then comes execution, on the compute nodes. In every training iteration, data is read from storage into memory, moved to the GPUs, and processed through a forward pass, a backward pass and a weight update. When using multiple GPUs, these must also synchronise. Periodically, the model is validated on held-out data and saved to disk as a checkpoint, so that training can resume if something fails.
Finally, in the post-execution stage, the trained model leaves the facility to be deployed elsewhere.
Image: Fig. 1 from the paper, typical lifecycle of AI training in HPC (Source: Fernandes et al., SC26 Workshops, © 2026 IEEE)
To measure each of these steps separately, the team built a custom monitoring tool that samples CPU and GPU energy consumption every 50 milliseconds, and instrumented the training code to map out each individual every phase. Matching both traces offline makes it possible to say how much energy each phase consumed, not just the run as a whole.
The execution stage was tested on two deliberately different workloads, using four NVIDIA A100 GPUs on Deucalion:
ResNet-50, an image classification model trained on ImageNet (167 GB), heavy on data reading (I/O-bound);
Qwen3-8B, a large language model trained on WikiText (644 MB), heavy on GPU computation (GPU-bound).
The preparation stage was tested separately, with 7 datasets (from 486 MB to 549 GB) and 5 models (from 25.6 million to 27 billion parameters). All of these are public benchmark models and datasets, chosen to represent typical resource profiles; none of the experiments used CDMS claims data.
Preparation: small per download, large at scale
Downloading data and models is rarely counted as part of training. The results show why it is easy to overlook, and why it should not be.
Both energy and time grow almost linearly with size. A larger model costs more to download because it is bigger, not because it is more complex. And because these steps are limited by the network, they take a long time while using little power.
In isolation and when compared to the actual execution phase, these numbers are negligible. The issue is repetition. On a shared supercomputer, the same popular datasets and models are downloaded again and again by hundreds of users, each paying the same cost for the same files. Energy that looks irrelevant per download becomes a recurring, avoidable cost at the scale of the infrastructure.
Execution: time is not energy
In both workloads, training dominates and validation adds only a small fraction. Beyond that, the two models behave very differently.
ResNet-50 spreads its energy across CPU and GPU: on average, 16% goes to the CPU and 84% to the GPUs, with the backward pass as the single largest consumer. Qwen3-8B concentrates almost everything on the GPUs, around 92%, in the forward and backward passes, the weight updates and the synchronisation between GPUs.
Image: Figs. 4 and 5 from the paper, per-phase energy and time for ResNet-50 and Qwen3-8B (Source: Fernandes et al., SC26 Workshops, © 2026 IEEE)
The more revealing result is in ResNet-50. Copying the dataset to the node and loading data account for about 39% and 7% of execution time, respectively, yet for a much smaller share of energy. Both steps run only on the CPU, which draws far less power than a GPU, while the GPUs sit idle. In Qwen3-8B, where nearly all time and power are on the GPU, time and energy follow each other closely.
In other words, two phases of similar length can have very different energy costs. Execution time alone is not a reliable proxy for energy consumption, and no single phase dominates across workloads. To know what to optimise, energy has to be measured phase by phase, not run by run.
Same optimisation with a different outcome
The study also tested two common practices: where the data lives, and whether the model is checkpointed. Neither has a universal answer.
Shared storage vs. local disk. Copying the dataset from the shared parallel file system (Lustre) to the node's local disk is a standard way to speed up training. For ResNet-50, it works: local storage cut 17 minutes and 88 kJ from data loading in training and 5 minutes and 4 kJ in validation, with fewer stalls waiting for data and GPUs kept busier. But the copy itself has a cost, and depending on dataset size and number of epochs, it can cancel out the gain. For Qwen3-8B, storage made almost no difference: WikiText is small enough to fit in GPU memory once read, and computation remains the bottleneck either way.
Checkpointing on vs. off. Intuition says that saving more data costs more energy. The results are more nuanced. For ResNet-50, checkpointing to shared storage noticeably increased time and energy, while checkpointing to local disk had minimal impact. Qwen3-8B saves a much larger checkpoint, with billions of parameters and optimiser state, yet the impact was minimal, because the GPU work around it is so heavy that the checkpoint barely registers.
The cost of an optimisation depends less on its size than on how it compares to the workload's dominant phase. What helps an I/O-heavy model can be irrelevant for a GPU-heavy one.
Post-execution: the cost nobody reports
The last stage, uploading the trained model out of the facility, behaves like the model download: network-bound and running on the CPU. The study treats it as equivalent to the download rather than measuring it separately, but the point stands: because it happens after training ends, it is usually left out of reported energy figures altogether.
From measurement to better decisions
The energy of AI training is not a single number. It depends on the model, the data, where that data is stored and how the pipeline is configured. No single phase or optimisation wins across every workload, and even minor steps such as downloads add up to a real, avoidable cost on shared infrastructure.
One caveat matters for reading the numbers: the measurements are software-based and cover CPU and GPU only, so they underestimate the total energy of the system. What they preserve are the proportions and trends between phases, which is exactly what is needed to decide where to act.
The next steps of the study extend the analysis to more models and parallelisation strategies, other HPC platforms, distributed training across nodes and, eventually, model serving, the stage where trained models are used in production.
Read the full paper (SC26 Workshops):
Comments