Monitoring and Analysis of Power Consumption on HPC Clusters using XDMoD

As part of the NSF funded XMS project we are developing tools and techniques for the audit and analysis of HPC infrastructure. This includes a suite of tools for the analysis of HPC jobs based on performance metrics collected from compute nodes. Although it may not be salient to the user, the energy consumption of an HPC system is an important part of the cost of maintenance and contributes a substantial fraction of the cost of calculations done with the system. We added support for energy usage analysis to the open-source XDMoD tool chain. This allows HPC centers to provide information directly to HPC stakeholders about the power consumption. This includes providing end users with energy usage information about their jobs as well as providing data to allow HPC center staff to analyze how the energy usage of the system is related to other system parameters. We explain how energy metrics were added to XDMoD and describe the issues we overcame in instrumenting a 1400 node academic HPC cluster. We present an analysis of 14 months of data collected on real jobs on the cluster. We performed a machine learning analysis of the data and show how energy usage is related to other system performance metrics.