Energy Efficiency in HPC Data Centers: Latest Advances to Build the Path to Exascale
Nowadays, moderating energy consumption and building eco-friendly computing infrastructure is a major goal in large data centers. Moreover, data center energy usage has risen dramatically over the past decade and will continue to grow in-step with the High Performance Computing (HPC) intensive workloads which are at the heart of our modern life. The recent advances in the technology has driven the data center into a new phase of expansion featuring solutions with higher density. To this end, much has been done to increase server efficiency and IT space utilization. In this chapter, we will provide a state-of-the-art overview as regards energy-efficiency in High Performance Computing (HPC) facilities while describing the open challenges the research community has to face in the coming years to enable the building and usage of an Exascale platform by 2020.
KeywordsCloud Computing High Performance Computing Computing Node High Performance Computing System Power Usage Effectiveness
The research presented in this paper is partially funded by a grant from Polish National Science Center under award number 2013/08/A/ST6/00296.
The experiments presented in this paper were carried out using the HPC facility of the University of Luxembourg and Poznan Supercomputing and Networking Center.
- 2.AMD Unveils Server Strategy and Roadmap. http:/www.amd.com/us/press-releases/Pages/amd-unveils-2013june18.aspx.
- 3.DOE Extreme-Scale Technology Acceleration FastForward. https://asc.llnl.gov/fastforward/.
- 4.European Mont-Blanc Project. http://www.montblanc-project.eu/.
- 5.Grid'5000. [online] http://grid5000.fr.
- 6.Iceotope Servers. http://www.iceotope.com/.
- 7.Intel Atom Processor N2600. http://ark.intel.com/products/58916/intel-atom-processor-n2600-(1m-cache-1_6-ghz).
- 8.Intel Core i7-3615QE. http://ark.intel.com/products/65709/Intel-Core-i7-3615QE-Processor-(6M-Cache-up-to-3_30-GHz).
- 9.Intel unveils new technologies for efficient cloud datacenters. http://newsroom.intel.com/community/intel_newsroom/blog/2013/09/04/intel-unveils-new-technologies-for-efficient-cloud-datacenters.
- 10.IOR HPC benchmark. [online] http://sourceforge.net/projects/ior-sio/.
- 11.Iozone filesystem benchmark. [online] http://www.iozone.org/.
- 12.SuperMUC - First Commercial IBM Hot-Water Cooled Supercomputer. http://www-03.ibm.com/press/us/en/pressrelease/38065.wss.
- 13.The Green500 List - November 2013. http://green500.org/lists/green201311.
- 14.Top500. [online] http://www.top500.org.
- 15.X-Stack Software. http://www.xstack.org/.
- 16.PUE (tm): A comprehensive examination of the metric. White paper, The Green Grid, 2012.Google Scholar
- 17.Q. Ali, V. Kiriansky, J. Simons, and P. Zaroo. Performance evaluation of HPC benchmarks on VMware’s ESXi server. In Proceedings of the 2011 international conference on Parallel Processing, Euro-Par'11, pages 213–222, Berlin, Heidelberg, 2012. Springer-Verlag.Google Scholar
- 18.P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP '03, pages 164–177, New York, NY, USA, 2003. ACM.Google Scholar
- 19.R. Bolze, F. Cappello, E. Caron, M. Daydé, F. Desprez, E. Jeannot, Y. Jégou, S. Lanteri, J. Leduc, N. Melab, G. Mornet, R. Namyst, P. Primet, B. Quetier, O. Richard, E.-G. Talbi, and I. Touche. Grid'5000: A large scale and highly reconfigurable experimental grid testbed. Int. J. High Perform. Comput. Appl., 20(4):481–494, Nov. 2006.CrossRefGoogle Scholar
- 20.N. Capit and al. A batch scheduler with high level components. In Cluster computing and Grid 2005 (CCGrid05), 2005.Google Scholar
- 21.S. J. Chapin, W. Cirne, D. G. Feitelson, J. P. Jones, S. T. Leutenegger, U. Schwiegelshohn, W. Smith, and D. Talby. Benchmarks and standards for the evaluation of parallel job schedulers. In D. G. Feitelson and L. Rudolph, editors, JSSPP, pages 67–90. 1999. Lect. Notes Comput. Sci. vol. 1659.Google Scholar
- 22.D. T. D. G. Feitelson and D. Krakov. Experience with the parallel workloads archive. Technical report, School of Computer Science and Engineering, The Hebrew University of Jerusalem, 2012.Google Scholar
- 23.G. Da-Costa, J.-P. Gelas, Y. Georgiou, L. Lefèvre, A.-C. Orgerie, J.-M. Pierson, O. Richard, and K. Sharma. The green-net framework: Energy efficiency in large scale distributed systems. In HPPAC 2009, 2009.Google Scholar
- 24.J. Emeras. Workload Traces Analysis and Replay in Large Scale Distributed Systems. PhD thesis, LIG, Grenoble - France, To be defended October 1st 2013. currently available at: https://forge.imag.fr/docman/view.php/359/754/thesis_emeras_28aug13.pdf.
- 25.M. Flynn. Some computer organizations and their effectiveness. IEEE Transactions on Computers, C(21):948–960, 1972.Google Scholar
- 26.Y. Georgiou. Contributions for Resource and Job Management in High Performance Computing. PhD thesis, LIG, Grenoble - France, Sep 2010.Google Scholar
- 27.M. Guzek, S. Varrette, V. Plugaru, J. E. Sanchez, and P. Bouvry. A Holistic Model of the Performance and the Energy-Efficiency of Hypervisors in an HPC Environment. In Proc. of the Intl. Conf. on Energy Efficiency in Large Scale Distributed Systems (EE-LSDS'13), volume 8046 of LNCS, Vienna, Austria, Apr 2013. Springer Verlag.Google Scholar
- 28.R. Januszewski, N. Meyer, and J. Nowicka. Evaluation of the impact of direct warm-water cooling of the HPC servers on the data center ecosystem. In To appear in International Supercomputing Conference 2014, Leipzig, Germany, 2014.Google Scholar
- 29.M. Jarus, S. Varette, A. Oleksiak, and P. Bouvry. Performance Evaluation and Energy Efficiency of High-Density HPC Platforms Based on Intel, AMD and ARM Processors. In Energy Efficiency in Large Scale Distributed Systems, Lecture Notes in Computer Science, pages 182–200. Springer Berlin Heidelberg, 2013.Google Scholar
- 30.A. Kivity and al. kvm: the Linux virtual machine monitor. In Ottawa Linux Symposium, pages 225–230, July 2007.Google Scholar
- 31.V. Kundra. Federal data center consolidation initiative. Memorandum for chief information officers, Office of Management and Budget of the USA, 2010.Google Scholar
- 32.E. Le Seur and G. Heiser. Dynamic voltage and frequency scaling: the laws of diminishing returns. In HotPower'10 Proceedings of the 2010 international conference on Power aware computing and systems, California, USA, 2010. USENIX Association Berkeley.Google Scholar
- 33.R. C. Murphy, K. B. Wheeler, B. W. Barrett, and J. A. Ang. Introducing the graph 500. In Cray User Group, 2010.Google Scholar
- 34.A.-C. Orgerie, L. Lefèvre, and J.-P. Gelas. Save watts in your grid: Green strategies for energy-aware framework in large scale distributed systems. In 14th IEEE International Conference on Parallel and Distributed Systems (ICPADS), Melbourne, Australia, Dec. 2008.Google Scholar
- 35.A. Petitet, C. Whaley, J. Dongarra, A. Cleary, and P. Luszczek. HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers.Google Scholar
- 36.N. Rasmussen. Air Distribution Architecture Options for Mission Critical Facilities Whitepaper #55. Technical report, American Power Conversion, 2003.Google Scholar
- 37.S. Sharma, C.-H. Hsu, and W. chun Feng. Making a case for a Green500 list. In Parallel and Distributed Processing Symposium, 2006. IPDPS 2006. 20th International, pages 8 pp.–, 2006.Google Scholar
- 38.S. Varrette, M. Guzek, V. Plugaru, X. Besseron, and P. Bouvry. HPC Performance and Energy-Efficiency of Xen, KVM and VMware Hypervisors. In Proc. of the 25th Symposium on Computer Architecture and High Performance Computing (SBAC-PAD 2013), Porto de Galinhas, Brazil, Oct. 2013. IEEE Computer Society.Google Scholar
- 39.M. vor dem Berge, J. Buchholz, L. Cupertino, G. Da Costa, A. Donoghue, G. Gallizo, M. Jarus, L. Lopez, A. Oleksiak, E. Pages, W. Piatek, J.-M. Pierson, T. Piontek, D. Rathgeb, J. Salom, L. Siso, E. Volk, W. U., and T. Zilio. CoolEmAll: Models and Tools for Planning and Operating Energy Efficient Data Centres. To appear in: Samee Khan, Albert Zomaya (eds.) Handbook on Data Centers.Google Scholar
- 40.M. vor dem Berge, G. Da Costa, A. Kopecki, A. Oleksiak, J.-M. Pierson, T. Piontek, E. Volk, and S. Wesner. Modeling and Simulation of Data Center Energy-Efficiency in CoolEmAll. Energy Efficient Data Centers. Lecture Notes in Computer Science, 7396:25–36, 2012.Google Scholar