Energy Efficiency in HPC Data Centers: Latest Advances to Build the Path to Exascale

  • Sébastien Varrette
  • Pascal Bouvry
  • Mateusz Jarus
  • Ariel Oleksiak


Nowadays, moderating energy consumption and building eco-friendly computing infrastructure is a major goal in large data centers. Moreover, data center energy usage has risen dramatically over the past decade and will continue to grow in-step with the High Performance Computing (HPC) intensive workloads which are at the heart of our modern life. The recent advances in the technology has driven the data center into a new phase of expansion featuring solutions with higher density. To this end, much has been done to increase server efficiency and IT space utilization. In this chapter, we will provide a state-of-the-art overview as regards energy-efficiency in High Performance Computing (HPC) facilities while describing the open challenges the research community has to face in the coming years to enable the building and usage of an Exascale platform by 2020.


Cloud Computing High Performance Computing Computing Node High Performance Computing System Power Usage Effectiveness 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The research presented in this paper is partially funded by a grant from Polish National Science Center under award number 2013/08/A/ST6/00296.

The experiments presented in this paper were carried out using the HPC facility of the University of Luxembourg and Poznan Supercomputing and Networking Center.


  1. 1.
  2. 2.
    AMD Unveils Server Strategy and Roadmap. http:/
  3. 3.
    DOE Extreme-Scale Technology Acceleration FastForward.
  4. 4.
    European Mont-Blanc Project.
  5. 5.
    Grid'5000. [online]
  6. 6.
    Iceotope Servers.
  7. 7.
    Intel Atom Processor N2600.
  8. 8.
    Intel Core i7-3615QE.
  9. 9.
  10. 10.
    IOR HPC benchmark. [online]
  11. 11.
    Iozone filesystem benchmark. [online]
  12. 12.
    SuperMUC - First Commercial IBM Hot-Water Cooled Supercomputer.
  13. 13.
    The Green500 List - November 2013.
  14. 14.
    Top500. [online]
  15. 15.
    X-Stack Software.
  16. 16.
    PUE (tm): A comprehensive examination of the metric. White paper, The Green Grid, 2012.Google Scholar
  17. 17.
    Q. Ali, V. Kiriansky, J. Simons, and P. Zaroo. Performance evaluation of HPC benchmarks on VMware’s ESXi server. In Proceedings of the 2011 international conference on Parallel Processing, Euro-Par'11, pages 213–222, Berlin, Heidelberg, 2012. Springer-Verlag.Google Scholar
  18. 18.
    P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP '03, pages 164–177, New York, NY, USA, 2003. ACM.Google Scholar
  19. 19.
    R. Bolze, F. Cappello, E. Caron, M. Daydé, F. Desprez, E. Jeannot, Y. Jégou, S. Lanteri, J. Leduc, N. Melab, G. Mornet, R. Namyst, P. Primet, B. Quetier, O. Richard, E.-G. Talbi, and I. Touche. Grid'5000: A large scale and highly reconfigurable experimental grid testbed. Int. J. High Perform. Comput. Appl., 20(4):481–494, Nov. 2006.CrossRefGoogle Scholar
  20. 20.
    N. Capit and al. A batch scheduler with high level components. In Cluster computing and Grid 2005 (CCGrid05), 2005.Google Scholar
  21. 21.
    S. J. Chapin, W. Cirne, D. G. Feitelson, J. P. Jones, S. T. Leutenegger, U. Schwiegelshohn, W. Smith, and D. Talby. Benchmarks and standards for the evaluation of parallel job schedulers. In D. G. Feitelson and L. Rudolph, editors, JSSPP, pages 67–90. 1999. Lect. Notes Comput. Sci. vol. 1659.Google Scholar
  22. 22.
    D. T. D. G. Feitelson and D. Krakov. Experience with the parallel workloads archive. Technical report, School of Computer Science and Engineering, The Hebrew University of Jerusalem, 2012.Google Scholar
  23. 23.
    G. Da-Costa, J.-P. Gelas, Y. Georgiou, L. Lefèvre, A.-C. Orgerie, J.-M. Pierson, O. Richard, and K. Sharma. The green-net framework: Energy efficiency in large scale distributed systems. In HPPAC 2009, 2009.Google Scholar
  24. 24.
    J. Emeras. Workload Traces Analysis and Replay in Large Scale Distributed Systems. PhD thesis, LIG, Grenoble - France, To be defended October 1st 2013. currently available at:
  25. 25.
    M. Flynn. Some computer organizations and their effectiveness. IEEE Transactions on Computers, C(21):948–960, 1972.Google Scholar
  26. 26.
    Y. Georgiou. Contributions for Resource and Job Management in High Performance Computing. PhD thesis, LIG, Grenoble - France, Sep 2010.Google Scholar
  27. 27.
    M. Guzek, S. Varrette, V. Plugaru, J. E. Sanchez, and P. Bouvry. A Holistic Model of the Performance and the Energy-Efficiency of Hypervisors in an HPC Environment. In Proc. of the Intl. Conf. on Energy Efficiency in Large Scale Distributed Systems (EE-LSDS'13), volume 8046 of LNCS, Vienna, Austria, Apr 2013. Springer Verlag.Google Scholar
  28. 28.
    R. Januszewski, N. Meyer, and J. Nowicka. Evaluation of the impact of direct warm-water cooling of the HPC servers on the data center ecosystem. In To appear in International Supercomputing Conference 2014, Leipzig, Germany, 2014.Google Scholar
  29. 29.
    M. Jarus, S. Varette, A. Oleksiak, and P. Bouvry. Performance Evaluation and Energy Efficiency of High-Density HPC Platforms Based on Intel, AMD and ARM Processors. In Energy Efficiency in Large Scale Distributed Systems, Lecture Notes in Computer Science, pages 182–200. Springer Berlin Heidelberg, 2013.Google Scholar
  30. 30.
    A. Kivity and al. kvm: the Linux virtual machine monitor. In Ottawa Linux Symposium, pages 225–230, July 2007.Google Scholar
  31. 31.
    V. Kundra. Federal data center consolidation initiative. Memorandum for chief information officers, Office of Management and Budget of the USA, 2010.Google Scholar
  32. 32.
    E. Le Seur and G. Heiser. Dynamic voltage and frequency scaling: the laws of diminishing returns. In HotPower'10 Proceedings of the 2010 international conference on Power aware computing and systems, California, USA, 2010. USENIX Association Berkeley.Google Scholar
  33. 33.
    R. C. Murphy, K. B. Wheeler, B. W. Barrett, and J. A. Ang. Introducing the graph 500. In Cray User Group, 2010.Google Scholar
  34. 34.
    A.-C. Orgerie, L. Lefèvre, and J.-P. Gelas. Save watts in your grid: Green strategies for energy-aware framework in large scale distributed systems. In 14th IEEE International Conference on Parallel and Distributed Systems (ICPADS), Melbourne, Australia, Dec. 2008.Google Scholar
  35. 35.
    A. Petitet, C. Whaley, J. Dongarra, A. Cleary, and P. Luszczek. HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers.Google Scholar
  36. 36.
    N. Rasmussen. Air Distribution Architecture Options for Mission Critical Facilities Whitepaper #55. Technical report, American Power Conversion, 2003.Google Scholar
  37. 37.
    S. Sharma, C.-H. Hsu, and W. chun Feng. Making a case for a Green500 list. In Parallel and Distributed Processing Symposium, 2006. IPDPS 2006. 20th International, pages 8 pp.–, 2006.Google Scholar
  38. 38.
    S. Varrette, M. Guzek, V. Plugaru, X. Besseron, and P. Bouvry. HPC Performance and Energy-Efficiency of Xen, KVM and VMware Hypervisors. In Proc. of the 25th Symposium on Computer Architecture and High Performance Computing (SBAC-PAD 2013), Porto de Galinhas, Brazil, Oct. 2013. IEEE Computer Society.Google Scholar
  39. 39.
    M. vor dem Berge, J. Buchholz, L. Cupertino, G. Da Costa, A. Donoghue, G. Gallizo, M. Jarus, L. Lopez, A. Oleksiak, E. Pages, W. Piatek, J.-M. Pierson, T. Piontek, D. Rathgeb, J. Salom, L. Siso, E. Volk, W. U., and T. Zilio. CoolEmAll: Models and Tools for Planning and Operating Energy Efficient Data Centres. To appear in: Samee Khan, Albert Zomaya (eds.) Handbook on Data Centers.Google Scholar
  40. 40.
    M. vor dem Berge, G. Da Costa, A. Kopecki, A. Oleksiak, J.-M. Pierson, T. Piontek, E. Volk, and S. Wesner. Modeling and Simulation of Data Center Energy-Efficiency in CoolEmAll. Energy Efficient Data Centers. Lecture Notes in Computer Science, 7396:25–36, 2012.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Sébastien Varrette
    • 1
  • Pascal Bouvry
    • 1
  • Mateusz Jarus
    • 2
  • Ariel Oleksiak
    • 2
  1. 1.Computer Science and Communication (CSC) Research UnitUniversity of LuxembourgLuxembourgLuxembourg
  2. 2.Poznań Supercomputing and Networking CenterPoznańPoland

Personalised recommendations