The Journal of Supercomputing

, Volume 73, Issue 5, pp 1978–2000 | Cite as

Energy efficiency of dynamic management of virtual cluster with heterogeneous hardware

  • Jukka KommeriEmail author
  • Tapio Niemi
  • Jukka K. Nurminen


Cloud computing is an essential part of today’s computing world. Continuously increasing amount of computation with varying resource requirements is placed in large data centers. The variation among computing tasks, both in their resource requirements and time of processing, makes it possible to optimize the usage of physical hardware by applying cloud technologies. In this work, we develop a prototype system for load-based management of virtual machines in an OpenStack computing cluster. Our prototype is based on an idea of ‘packing’ idle virtual machines into special park servers optimized for this purpose. We evaluate the method by running real high-energy physics analysis software in an OpenStack test cluster and by simulating the same principle using the Cloudsim simulator software. The results show a clear improvement, 9–48 % , in the total energy efficiency when using our method together with resource overbooking and heterogeneous hardware.


Energy efficiency OpenStack Cloudsim Over-commit Heterogeneous hardware 



This paper has received funding from the European Union’s Horizon 2020 research and innovation program 2014–2018 under Grant Agreement No. 644866.


  1. 1.
    Ahmed A, Sabyasachi AS (2014) Cloud computing simulators: a detailed survey and future direction. In: 2014 IEEE International Advance Computing Conference (IACC), pp 866–872. doi: 10.1109/IAdCC.2014.6779436
  2. 2.
    Antcheva I, Ballintijn M, Bellenot B, Biskup M (2009) Root-a c++ framework for petabyte data storage, statistical analysis and visualization. Comput Phys Commun 180(12):2499–2512CrossRefGoogle Scholar
  3. 3.
    Banga G, Druschel P, Mogul JC (1999) Resource containers: a new facility for resource management in server systems. OSDI 99:45–58Google Scholar
  4. 4.
    Barroso LA, Hölzle U (2007) The case for energy-proportional computing. Computer 40:33–37CrossRefGoogle Scholar
  5. 5.
    Beloglazov A, Buyya R (2010) Energy efficient allocation of virtual machines in cloud data centers. In: 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid), pp 577–578. doi: 10.1109/CCGRID.2010.45
  6. 6.
    Beloglazov A, Buyya R (2012) Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers. Concurr Comput Pract Exp 24(13):1397–1420. doi: 10.1002/cpe.1867 CrossRefGoogle Scholar
  7. 7.
    Breitgand D, Dubitzky Z, Epstein A, Glikson A, Shapira I (2013) Sla-aware resource over-commit in an iaas cloud. In: Proceedings of the 8th International Conference on Network and Service Management, CNSM ’12, pp 73–81. International Federation for Information Processing, Laxenburg, Austria.
  8. 8.
    Calheiros RN, Ranjan R, Beloglazov A, De Rose CAF, Buyya R (2011) Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw Pract Exp 41(1):23–50. doi: 10.1002/spe.995 CrossRefGoogle Scholar
  9. 9.
    Chung-Hsing H, Poole S (2013) Revisiting server energy proportionality. In: 2013 42nd International Conference on Parallel Processing (ICPP), pp 834–840. doi: 10.1109/ICPP.2013.99
  10. 10.
    Cioara T, Anghel I, Salomie I, Copil G, Moldovan D, Kipp A (2011) Energy aware dynamic resource consolidation algorithm for virtualized service centers based on reinforcement learning. In: 2011 10th International Symposium on Parallel and Distributed Computing (ISPDC), pp 163–169. doi: 10.1109/ISPDC.2011.32
  11. 11.
    Corradi A, Fanelli M, Foschini L (2014) VM consolidation: a real case based on openstack cloud. Future Gen Comput Syst 32(0): 118–127. doi: 10.1016/j.future.2012.05.012. Special Section: The Management of Cloud Systems, Special Section: Cyber-Physical Society and Special Section: Special Issue on Exploiting Semantic Technologies with Particularization on Linked Data over Grid and Cloud Architectures
  12. 12.
    Crago S, Dunn K, Eads P, Hochstein L, Kang DI, Kang M, Modium D, Singh K, Suh J, Walters J (2011) Heterogeneous cloud computing. In: 2011 IEEE International Conference on Cluster Computing (CLUSTER), pp 378–385. doi: 10.1109/CLUSTER.2011.49
  13. 13.
    Dabbagh M, Hamdaoui B, Guizani M, Rayes A (2015) Toward energy-efficient cloud computing: Prediction, consolidation, and overcommitment. IEEE Netw 29(2):56–61. doi: 10.1109/MNET.2015.7064904 CrossRefGoogle Scholar
  14. 14.
    Dinda PA, O’Hallaron DR (2000) Host load prediction using linear models. Cluster Comput 3(4):265–280CrossRefGoogle Scholar
  15. 15.
    Fabozzi F, Jones C, Hegner B, Lista L (2008) Physics analysis tools for the cms experiment at lhc. IEEE Trans Nucl Sci 55:3539–3543CrossRefGoogle Scholar
  16. 16.
    Ghosh R, Naik V (2012) Biting off safely more than you can chew: predictive analytics for resource over-commit in iaas cloud. In: 2012 IEEE 5th International Conference on Cloud Computing (CLOUD), pp 25–32. doi: 10.1109/CLOUD.2012.131
  17. 17.
    Grzonka D, Kolodziej J, Tao J et al (2015) The analysis of openstack cloud computing platform: Features and performance. J Telecommun Inf Technol 52(3):52–57Google Scholar
  18. 18.
    Gupta A, Milojicic D, Kalé LV (2012) Optimizing vm placement for hpc in the cloud. In: Proceedings of the 2012 Workshop on Cloud Services, Federation, and the 8th Open Cirrus Summit, FederatedClouds ’12, pp 1–6. ACM, New York, NY, USA. doi: 10.1145/2378975.2378977
  19. 19.
    He S, Guo L, Guo Y, Wu C, Ghanem M, Han R (2012) Elastic application container: a lightweight approach for cloud resource provisioning. In: 2012 IEEE 26th International Conference on Advanced Information Networking and Applications, pp 15–22. doi: 10.1109/AINA.2012.74
  20. 20.
    Hermenier F, Lorca X, Menaud JM, Muller G, Lawall J (2009) Entropy: a consolidation manager for clusters. In: Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments VEE ’09, pp 41–50. ACM, New York, NY, USA. doi: 10.1145/1508293.1508300
  21. 21.
    Hosseinimotlagh S, Khunjush F, Samadzadeh R (2015) Seats: smart energy-aware task scheduling in real-time cloud computing. J Supercomput 71(1):45–66. doi: 10.1007/s11227-014-1276-9 CrossRefGoogle Scholar
  22. 22.
    Intel (2016) Your source for intel product specifications. Tech. rep., Intel CorporationGoogle Scholar
  23. 23.
    Jackson K, Bunch C, Sigler E (2015) OpenStack cloud computing cookbook. Packt Publishing Ltd, BirminghamGoogle Scholar
  24. 24.
    Kommeri J, Niemi T, Helin O (2012) Energy efficiency of server virtualization. Int J Adv Intell Syst 5(3–4):90–95Google Scholar
  25. 25.
    von Laszewski G, Diaz J, Wang F, Fox GC (2012) Comparison of multiple cloud frameworks. In: 2012 IEEE 5th International Conference on Cloud Computing (CLOUD), pp 734–741. doi: 10.1109/CLOUD.2012.104
  26. 26.
    Li B, Li J, Huai J, Wo T, Li Q, Zhong L (2009) Enacloud: an energy-saving application live placement approach for cloud computing environments. In: IEEE International Conference on Cloud Computing, 2009. CLOUD ’09, pp 17–24. doi: 10.1109/CLOUD.2009.72
  27. 27.
    Litvinski O, Gherbi A (2013) Experimental evaluation of openstack compute scheduler. Proc Comput Sci 19:116–123CrossRefGoogle Scholar
  28. 28.
    Medrano Llamas R, Barreiro M, Fernando H, Kucharczyk K, Denis MK, Cinquilli M (2013) Commissioning the cern it agile infrastructure with experiment workloads. In: 20th International Conference on Computing in High Energy and Nuclear PhysicsGoogle Scholar
  29. 29.
    Meinhard H (2012) Virtualization, clouds and iaas at cern. In: Proceedings of the 6th International Workshop on Virtualization Technologies in Distributed Computing Date, VTDC ’12, pp 27–28. ACM, New York, NY, USA. doi: 10.1145/2287056.2287064
  30. 30.
    Meusel R, Blomer J, Buncic P, Ganis G, Heikkilä SS (2015) Recent developments in the cernvm-file system server backend. J Phys Conf Ser 608(1):012,031CrossRefGoogle Scholar
  31. 31.
    Nathuji R, Schwan K (2007) Virtualpower: coordinated power management in virtualized enterprise systems. SIGOPS Oper Syst Rev 41(6):265–278. doi: 10.1145/1323293.1294287 CrossRefGoogle Scholar
  32. 32.
    Niemi T, Hameri AP (2012) Memory-based scheduling of scientific computing clusters. J Supercomput 61(3):520–544CrossRefGoogle Scholar
  33. 33.
    Ou Z, Zhuang H, Nurminen JK, Ylä-Jääski A, Hui P (2012) Exploiting hardware heterogeneity within the same instance type of amazon ec2. In: Proceedings of the 4th USENIX Conference on Hot Topics in Cloud Ccomputing, HotCloud’12, pp 4–4. USENIX Association, Berkeley, CA, USA.
  34. 34.
    Pahlavan A, Momtazpour M, Goudarzi M (2014) Power reduction in hpc data centers: a joint server placement and chassis consolidation approach. J Supercomput 70(2):845–879. doi: 10.1007/s11227-014-1265-z CrossRefGoogle Scholar
  35. 35.
    Peng J, Zhang X, Lei Z, Zhang B, Zhang W, Li Q (2009) Comparison of several cloud computing platforms. In: 2009 Second International Symposium on Information Science and Engineering, pp 23–27. doi: 10.1109/ISISE.2009.94
  36. 36.
    Ponce S, Hersch RD (2004) Parallelization and scheduling of data intensive particle physics analysis jobs on clusters of pcs. In: CD-ROM/Abstracts Proceedings 18th International Parallel and Distributed Processing Symposium (IPDPS 2004), 26-30 April 2004, Santa Fe, New Mexico, USA. doi: 10.1109/IPDPS.2004.1303280
  37. 37.
    Ross SM (1996) Stochastic Processes, 2nd edn. Wiley, New YorkzbMATHGoogle Scholar
  38. 38.
    Sato K, Samejima M, Komoda N (2013) Dynamic optimization of virtual machine placement by resource usage prediction. In: 2013 11th IEEE International Conference on Industrial Informatics (INDIN), pp 86–91. doi: 10.1109/INDIN.2013.6622863
  39. 39.
    Sevalnev M, Aalto S, Kommeri J, Niemi T (2012) Using queuing theory for controlling the number of computing servers. In: ICGREEN 2012 (Third International Conference on Green IT Solutions (2012)Google Scholar
  40. 40.
    Sharifi M, Salimi H, Najafzadeh M (2012) Power-efficient distributed scheduling of virtual machines using workload-aware consolidation techniques. J Supercomput 61(1):46–66. doi: 10.1007/s11227-011-0658-5 CrossRefGoogle Scholar
  41. 41.
    Srikantaiah S, Kansal A, Zhao F (2008) Energy aware consolidation for cloud computing. In: Proceedings of the 2008 Conference on Power Aware Computing and Systems, HotPower’08, p 10. USENIX Association, Berkeley, CA, USA.
  42. 42.
    Takahiro H, Hidemoto N, Satoshi I, Satoshi S (2012) Reactive cloud: consolidating virtual machines with postcopy live migration. Inf Media Technol 7(2):614–626Google Scholar
  43. 43.
    Takouna I, Meinel C (2014) Coordinating vms’ memory demand heterogeneity and memory dvfs for energy-efficient vms consolidation. In: IEEE and Cyber, Physical and Social Computing(CPSCom), IEEE Internet of Things (iThings), 2014 IEEE International Conference on, and Green Computing and Communications (GreenCom), pp 478–485. doi: 10.1109/iThings.2014.85
  44. 44.
    Tomás L, Tordsson J (2013) Improving cloud infrastructure utilization through overbooking. In: Proceedings of the 2013 ACM Cloud and Autonomic Computing Conference, CAC ’13, pp 5:1–5:10. ACM, New York, NY, USA. doi: 10.1145/2494621.2494627
  45. 45.
    Verma A, Ahuja P, Neogi A (2008) Power-aware dynamic placement of hpc applications. In: Proceedings of the 22nd Annual International Conference on Supercomputing, ICS ’08ACM, New York, NY, USA, pp 175–184Google Scholar
  46. 46.
    Wang X, Liu X, Fan L, Jia X (2013) A decentralized virtual machine migration approach of data centers for cloud computing. Math Probl Eng 10:878542. doi: 10.1155/2013/878542
  47. 47.
    Wen X, Gu G, Li Q, Gao Y, Zhang X (2012) Comparison of open-source cloud management platforms: Openstack and opennebula. In: 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD), pp 2457–2461. doi: 10.1109/FSKD.2012.6234218
  48. 48.
    Wuhib F, Stadler R, Lindgren H (2012) Dynamic resource allocation with management objective: implementation for an openstack cloud. In: 2012 8th International Conference and 2012 Workshop on Systems Virtualiztion Management (svm) Network and service management (cnsm), pp 309–315Google Scholar
  49. 49.
    Younge A, Fox G (2014) Advanced virtualization techniques for high performance cloud cyberinfrastructure. In: 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp 583–586. doi: 10.1109/CCGrid.2014.93

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Jukka Kommeri
    • 1
    Email author
  • Tapio Niemi
    • 2
  • Jukka K. Nurminen
    • 3
  1. 1.Helsinki Institute of PhysicsHelsinkiFinland
  2. 2.Helsinki Institute of Physics, CERNGenevaSwitzerland
  3. 3.Aalto UniversityEspooFinland

Personalised recommendations