Cluster Computing

, Volume 18, Issue 3, pp 1041–1062 | Cite as

Agent-based load balancing in Cloud data centers

  • J. Octavio Gutierrez-Garcia
  • Adrian Ramirez-Nafarrate


Cloud data centers are generally composed of heterogeneous commodity servers hosting multiple virtual machines (VMs) with potentially different specifications and fluctuating resource usages. This may cause a resource usage imbalance within servers that may result in performance degradation and violations to service level agreements. This work proposes a collaborative agent-based problem solving technique capable of balancing workloads across commodity, heterogeneous servers by making use of VM live migration. The agents are endowed with (i) migration heuristics to determine which VMs should be migrated and their destination hosts, (ii) migration policies to decide when VMs should be migrated, (iii) VM acceptance policies to determine which VMs should be hosted, and (iv) front-end load balancing heuristics. The results show that agents, through autonomous and dynamic collaboration, can efficiently balance loads in a distributed manner outperforming centralized approaches with a performance comparable to commercial solutions, namely Red Hat, while migrating fewer VMs.


Load balancing Virtual machine live migration Data centers Cloud computing  Intelligent agents Multiagent systems 



The authors would like to thank the Editor-in-Chief and the anonymous referees for their comments and suggestions. This work has been supported by Asociación Mexicana de Cultura A.C.


  1. 1.
    Amazon EC2: Elastic load balancing. (2015). Accessed 1 Feb 2015
  2. 2.
    Anderson, P., Bijani, S., Vichos, A.: Multi-agent negotiation of virtual machine migration using the lightweight coordination calculus. In: Jezic, G., Kusek, M., Nguyen, N.-T., Howlett, R.J., Jain, L.C. (eds.) 6th KES International Conference on Agent and Multi-agent Systems: Technologies and Applications. Lecture Notes in Computer Science, vol. 7327, pp. 124–133. Springer, Berlin (2012)CrossRefGoogle Scholar
  3. 3.
    Andreolini, M., Casolari, S., Colajanni, M., Messori, M.: Dynamic load management of virtual machines in cloud architectures. In: Avresky, D.R., Diaz, M., Bode, A., Ciciani, B., Dekel, E. (eds.) Cloud Computing. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, vol. 34, pp. 201–214. Springer, Berlin (2010)CrossRefGoogle Scholar
  4. 4.
    Bansal, N., Lodi, A., Sviridenko, M.: A tale of two dimensional bin packing. In: Proceedings of the 46th Annual IEEE Symposium on Foundations of Computer Science, pp. 657–666 (2005)Google Scholar
  5. 5.
    Bellifemine, F., Caire, G., Poggi, A., Rimassa, G.: JADE—a white paper. EXP Search Innov. 3(3), 6–19 (2003)Google Scholar
  6. 6.
    Bobroff, N., Kochut, A., Beaty K.: Dynamic placement of virtual machines for managing sla violations. In: Proceedings of the 10th IFIP/IEEE International Symposium on Integrated Network Management, pp. 119–128 (2007)Google Scholar
  7. 7.
    Breitgand, D., Marashini, A., Tordsson, J.: Policy-driven service placement optimization in federated clouds. Technical Report, IBM Research Division (2011)Google Scholar
  8. 8.
    Buyya, R., Beloglazov, A., Abawajy, J.: Energy-efficient management of data center resources for cloud computing: a vision, architectural elements, and open challenges. In: Proceedings of the 2010 International Conference on Parallel and Distributed Processing Techniques and Application, pp. 12–15 (2010)Google Scholar
  9. 9.
    Canturk, I., Chengwei, W., Bhatt, C., Shanmuganathan, G., Holler, A.M.: Process demand prediction for distributed power and resource management. U.S. Patent Application 13/281,234, 2011Google Scholar
  10. 10.
    Chekuri, C., Khanna, S.: On multi-dimensional packing problems. SIAM J. Comput. 33(4), 837–851 (2014)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Cisco systems: Cisco USB B440 M2 high-performance blade server data sheet. (2015). Accessed 1 Feb 2015
  12. 12.
    Clark, C., Fraser, K., Hand, S., Hansen, J.G., Jul, E., Limpach, C., Pratt, I., Warfield, A.: Live migration of virtual machines. In: Proceedings of the 2nd Conference on Symposium on Networked Systems Design & Implementation, USENIX Association, vol. 2, pp. 273–286 (2005)Google Scholar
  13. 13.
    Coffman, E.G., Garey, M.R., Johnson, D.S., Tarjan, R.E.: Performance bounds for level-oriented two-dimensional packing algorithms. SIAM J. Comput. 9(4), 808–826 (1980)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Daniels, J.: Server virtualization architecture and implementation. Crossroads 16(1), 8–12 (2009)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Das, B., Bharghavan, V.: Routing in ad-hoc networks using minimum connected dominating sets. In: Proceedings of the IEEE International Conference on Communications, pp. 376–380 (1997)Google Scholar
  16. 16.
    Gutierrez-Garcia, J.O., Ramirez-Nafarrate, A.: Policy-based agents for virtual machine migration in cloud data centers. In: Proceedings of the IEEE International Conference on Services Computing, pp. 603–610 (2013)Google Scholar
  17. 17.
    Gutierrez-Garcia, J.O., Sim, K.M.: A family of heuristics for agent-based elastic cloud bag-of-tasks concurrent scheduling. Future Gener. Comput. Sys. 29(7), 1682–1699 (2013)CrossRefGoogle Scholar
  18. 18.
    Gutierrez-Garcia, J.O., Sim, K.M.: Agent-based Cloud Service Composition. Appl. Intell. 38(3), 436–464 (2013)CrossRefGoogle Scholar
  19. 19.
    Hyser, C., Mckee, B., Gardner, R., Watson, B.J.: Autonomic virtual machine placement in the data center. Hewlett Packard Laboratories, Technical Report HPL-2007-189, 2007–189 (2007)Google Scholar
  20. 20.
    Ji, M., Waldspurger, C.A., Zedlewski, J.: Method and system for determining a cost-benefit metric for potential virtual machine migrations. U.S. Patent No. 8,095,929, 2012. Washington, DC: U.S. Patent and Trademark OfficeGoogle Scholar
  21. 21.
    Kerr, A., Diamos, G., Yalamanchili, S.: A characterization and analysis of GPGPU kernels. Technical Report GIT-CERCS-09-06, Georgia Institute of Technology (2009)Google Scholar
  22. 22.
    Kleiminger, W., Kalyvianaki, E., Pietzuch, P.: Balancing load in stream processing with the cloud. In: Proceedings of the 2011 IEEE 27th International Conference on Data Engineering Workshops, pp. 16–21 (2011)Google Scholar
  23. 23.
    Lang, W., Patel, J.M., Naughton, J.F.: On energy management, load balancing and replication. ACM SIGMOD Rec. 38(4), 35–42 (2010)CrossRefGoogle Scholar
  24. 24.
    Lin, C.C., Liu, P., Wu, J.J.: Energy-efficient virtual machine provision algorithms for cloud systems. In: Proceedings of the 2011 Fourth IEEE International Conference on Utility and Cloud Computing, pp. 81–88 (2011)Google Scholar
  25. 25.
    Liu, H., Xu, C.Z., Jin, H., Gong, J., Liao, X.: Performance and energy modeling for live migration of virtual machines. In: Proceedings of the 20th International Symposium on High Performance Distributed Computing, pp. 171–182 (2011)Google Scholar
  26. 26.
    Metri, G., Srinivasaraghavan, S., Shi, W., Brockmeyer, M.: Experimental analysis of application specific energy efficiency of data centers with heterogeneous servers. In: Proceedings of IEEE 5th International Conference on Cloud Computing, pp. 786–793 (2012)Google Scholar
  27. 27.
    Montgomery, D.C., Jennings, C.L., Kulahci, M.: Introduction to Time Series Analysis and Forecasting. Wiley, Hoboken (2011)Google Scholar
  28. 28.
    Patel, P., Bansal, D., Yuan, L., Murthy, A., Greenberg, A., Maltz, D. A., Kern, R., Kumar, H., Zikos, M., Wu, H., Kim, C., Karri, N.: Ananta: Cloud scale load balancing. In: Proceedings of the ACM Conference on SIGCOMM, pp. 207–218 (2013)Google Scholar
  29. 29.
  30. 30.
  31. 31.
    Red Hat: Red Hat Enterprise Virtualization 3.2—explanation of settings and controls in the Edit cluster policy window. (2015). Accessed 1 Feb 2015
  32. 32.
    Red Hat: Red Hat Enterprise Virtualization 3.2—technical reference guide. (2015). Accessed 1 Feb 2015
  33. 33.
    Ren, G., Tune, E., Moseley, T., Shi, Y., Rus, S., Hundt, R.: Google-wide profiling: a continuous profiling infrastructure for data centers. IEEE Micro 30(4), 65–79 (2010)CrossRefGoogle Scholar
  34. 34.
    Ren, X., Lin, R., Zou, H.: A dynamic load balancing strategy for cloud computing platform based on exponential smoothing forecast. In: Proceedings of the 2011 IEEE International Conference on Cloud Computing and Intelligence Systems, pp. 220–224 (2011)Google Scholar
  35. 35.
    Siebert, E.: Understanding VMware ESX virtual machine physical and virtual hardware. (2015). Accessed 1 Feb 2015
  36. 36.
    Sim, K.M.: Agent-based cloud computing. IEEE T. Serv. Comput. 5(4), 564–577 (2012)CrossRefGoogle Scholar
  37. 37.
    Skiena, S.: The Algorithm Design Manual: Text, vol. 1. Springer, Berlin (1998)Google Scholar
  38. 38.
    SPEC: All published SPECvirt_sc2010 performance results SPEC. (2015). Accessed 1 Feb 2015
  39. 39.
    Speitkamp, B., Bichler, M.: A mathematical programming approach for server consolidation problems in virtualized data centers. IEEE T. Serv. Comput. 3(4), 266–278 (2010)CrossRefGoogle Scholar
  40. 40.
    Tsai, Y.L., Huang, K.C., Chang, H.Y., Ko, J., Wang, E.T., Hsu, C.H.: Scheduling multiple scientific and engineering workflows through task clustering and best-fit allocation. In: Proceedings of the 2012 IEEE Eighth World Congress on Services, pp. 1–8 (2012)Google Scholar
  41. 41.
    Vaquero, L.M., Rodero-Merino, L., Caceres, J., Lindner, M.: A break in the clouds: towards a cloud definition. ACM SIGCOMM Computer Communication Review 39(1), 50–55 (2008)CrossRefGoogle Scholar
  42. 42.
    VMware: Distributed resource scheduler—recommendation page. (2015). Accessed 1 Feb 2015
  43. 43.
    VMware: VMware Infrastructure—Resource management with VMware distributed resource scheduler. (2015). Accessed 1 Feb 2015
  44. 44.
    Von Laszewski, G., Diaz, J., Wang, F., Fox, G.C.: Comparison of multiple cloud frameworks. In: Proceedings of the 2012 IEEE 5th International Conference on Cloud Computing, pp. 734–741 (2012)Google Scholar
  45. 45.
    Voorsluys, W., Broberg, J., Venugopal, S., Buyya, R.: Cost of virtual machine live migration in clouds: a performance evaluation. Cloud Computing, pp. 254–265. Springer, Berlin (2009)CrossRefGoogle Scholar
  46. 46.
    Wood, T., Shenoy, P., Venkataramani, A., Yousif, M.: Black-box and gray-box strategies for virtual machine migration. In: Proceedings of the Fourth USENIX Symposium on Networked Systems Design & Implementation, pp. 701–708 (2007)Google Scholar
  47. 47.
    Wu, Y., Yuan, Y., Yang, G., Zheng, W.: Load prediction using hybrid model for computational grid. In: Proceedings of the 8th IEEE/ACM International Conference on Grid Computing, pp. 235–242 (2007)Google Scholar
  48. 48.
    Xing, L.: A self-management approach to service optimization and system integrity through multi-agent systems. Master’s Thesis, University of Oslo (2008)Google Scholar
  49. 49.
    Zhang, H., Jiang, F., Yoshihira, K., Chen, H., Saxena, A.: Intelligent workload factoring for a hybrid cloud computing model. In: Proceedings of the 2009 World Conference on Services-I, pp. 701–708 (2009)Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • J. Octavio Gutierrez-Garcia
    • 1
  • Adrian Ramirez-Nafarrate
    • 2
  1. 1.Department of Computer ScienceInstituto Tecnológico Autónomo de MéxicoMexico CityMexico
  2. 2.Department of Industrial & Operations EngineeringInstituto Tecnológico Autónomo de MéxicoMexico CityMexico

Personalised recommendations