Skip to main content
SpringerLink
Log in

Scalable Virtual Machine Migration using Reinforcement Learning

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

Heuristic approaches require fixed knowledge of how resource allocation should be carried out, and this can be limiting when managing variable cloud workloads. Solutions based on Reinforcement Learning (RL) have been presented to manage cloud infrastructure, however, these tend to be centralized and suffer in their ability to maintain Quality of Service (QoS) for data centres with thousands of nodes. To address this, we propose a reinforcement learning management policy, which can run decentralized, and achieve fast convergence towards efficient resource allocation, resulting in lower SLA violations compared to centralized architectures. To address some of the common challenges in applying RL to cloud resource management, such as slow learning and state/action management, we use parallel learning and reduction of the state/action space. We apply a decision making approach to optimize the migration of a VM and choose a target node to host the VM in such a way that brings response time within SLA level. We have also demonstrate unique, multi-level reinforcement learning cooperation, that further reduces SLA violations. We use simulation to evaluate and demonstrate our proposal in practice, and compare the results obtained with an established heuristic, demonstrating significant improvement to SLA violations and higher scalability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

The data traces used as input for this paper are part of the DCSim [57] simulator, and are available from Github: https://github.com/digs-uwo/dcsim/tree/master/traces. Most of the data generated by this research is contained in the results section of this paper, and the full datasets generated are available from the corresponding author on request.

References

  1. Hpe proliant (2016). https://www.hpe.com

  2. Vmware. (2016). http://www.vmware.com/

  3. Ahmad, R. W., Gani, A., Hamid, S. H. A., Shiraz, M., Yousafzai, A., Xia, F.: A survey on virtual machine migration and server consolidation frameworks for cloud data centers. J. Netw. Comput. Appl. 52, 11–25 (2015). https://doi.org/10.1016/j.jnca.2015.02.002

    Article  Google Scholar 

  4. Aldhalaan, A., Menascé, D. A.: Autonomic Allocation of Communicating Virtual Machines in Hierarchical Cloud Data Centers. In: 2014 International Conference On Cloud and Autonomic Computing (ICCAC), pp. 161–171. 2014 International Conference On Cloud and Autonomic Computing (2014) (2014)

  5. Arabnejad, H., Pahl, C., Jamshidi, P., Estrada, G.: A Comparison of Reinforcement Learning Techniques for Fuzzy Cloud Auto-Scaling. In: 2017 17Th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 64–73. https://doi.org/10.1109/CCGRID.2017.15 (2017)

  6. Barrett, E., Howley, E., Duggan, J.: Applying reinforcement learning towards automating resource allocation and application scalability in the cloud. Concurr. Comput. Pract. Exper. 25(12), 1656–1674 (2013). https://doi.org/10.1002/cpe.2864

    Article  Google Scholar 

  7. Beloglazov, A., Buyya, R.: Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers. Concurr. Comput. Pract. Exper. 24, 1397–1420 (2012)

    Article  Google Scholar 

  8. Bibal Benifa, J. V., Dejey, D.: Rlpas: Reinforcement learning-based proactive auto-scaler for resource provisioning in cloud environment. Mobile Netw. Appl. 24(4), 1348–1363 (2019). https://doi.org/10.1007/s11036-018-0996-0

    Article  Google Scholar 

  9. Bitsakos, C., Konstantinou, I., Koziris, N.: Derp: a Deep Reinforcement Learning Cloud System for Elastic Resource Provisioning. In: 2018 IEEE International Conference on Cloud Computing Technology and Science (Cloudcom), pp. 21–29. https://doi.org/10.1109/CloudCom2018.2018.00020 (2018)

  10. Bu, X., Rao, J., Xu, C. Z.: Coordinated self-configuration of virtual machines and appliances using a model-free learning approach. IEEE Trans. Parallel Distrib. Syst. 24(4), 681–690 (2013). https://doi.org/10.1109/TPDS.2012.174

    Article  Google Scholar 

  11. Calcavecchia, N. M., Caprarescu, B. A., Di Nitto, E., Dubois, D. J., Petcu, D.: Depas: a decentralized probabilistic algorithm for auto-scaling. Computing 94(8), 701–730 (2012)

    Article  Google Scholar 

  12. Chen, Z., Hu, J., Min, G.: Learning-Based Resource Allocation in Cloud Data Center Using Advantage Actor-Critic. In: ICC 2019 - 2019 IEEE International Conference on Communications (2019), pp. 1–6. https://doi.org/10.1109/ICC.2019.8761309 (2019)

  13. Chowdhury, M. R., Mahmud, M. R., Rahman, R. M.: Implementation and performance analysis of various vm placement strategies in cloudsim. J. Cloud Comput. 4(1), 20 (2015). https://doi.org/10.1186/s13677-015-0045-5

    Article  Google Scholar 

  14. Citrix: Xen. (2016). http://www.xenserver.org

  15. Duggan, M., Flesk, K., Duggan, J., Howley, E.: Barrett e.: a reinforcement learning approach for dynamic selection of virtual machines in cloud data centres. https://doi.org/10.1109/INTECH.2016.7845053(2016)

  16. Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., Hester, T.: Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning. https://doi.org/10.1007/s10994-021-05961-4 (2021)

  17. Gahlawat, M., Sharma, P.: Survey of Virtual Machine Placement in Federated Clouds. In: 2014 IEEE International Advance Computing Conference (IACC), pp. 735–738. https://doi.org/10.1109/IAdCC.2014.6779415 (2014)

  18. Ghanbari, H., Simmons, B., Litoiu, M., Barna, C., Iszlai, G.: Optimal autoscaling in a iaas cloud. In: Proceedings of the 9th International Conference on Autonomic Computing, ICAC ’12, pp 173–178. Association for Computing Machinery, New York (2012), https://doi.org/10.1145/2371536.2371567

  19. Ghobaei-Arani, M., Jabbehdari, S., Pourmina, M. A.: An autonomic resource provisioning approach for service-based cloud applications: a hybrid approach. Fut. Gener. Comput. Syst. 78, 191–210 (2018). https://doi.org/10.1016/j.future.2017.02.022

    Article  Google Scholar 

  20. Gholipour, N., Arianyan, E., Buyya, R.: A novel energy-aware resource management technique using joint vm and container consolidation approach for green computing in cloud data centers. Simul. Model. Pract. Theory 104, 102127 (2020). https://doi.org/10.1016/j.simpat.2020.102127

    Article  Google Scholar 

  21. Guo, W., Tian, W., Ye, Y., Xu, L., Wu, K.: Cloud resource scheduling with deep reinforcement learning and imitation learning. IEEE Internet Things J. 8(5), 3576–3586 (2021). https://doi.org/10.1109/JIOT.2020.3025015

    Article  Google Scholar 

  22. Gupta, M. K., Amgoth, T.: Resource-aware virtual machine placement algorithm for iaas cloud. J. Supercomput. 74(1), 122–140 (2018). https://doi.org/10.1007/s11227-017-2112-9

    Article  Google Scholar 

  23. Hummaida, A. R., Paton, N. W., Sakellariou, R.: Adaptation in cloud resource configuration: a survey. J. Cloud Comput. 5(1), 1–16 (2016)

    Article  Google Scholar 

  24. Hummaida, A. R., Paton, N. W., Sakellariou, R.: Shdf - a Scalable Hierarchical Distributed Framework for Data Centre Management. In: 2017 16Th International Symposium on Parallel and Distributed Computing (ISPDC), pp. 102–111. 16Th International Symposium on Parallel and Distributed Computing (2017). https://doi.org/10.1109/ISPDC.2017.15 (2017)

  25. Jamshidi, P., Sharifloo, A. M., Pahl, C., Metzger, A., Estrada, G.: Self-Learning Cloud Controllers: Fuzzy Q-Learning for Knowledge Evolution. In: 2015 International Conference on Cloud and Autonomic Computing, pp. 208–211. https://doi.org/10.1109/ICCAC.2015.35 (2015)

  26. Jangiti, S., Sriram, V.S.: Scalable and direct vector bin-packing heuristic based on residual resource ratios for virtual machine placement in cloud data centers. Comput. Electr. Eng. 68, 44–61 (2018). https://doi.org/10.1016/j.compeleceng.2018.03.029

    Article  Google Scholar 

  27. Jauro, F., Chiroma, H., Gital, A. Y., Almutairi, M., Abdulhamid, S. M., Abawajy, J. H.: Deep learning architectures in emerging cloud computing architectures: Recent development, challenges and next research trend. Appl. Soft Comput. 106582, 96 (2020). https://doi.org/10.1016/j.asoc.2020.106582

    Article  Google Scholar 

  28. Jin, Y., Bouzid, M., Kostadinov, D., Aghasaryan, A.: Resource management of cloud-enabled systems using model-free reinforcement learning. Ann. Telecommun. 74(9), 625–636 (2019). https://doi.org/10.1007/s12243-019-00720-y

    Article  Google Scholar 

  29. John, I., Sreekantan, A., Bhatnagar, S.: Efficient adaptive resource provisioning for cloud applications using reinforcement learning. In: 2019 IEEE 4Th International Workshops on Foundations and Applications of Self* Systems (FAS*W), pp. 271–272. https://doi.org/10.1109/FAS-W.2019.00077 (2019)

  30. Kardani-Moghaddam, S., Buyya, R., Ramamohanarao, K.: Adrl: a hybrid anomaly-aware deep reinforcement learning-based resource scaling in clouds. IEEE Trans Parallel Distrib Syst 32 (3), 514–526 (2021). https://doi.org/10.1109/TPDS.2020.3025914

    Article  Google Scholar 

  31. Keller, G., Tighe, M., Lutfiyya, H., Bauer, M.: A Hierarchical, Topology-Aware Approach to Dynamic Data Centre Management. In: Network Operations and Management Symposium (NOMS), pp. 1 –7. Network Operations and Management Symposium (2014) (2014)

  32. Khan, T., Tian, W., Buyya, R.: Machine learning (ml)-centric resource management in cloud computing: A review and future directions (2021)

  33. Kim, S., Choi, Y.R.: Constraint-aware vm placement in heterogeneous computing clusters. Clust. Comput. 23(1), 71–85 (2020). https://doi.org/10.1007/s10586-019-02966-6

    Article  Google Scholar 

  34. Lebre, A., Pastor, J., Simonet, A., Südholt, M.: Putting the next 500 vm placement algorithms to the acid test: The infrastructure provider viewpoint. IEEE Trans. Parallel Distrib. Syst. 30(1), 204–217 (2019). https://doi.org/10.1109/TPDS.2018.2855158

    Article  Google Scholar 

  35. Liu, N., Li, Z., Xu, J., Xu, Z., Lin, S., Qiu, Q., Tang, J., Wang, Y.: A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning. In: 2017 IEEE 37Th International Conference on Distributed Computing Systems (ICDCS), pp. 372–382. https://doi.org/10.1109/ICDCS.2017.123 (2017)

  36. Lolos, K., Konstantinou, I., Kantere, V., Koziris, N.: Elastic Management of Cloud Applications Using Adaptive Reinforcement Learning. In: 2017 IEEE International Conference on Big Data (Big Data), pp. 203–212. https://doi.org/10.1109/BigData.2017.8257928 (2017)

  37. Masdari, M., Zangakani, M.: Green cloud computing using proactive virtual machine placement: Challenges and issues. J. Grid Comput. 18(4), 727–759 (2020). https://doi.org/10.1007/s10723-019-09489-9

    Article  Google Scholar 

  38. Matignon, L., Laurent, G. J., Fort-piat, N. L.: Improving Reinforcement Learning Speed for Robot Control. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3172–3177. https://doi.org/10.1109/IROS.2006.282341 (2006)

  39. Maurer, M., Brandic, I., Sakellariou, R.: Adaptive resource configuration for cloud infrastructure management. Futur. Gener. Comput. Syst. 29(2), 472–487 (2013)

    Article  Google Scholar 

  40. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  41. Moreno-Vozmediano, R., Montero, R. S., Huedo, E., Llorente, I. M.: Efficient resource provisioning for elastic cloud services based on machine learning techniques. J. Cloud Comput. 8(1), 5 (2019). https://doi.org/10.1186/s13677-019-0128-9

    Article  Google Scholar 

  42. Muller-Merbach, H.: Heuristics and their design: a survey. Eur. J. Oper. Res. 8(1), 1–23 (1981). https://ideas.repec.org/a/eee/ejores/v8y1981i1p1-23.html

  43. Nouri, S. M. R., Li, H., Venugopal, S., Guo, W., He, M., Tian, W.: Autonomic decentralized elasticity based on a reinforcement learning controller for cloud applications. Fut. Gener. Comput. Syst. 94, 765–780 (2019). https://doi.org/10.1016/j.future.2018.11.049

    Article  Google Scholar 

  44. Pantazoglou, M., Tzortzakis, G., Delis, A.: Decentralized and energy-efficient workload management in enterprise clouds. IEEE Trans. Cloud Comput. 4(2), 196–209 (2016)

    Article  Google Scholar 

  45. Pietri, I., Sakellariou, R.: Mapping virtual machines onto physical machines in cloud computing: A survey. ACM Comput. Surv 49(3). https://doi.org/10.1145/2983575 (2016)

  46. Rao, J., Bu, X., Xu, C. Z., Wang, K.: A Distributed Self-Learning Approach for Elastic Provisioning of Virtualized Cloud Resources. In: 2011 IEEE 19Th Annual International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, pp. 45–54. https://doi.org/10.1109/MASCOTS.2011.47 (2011)

  47. Ren, H., Wang, Y., Xu, C., Chen, X.: Smig-rl: An evolutionary migration framework for cloud services based on deep reinforcement learning. ACM Trans. Internet Technol 20(4). https://doi.org/10.1145/3414840 (2020)

  48. Sedaghat, M., Hernández-Rodriguez, F., Elmroth, E., Girdzijauskas, S.: Divide the Task, Multiply the Outcome: Cooperative Vm Consolidation. In: IEEE International Conference on Cloud Computing Technology and Science, pp. 300–305. IEEE International Conference on Cloud Computing Technology and Science, IEEE, Washington (2014)

  49. Shaw, R., Howley, E., Barrett, E.: Applying reinforcement learning towards automating energy efficient virtual machine consolidation in cloud data centers. Inf. Syst., 101722. https://doi.org/10.1016/j.is.2021.101722 (2021)

  50. Silva Filho, M.C., Monteiro, C.C., Inácio, P.R., Freire, M.M.: Approaches for optimizing virtual machine placement and migration in cloud environments: a survey. J. Parallel Distrib. Comput. 111, 222–250 (2018). https://doi.org/10.1016/j.jpdc.2017.08.010

    Article  Google Scholar 

  51. Sina, M., Dehghan, M., Rahmani, A. M.: Car-plive: Cloud-assisted reinforcement learning based p2p live video streaming: a hybrid approach. Multimed. Tools Appl. 78(23), 34095–34127 (2019). https://doi.org/10.1007/s11042-019-08102-1

    Article  Google Scholar 

  52. Sniezynski, B., Nawrocki, P., Wilk, M., Jarzab, M., Zielinski, K.: Vm reservation plan adaptation using machine learning in cloud computing. J. Grid Comput. 17(4), 797–812 (2019). https://doi.org/10.1007/s10723-019-09487-x

    Article  Google Scholar 

  53. Song, B., Hassan, M., Huh, E.N.: A Novel Heuristic-Based Task Selection and Allocation Framework in Dynamic Collaborative Cloud Service Platform. In: 2010 IEEE Second International Conference on Cloud Computing Technology and Science, pp. 360–367. https://doi.org/10.1109/CloudCom.2010.53 (2010)

  54. Sutton, R. S., Barto, A. G.: Reinforcement Learning: an Introduction, vol. 1. MIT press, Cambridge (1998)

    MATH  Google Scholar 

  55. Thanh Binh, H. T., Phi Le, N., Minh, N. B., Thu Hai, T., Minh, N. Q., Bao Son, D.: A Reinforcement Learning Algorithm for Resource Provisioning in Mobile Edge Computing Network. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–7. https://doi.org/10.1109/IJCNN48605.2020.9206947 (2020)

  56. Tighe, M., Keller, G., Bauer, M.: Lutfiyya: a distributed approach to dynamic vm management. In: Proceedings of the 9th International Conference on Network and Service Management, pp. 166 to 170. Proceedings of the 9th International Conference on Network and Service Management (2013)

  57. Tighe, M., Keller, G., Bauer, M., Lutfiyya, H.: Dcsim: a Data Centre Simulation Tool for Evaluating Dynamic Virtualized Resource Management. In: Network and Service Management (Cnsm), 2012 8Th International Conference and 2012 Workshop on Systems Virtualiztion Management (Svm), pp. 385–392. Network and Service Management (Cnsm), 2012 8Th International Conference and 2012 Workshop on Systems Virtualiztion Management (Svm) (2012)

  58. Walsh, W., Tesauro, G., Kephart, J., Das, R.: Utility Functions in Autonomic Systems. In: 2004. Proceedings. International Conference on Autonomic Computing, pp. 70–77. https://doi.org/10.1109/ICAC.2004.1301349 (2004)

  59. Watkins, C. J. C. H.: Learning from Delayed Rewards. In: Ph.D. Thesis (1989)

  60. Witanto, J. N., Lim, H., Atiquzzaman, M.: Adaptive selection of dynamic vm consolidation algorithm using neural network for cloud resource management. Futur. Gener. Comput. Syst. 87, 35–42 (2018). https://doi.org/10.1016/j.future.2018.04.075

    Article  Google Scholar 

  61. Wu, Y., Tang, M., Fraser, W.: A Simulated Annealing Algorithm for Energy Efficient Virtual Machine Placement. In: 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1245–1250. https://doi.org/10.1109/ICSMC.2012.6377903 (2012)

  62. Wuhib, F., Stadler, R., Spreitzer, M.: Dynamic resource allocation with management objectives: implementation for an openstack cloud. IEEE Trans. Netw. Serv. Manag. 9(2), 213–225 (2012)

    Article  Google Scholar 

  63. Xu, H., Liu, Y., Wei, W., Xue, Y.: Migration cost and energy-aware virtual machine consolidation under cloud environments considering remaining runtime. Int. J. Parallel Prog. 47(3), 481–501 (2019). https://doi.org/10.1007/s10766-018-00622-x

    Article  Google Scholar 

  64. Yadav, M. P., Rohit Yadav, D.K.: Resource provisioning through machine learning in cloud services. Arabian Journal for Science and Engineering. https://doi.org/10.1007/s13369-021-05864-5 (2021)

  65. Yadav, R., Zhang, W., Li, K., Liu, C., Shafiq, M., Karn, N. K.: An adaptive heuristic for managing energy consumption and overloaded hosts in a cloud data center. Wirel. Netw 26(3), 1905–1919 (2020). https://doi.org/10.1007/s11276-018-1874-1

    Article  Google Scholar 

  66. Ying, C., Li, B., Ke, X., Guo, L.: Raven: Scheduling virtual machine migration during datacenter upgrades with reinforcement learning. Mobile Networks and Applications. https://doi.org/10.1007/s11036-020-01632-1 (2020)

  67. Zolfaghari, R., Sahafi, A., Rahmani, A. M., Rezaei, R.: Application of virtual machine consolidation in cloud computing systems. Sustain. Comput. Inf. Syst. 30, 100524 (2021). https://doi.org/10.1016/j.suscom.2021.100524

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Manchester, Kilburn Building, Oxford Rd, Manchester, M13 9PL, UK

    Abdul Rahman Hummaida, Norman W. Paton & Rizos Sakellariou

Authors
  1. Abdul Rahman Hummaida
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Norman W. Paton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rizos Sakellariou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abdul Rahman Hummaida.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hummaida, A.R., Paton, N.W. & Sakellariou, R. Scalable Virtual Machine Migration using Reinforcement Learning. J Grid Computing 20, 15 (2022). https://doi.org/10.1007/s10723-022-09603-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10723-022-09603-4

Keywords

Search

Navigation