Skip to main content

Advertisement

SpringerLink
Log in

Virtual machine migration policy for multi-tier application in cloud computing based on Q-learning algorithm

  • Regular Paper
  • Published:
Computing Aims and scope Submit manuscript

Abstract

Cloud computing technology provides shared computing which can be accessed over the Internet. When cloud data centers are flooded by end-users, how to efficiently manage virtual machines to balance both economical cost and ensure QoS becomes a mandatory work to service providers. Virtual machine migration feature brings a plenty of benefits to stakeholders such as cost, energy, performance, stability, availability. However, stakeholders’ objectives are usually conflict with each other. Furthermore, the optimal resource allocation problem in cloud infrastructure is usually NP-Hard or NP-Complete class. In this paper, the virtual migration problem is formulated by applying the game theory to ensure both load balance and resource utilization. The virtual machine migration algorithm, named V2PQL, is proposed based on Markov decision process and Q-learning algorithm. The results of the simulation demonstrate the efficiency of our proposal which are divided into training phase and extraction phase. The proposed V2PQL algorithm has been benchmarked to the Round-Robin, inverse Ant System, Max–Min Ant System, and Ant System algorithms in order to highlight its strength and feasibility in extraction phase.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Bai WH, Xi JQ, Zhu JX, Huang SW (2015) Performance analysis of heterogeneous data centers in cloud computing using a complex queuing model. Math Probl Eng. https://doi.org/10.1155/2015/980945

    Article  MathSciNet  MATH  Google Scholar 

  2. Baker T, Mackay M, Randles M, Taleb-Bendiab A (2013) Intention-oriented programming support for runtime adaptive autonomic cloud-based applications. Comput Electr Eng 39(7):2400–2412. https://doi.org/10.1016/j.compeleceng.2013.04.019

    Article  Google Scholar 

  3. Bui KT, Ho HD, Pham TV, Tran HC (2020) Virtual machines migration game approach for multi-tier application in infrastructure as a service cloud computing. IET Netw 9(6):326–337. https://doi.org/10.1049/iet-net.2019.0204

    Article  Google Scholar 

  4. Bui KT, Nguyen LV, Tran TV, Pham TV, Tran HC (2021) A load balancing vms migration approach for multi-tier application in cloud computing based on fuzzy set and q-learning algorithm. In: Research in intelligent and computing in engineering. Springer, pp 617–628. https://doi.org/10.1007/978-981-15-7527-3_58

  5. Bui KT, Pham TV, Tran HC (2016) A load balancing game approach for vm provision cloud computing based on ant colony optimization. In: International conference on context-aware systems and applications. Springer, pp 52–63. https://doi.org/10.1007/978-3-319-56357-2_6

  6. Duong T, Chu YJ, Nguyen T, Chakareski J (2015) Virtual machine placement via q-learning with function approximation. In: 2015 IEEE global communications conference (GLOBECOM), pp 1–6. IEEE. https://doi.org/10.1109/GLOCOM.2015.7417491

  7. Farahnakian F, Liljeberg P, Plosila J (2014) Energy-efficient virtual machines consolidation in cloud data centers using reinforcement learning. In: 2014 22nd Euromicro international conference on parallel, distributed, and network-based processing. IEEE, pp 500–507. https://doi.org/10.1109/PDP.2014.109

  8. Ficco M, Esposito C, Palmieri F, Castiglione A (2018) A coral-reefs and game theory-based approach for optimizing elastic cloud resource allocation. Fut Gener Comput Syst 78:343–352. https://doi.org/10.1016/j.future.2016.05.025

    Article  Google Scholar 

  9. Fujiwara-Greve T (1989) Learning from delayed rewards, vol 1. King’s College, Cambridge

  10. Fujiwara-Greve T (2015) Non-cooperative game theory, vol 1. Springer, Berlin

    Book  Google Scholar 

  11. Gao Y, Guan H, Qi Z, Hou Y, Liu L (2013) A multi-objective ant colony system algorithm for virtual machine placement in cloud computing. J Comput Syst Sci 79(8):1230–1242. https://doi.org/10.1016/j.jcss.2013.02.004

    Article  MathSciNet  MATH  Google Scholar 

  12. Ghasemi A, Toroghi Haghighat A (2020) A multi-objective load balancing algorithm for virtual machine placement in cloud data centers based on machine learning. Computing 102:2049–2072. https://doi.org/10.1007/s00607-020-00813-w

    Article  MathSciNet  Google Scholar 

  13. Ghumman NS, Kaur R (2015) Dynamic combination of improved max-min and ant colony algorithm for load balancing in cloud system. In: 2015 6th International conference on computing, communication and networking technologies (ICCCNT). IEEE, pp 1–5. https://doi.org/10.1109/ICCCNT.2015.7395172

  14. Guo Y, Stolyar A, Walid A (2018) Online vm auto-scaling algorithms for application hosting in a cloud. IEEE Trans Cloud Comput. https://doi.org/10.1109/TCC.2018.2830793

    Article  Google Scholar 

  15. Hartmanis J (1982) Computers and intractability: a guide to the theory of np-completeness. SIAM Rev 24(1):90. https://doi.org/10.1137/1024022

    Article  Google Scholar 

  16. Hsieh SY, Liu CS, Buyya R, Zomaya AY (2020) Utilization-prediction-aware virtual machine consolidation approach for energy-efficient cloud data centers. J Parallel Distrib Comput 139:99–109. https://doi.org/10.1016/j.jpdc.2019.12.014

    Article  Google Scholar 

  17. Huang G, Wang S, Zhang M, Li Y, Qian Z, Chen Y, Zhang S (2016) Auto scaling virtual machines for web applications with queueing theory. In: 2016 3rd International conference on systems and informatics (ICSAI). IEEE, pp 433–438. https://doi.org/10.1109/ICSAI.2016.7810994

  18. Jamshidi P, Sharifloo AM, Pahl C, Metzger A, Estrada G (2015) Self-learning cloud controllers: fuzzy q-learning for knowledge evolution. In: 2015 International conference on cloud and autonomic computing. IEEE, pp 208–211. https://doi.org/10.1109/ICCAC.2015.35

  19. Levin E, Pieraccini R, Eckert W (1998) Using Markov decision process for learning dialogue strategies. In: Proceedings of the 1998 IEEE international conference on acoustics, speech and signal processing (ICASSP’98) (Cat. No. 98CH36181), vol 1. IEEE, pp 201–204. https://doi.org/10.1109/ICASSP.1998.674402

  20. Minarolli D, Freisleben B, (2011) Utility-based resource allocation for virtual machines in cloud computing. In: 2011 IEEE symposium on computers and communications (ISCC). IEEE, pp 410–417. https://doi.org/10.1109/ISCC.2011.5983872

  21. Morton T, Pentico DW (1993) Heuristic scheduling systems: with applications to production systems and project management, vol 3. Wiley

  22. Noshy M, Ibrahim A, Ali HA (2018) Optimization of live virtual machine migration in cloud computing: a survey and future directions. J Netw Comput Appl 110:1–10. https://doi.org/10.1016/j.jnca.2018.03.002

    Article  Google Scholar 

  23. Rolik O, Zharikov E, Koval A, Telenyk S (2018) Dynamie management of data center resources using reinforcement learning. In: 2018 14th International conference on advanced trends in radioelecrtronics, telecommunications and computer engineering (TCSET). IEEE, pp 237–244. https://doi.org/10.1109/TCSET.2018.8336194

  24. Rybina K, Schill A (2016) Estimating energy consumption during live migration of virtual machines. In: 2016 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom), pp 1–5. IEEE. https://doi.org/10.1109/BlackSeaCom.2016.7901567

  25. Sahni J, Vidyarthi DP (2017) Heterogeneity-aware adaptive auto-scaling heuristic for improved qos and resource usage in cloud environments. Computing 99(4):351–381. https://doi.org/10.1007/s00607-016-0530-9

    Article  MathSciNet  Google Scholar 

  26. Saovapakhiran B, Michailidis G, Devetsikiotis M, (2011) Aggregated-dag scheduling for job flow maximization in heterogeneous cloud computing. In: 2011 IEEE global telecommunications conference-GLOBECOM 2011. IEEE, pp 1–6. https://doi.org/10.1109/GLOCOM.2011.6133611

  27. Siar H, Kiani K, Chronopoulos AT (2015) An effective game theoretic static load balancing applied to distributed computing. Clust Comput 18(4):1609–1623. https://doi.org/10.1007/s10586-015-0486-0

    Article  Google Scholar 

  28. Tsai CW, Rodrigues JJ (2013) Metaheuristic scheduling for cloud: a survey. IEEE Syst J 8(1):279–291. https://doi.org/10.1109/JSYST.2013.2256731

    Article  Google Scholar 

  29. Van Laarhoven PJ, Aarts EH, Lenstra JK (1992) Job shop scheduling by simulated annealing. Oper Res 40(1):113–125. https://doi.org/10.1287/opre.40.1.113

    Article  MathSciNet  MATH  Google Scholar 

  30. Van Otterlo M, Wiering M (2012) Reinforcement learning and markov decision processes. In: Reinforcement learning. Springer, pp 3–42. https://doi.org/10.1007/978-3-642-27645-3_1

  31. Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 8(3–4):279–292

    MATH  Google Scholar 

  32. Xiao Z, Song W, Chen Q (2012) Dynamic resource allocation using virtual machines for cloud computing environment. IEEE Trans Parallel Distrib Syst 24(6):1107–1117. https://doi.org/10.1109/TPDS.2012.283

    Article  Google Scholar 

  33. Xu X, Yu H (2014) A game theory approach to fair and efficient resource allocation in cloud computing. Math Probl Eng. https://doi.org/10.1155/2014/915878

    Article  MathSciNet  MATH  Google Scholar 

  34. Yang L, Feng Y, Li K (2017) Optimization of virtual resources provisioning for cloud applications to cope with traffic burst. In: 2017 IEEE international symposium on parallel and distributed processing with applications and 2017 IEEE international conference on ubiquitous computing and communications (ISPA/IUCC). IEEE, pp 80–87. https://doi.org/10.1109/ISPA/IUCC.2017.00021

  35. Ye D, Chen J (2013) Non-cooperative games on multidimensional resource allocation. Fut Gener Comput Syst 29(6):1345–1352. https://doi.org/10.1016/j.future.2013.02.004

    Article  Google Scholar 

  36. Zhang Q, Cheng L, Boutaba R (2010) Cloud computing: state-of-the-art and research challenges. J Internet Serv Appl 1(1):7–18. https://doi.org/10.1007/s13174-010-0007-6

    Article  Google Scholar 

Download references

Acknowledgements

This research is funded by Thu Dau Mot University under Grant Number DT.21.1-080. We would like to thank Ho Chi Minh City University of Technology (HCMUT), VNU-HCM for the support of time and facilities for this study.

Author information

Authors and Affiliations

  1. Training and Science Technology Department, Posts and Telecommunications Institute of Technology, 11 Nguyen Dinh Chieu, Ho Chi Minh, Vietnam

    Cong Hung Tran

  2. Institute of Engineering and Technology, Thu Dau Mot University, 06 Tran Van On Street, Thu Dau Mot, Binh Duong, Vietnam

    Thanh Khiet Bui

  3. Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), 268 Ly Thuong Kiet Street, District 10, Ho Chi Minh, Vietnam

    Thanh Khiet Bui & Tran Vu Pham

  4. Vietnam National University Ho Chi Minh City, Linh Trung Ward, Thu Duc District, Ho Chi Minh, Vietnam

    Thanh Khiet Bui & Tran Vu Pham

Authors
  1. Cong Hung Tran
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thanh Khiet Bui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tran Vu Pham
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thanh Khiet Bui.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tran, C.H., Bui, T.K. & Pham, T.V. Virtual machine migration policy for multi-tier application in cloud computing based on Q-learning algorithm. Computing 104, 1285–1306 (2022). https://doi.org/10.1007/s00607-021-01047-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-021-01047-0

Keywords

Mathematics Subject Classification

Search

Navigation