Skip to main content

Advertisement

SpringerLink
Log in

Bi-objective cloud resource management for dependent tasks using Q-learning and NSGA-3

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Efficient management of cloud resources for more resource utilization on the one hand, and reducing the makespan on the other hand, has always been an important research issue in cloud environment. Since the proper allocation of cloud resources is an optimization problem, the learning-based and population-based meta-heuristic methods are used for this purpose. Most learning methods have scalability problem and may not converge to the optimal solution as the problem space becomes larger, and population-based optimization methods usually need a lot of iterations. The proposed method of this paper aims to overcome the limitations of these two techniques and uses their advantages to increase cloud resource utilization and improve the execution time. A Pareto-based algorithm is also exploited to solve this bi-objective problems. Since increasing the size of population leads to increase the convergence time, a proper learning-based selecting method of population is also utilized to reduce the number of iterations. The proposed method of this paper, called multi-agent bi-objective cloud resource management for dependent tasks using Q-learning and NSGA-3 (BCRN), employs an improved version of Q-learning to reduce the makespan and enhance resource utilization. To overcome the scalability problem of Q-learning, the number of states and actions are reduced in the BCRN, which reduces the complexity of the learning process and leads to better convergence time of the learning process. Two learning agents are also utilized in the BCRN, each of which seeks to improve the objectives of the problem. The NSGA-3 (non-deterministic sorting genetic algorithm-3) algorithm is used as to address the bi-objective problem. In the NSGA-3 algorithm, the initial population is often generated randomly, which leads to more convergence time of the learning process. Using a modified bi-objective Q-learning model, the initial population of the BCRN is generated considering the both objectives of the method to reduce the convergence time of the NSGA-3 algorithm. The long-term learning process attempts to find the best mapping between resources and tasks, and avoid local optima. The results of empirical experiments indicate that the proposed method reduces the makespan, and causes more efficient use of cloud resources.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability statement

My manuscript has no associated data.

References

  • Abd Elaziz M, Attiya I (2021) An improved Henry gas solubility optimization algorithm for task scheduling in cloud computing. Artif Intell Rev 54(5):3599–3637

    Article  Google Scholar 

  • Asghari A, Sohrabi MK (2021) Combined use of coral reefs optimization and reinforcement learning for improving resource utilization and load balancing in cloud environments. Computing 103(7):1545–1567

    Article  Google Scholar 

  • Asghari A, Sohrabi MK (2022) Combined use of coral reefs optimization and multi-agent deep Q-network for energy-aware resource provisioning in cloud data centers using DVFS technique. Clust Comput 25(1):119–140

    Article  Google Scholar 

  • Asghari A, Sohrabi MK, Yaghmaee F (2020a) A Online scheduling of dependent tasks of cloud’s workflows to enhance resource utilization and reduce the makespan using multiple reinforcement learning-based agents. Soft Comput 24(21):16177–16199

    Article  Google Scholar 

  • Asghari A, Sohrabi MK, Yaghmaee F (2020b) A cloud resource management framework for multiple online scientific workflows using cooperative reinforcement learning agents. Comput Netw 179:107340

    Article  Google Scholar 

  • Asghari A, Sohrabi MK, Yaghmaee F (2021) Task scheduling, resource provisioning, and load balancing on scientific workflows using parallel SARSA reinforcement learning agents and genetic algorithm. J Supercomput 77(3):2800–2828

    Article  Google Scholar 

  • Babukarthik RG, Raju R, Dhavachelvan P (2013) Hybrid algorithm for job scheduling: combining the benefits of ACO and Cuckoo search. Advances in computing and information technology. Springer, Berlin, Heidelberg, pp 479–490

    Chapter  Google Scholar 

  • Barker A, Hemert JV (2007) Scientific workflow: a survey and research directions. In: International Conference on Parallel Processing and Applied Mathematics, pp 746–753

  • Berriman GB, Deelman E, Good JC, Jacob JC, Katz DS, Kesselman C, Laity AC, Prince TA, Singh G, Su MH (2004) Montage: a grid-enabled engine for delivering custom science-grade mosaics on demand. Optim Sci Return Astron through Inform Technol 5493:221–232

    ADS  Google Scholar 

  • Calheiros RN, Ranjan R, Beloglazov A, De Rose CA, Buyya R (2011) CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw Pract Exp 41(1):23–50

    Article  Google Scholar 

  • Chen J, Xiao Z, Xing H, Dai P, Luo S, Iqbal M A (2020a) STDPG: a spatio-temporal deterministic policy gradient agent for dynamic routing in SDN. ICC 2020a–2020a. IEEE International Conference on Communications (ICC), pp 1–6. IEEE

  • Chen X, Wang H, Ma Y, Zheng X, Guo L (2020b) Self-adaptive resource allocation for cloud-based software services based on iterative QoS prediction model. Futur Gener Comput Syst 105:287–296

    Article  Google Scholar 

  • Chen J, Xing H, Xiao Z, Xu L, Tao T (2021) A DRL agent for jointly optimizing computation offloading and resource allocation in MEC. IEEE Internet Things J 8(24):7508–17524

    Article  Google Scholar 

  • Coello CAC, Lamont GB, Van Veldhuizen DA (2007) Evolutionary algorithms for solving multi-objective problems, vol 5. Springer, New York, pp 79–104

    Google Scholar 

  • Cui D, Peng Z, Xiong J, Xu B, Lin W (2017) A reinforcement learning-based mixed job scheduler scheme for grid or IaaS cloud. IEEE Trans Cloud Comput 8(4):1030–1039

    Article  Google Scholar 

  • de Oliveira LL, de Oliveira PS, Tinós R (2015) A multiobjective approach to the genetic code adaptability problem. BMC Bioinform 16(1):1–20

    Article  Google Scholar 

  • Deb K, Jain H (2013) An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part I: solving problems with box constraints. IEEE Trans Evol Comput 18(4):577–601

    Article  Google Scholar 

  • Deb K, Agrawal S, Pratap A, Meyarivan T (2000) A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II. In: International conference on parallel problem solving from nature. Springer, Berlin, Heidelberg, pp 849–858

  • Ding D, Fan X, Zhao Y, Kang K, Yin Q, Zeng J (2020) Q-learning based dynamic task scheduling for energy-efficient cloud computing. Futur Gener Comput Syst 108:361–371

    Article  Google Scholar 

  • Dordaie N, Navimipour NJ (2018) A hybrid particle swarm optimization and hill climbing algorithm for task scheduling in the cloud environments. ICT Express 4(4):199–202

    Article  Google Scholar 

  • Duggan M, Flesk K, Duggan J, Howley E, Barrett E (2016) A reinforcement learning approach for dynamic selection of virtual machines in cloud data centres. In: 2016 Sixth International Conference on Innovative Computing Technology (INTECH). IEEE, pp 92–97

  • Duggan M, Duggan J, Howley E, Barrett E (2017) A reinforcement learning approach for the scheduling of live migration from under utilised hosts. Memet Comput 9(4):283–293

    Article  Google Scholar 

  • Durgadevi P, Srinivasan S (2020) Resource allocation in cloud computing using SFLA and cuckoo search hybridization. Int J Parallel Prog 48(3):549–565

    Article  Google Scholar 

  • Fard HM, Prodan R, Fahringer T (2014) Multi-objective list scheduling of workflow applications in distributed computing infrastructures. J Parallel Distrib Comput 74(3):2152–2165

    Article  Google Scholar 

  • Fatema K, Emeakaroha VC, Healy PD, Morrison JP, Lynn T (2014) A survey of cloud monitoring tools: taxonomy, capabilities and objectives. J Parallel Distrib Comput 74(10):2918–2933

    Article  Google Scholar 

  • Ga̧sior J, Seredyński F (2013) Multi-objective parallel machines scheduling for fault-tolerant cloud systems. In: International Conference on Algorithms and Architectures for Parallel Processing. Springer, Cham, pp 247–256

  • Geetha P, Robin CR (2021) Power conserving resource allocation scheme with improved QoS to promote green cloud computing. J Ambient Intell Humaniz Comput 12(7):7153–7164

    Article  Google Scholar 

  • George S (2015) Truthful workflow scheduling in cloud computing using hybrid PSO-ACO. In: 2015 International Conference on Developments of E-Systems Engineering (DeSE). IEEE, pp 60–64

  • Ghani JA, Choudhury IA, Hassan HH (2004) Application of Taguchi method in the optimization of end milling parameters. J Mater Process Technol 145(1):84–92

    Article  CAS  Google Scholar 

  • Ghobaei-Arani M, Jabbehdari S, Pourmina MA (2018) An autonomic resource provisioning approach for service-based cloud applications: a hybrid approach. Future Gener Comput Syst 78:191–210

    Article  Google Scholar 

  • Grandinetti L, Pisacane O, Sheikhalishahi M (2013) An approximate ϵ-constraint method for a multi-objective job scheduling in the cloud. Future Gener Comput Syst 29(8):1901–1908

    Article  Google Scholar 

  • Guo M, Guan Q, Ke W (2018) Optimal scheduling of VMs in queueing cloud computing systems with a heterogeneous workload. IEEE Access 6:15178–15191

    Article  Google Scholar 

  • Hashem IAT, Anuar NB, Marjani M, Gani A, Sangaiah AK, Sakariyah AK (2018) Multi-objective scheduling of MapReduce jobs in big data processing. Multimed Tools Appl 77(8):9979–9994

    Article  Google Scholar 

  • He H, Xu G, Pang S, Zhao Z (2016) AMTS: adaptive multi-objective task scheduling strategy in cloud computing. China Commun 13(4):162–171

    Article  Google Scholar 

  • Ismayilov G, Topcuoglu HR (2020) Neural network based multi-objective evolutionary algorithm for dynamic workflow scheduling in cloud computing. Future Gener Comput Syst 102:307–322

    Article  Google Scholar 

  • Jena RK (2015) Multi objective task scheduling in cloud environment using nested PSO framework. Procedia Comput Sci 57:1219–1227

    Article  Google Scholar 

  • Jones D, Tamiz M (2010) Practical goal programming, vol 141. Springer, New York

    Google Scholar 

  • Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Future Gener Comput Syst 29(3):682–692

    Article  Google Scholar 

  • Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4:237–285

    Article  Google Scholar 

  • Kiresuk TJ, Smith A, Cardillo JE (2014) Goal attainment scaling: applications, theory, and measurement. Psychology Press

    Book  Google Scholar 

  • Labidi T, Mtibaa A, Gaaloul W, Tata S, Gargouri F (2017) Cloud SLA modeling and monitoring. In: 2017 IEEE International Conference on Services Computing (SCC), pp 338–345

  • Lakra AV, Yadav DK (2015) Multi-objective tasks scheduling algorithm for cloud computing throughput optimization. Procedia Comput Sci 48:107–113

    Article  Google Scholar 

  • Laumanns M, Thiele L, Zitzler E (2006) An efficient, adaptive parameter variation scheme for metaheuristics based on the epsilon-constraint method. Eur J Oper Res 169(3):932–942

    Article  MathSciNet  Google Scholar 

  • Li JQ, Han YQ (2020) A hybrid multi-objective artificial bee colony algorithm for flexible task scheduling problems in cloud computing system. Clust Comput 23(4):2483–2499

    Article  Google Scholar 

  • Liu J, Pacitt E, Valduriez P, De Oliveira D, Mattoso M (2016) Multi-objective scheduling of scientific workflows in multisite clouds. Future Gener Comput Syst 63:76–95

    Article  Google Scholar 

  • Livny J, Teonadi H, Livny M, Waldor MK (2008) High-throughput, kingdom-wide prediction and annotation of bacterial non-coding RNAs. PLoS ONE 3(9):e3197

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  • Maechling P, Deelman E, Zhao L, Graves R, Mehta G, Gupta N, Mehringer J, Kesselman C, Callaghan S, Okaya D, Francoeur H, Gupta V, Cui Y, Vahi K, Jordan T, Field E (2007) SCEC CyberShake workflows—automating probabilistic seismic hazard analysis calculations. Workflows for e-Science. Springer, London, pp 143–163

    Chapter  Google Scholar 

  • Marler RT, Arora JS (2004) Survey of multi-objective optimization methods for engineering. Struct Multidiscip Optim 26(6):369–395

    Article  MathSciNet  Google Scholar 

  • Mateos C, Pacini E, Garino CG (2013) An ACO-inspired algorithm for minimizing weighted flowtime in cloud-based parameter sweep experiments. Adv Eng Softw 56:38–50

    Article  Google Scholar 

  • Orhean AI, Pop F, Raicu I (2018) New scheduling approach using reinforcement learning for heterogeneous distributed systems. J Parallel Distrib Comput 117:292–302

    Article  Google Scholar 

  • Pegasus WorkFlow Generator: http://www.confluence.pegasus.isi.edu/display/pegasus/WorkflowGenerator, Access date: 2020

  • Peng Z, Cui D, Zuo J, Li Q, Xu B, Lin W (2015) Random task scheduling scheme based on reinforcement learning in cloud computing. Clust Comput 18(4):1595–1607

    Article  Google Scholar 

  • Praveenchandar J, Tamilarasi A (2021) Dynamic resource allocation with optimized task scheduling and improved power management in cloud computing. J Ambient Intell Humaniz Comput 12(3):4147–4159

    Article  Google Scholar 

  • Qiu C, Cui S, Yao H, Xu F, Yu FR, Zhao C (2019) A novel QoS-enabled load scheduling algorithm based on reinforcement learning in software-defined energy internet. Future Gener Comput Syst 92:43–51

    Article  Google Scholar 

  • Rajagopalan A, Modale DR, Senthilkumar R (2020) Optimal scheduling of tasks in cloud computing using hybrid firefly-genetic algorithm. Advances in decision sciences, image processing, security and computer vision. Springer, Cham, pp 678–687

    Chapter  Google Scholar 

  • Raju R, Amudhavel J, Kannan N, Monisha M (2014) A bio inspired Energy-Aware Multi objective Chiropteran Algorithm (EAMOCA) for hybrid cloud computing environment. In: 2014 International conference on green computing communication and electrical engineering (ICGCCEE). IEEE, pp 1–5

  • Saeedi S, Khorsand R, Bidgoli SG, Ramezanpour M (2020) Improved many-objective particle swarm optimization algorithm for scientific workflow scheduling in cloud computing. Comput Ind Eng 147:106649

    Article  Google Scholar 

  • Shabeera TP, Kumar SM, Salam SM, Krishnan KM (2017) Optimizing VM allocation and data placement for data-intensive applications in cloud using ACO metaheuristic algorithm. Eng Sci Technol 20(2):616–628

    Google Scholar 

  • Shaw R, Howley E, Barrett E (2017) An advanced reinforcement learning approach for energy-aware virtual machine consolidation in cloud data centers. In: 2017 12th International Conference for Internet Technology and Secured Transactions (ICITST). IEEE, pp 61–66

  • Song F, Xing H, Luo S, Zhan D, Dai P, Qu R (2020) A multiobjective computation offloading algorithm for mobile-edge computing. IEEE Internet Things J 7(9):8780–8799

    Article  Google Scholar 

  • Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT Press

    Google Scholar 

  • Tanha M, Hosseini Shirvani M, Rahmani AM (2021) A hybrid meta-heuristic task scheduling algorithm based on genetic and thermodynamic simulated annealing algorithms in cloud computing environments. Neural Comput Appl 33(24):16951–16984

    Article  Google Scholar 

  • Tchernykh A, Lozano L, Schwiegelshohn U, Bouvry P, Pecero JE, Nesmachnow S, Drozdov AY (2016) Online bi-objective scheduling for IaaS clouds ensuring quality of service. J Grid Comput 14(1):5–22

    Article  Google Scholar 

  • Teich J (2001) Pareto-front exploration with uncertain objectives. In: International Conference on Evolutionary Multi-Criterion Optimization. Springer, Berlin, Heidelberg, pp 314–328

  • Teylo L, de Paula U, Frota Y, de Oliveira D, Drummond LM (2017) A hybrid evolutionary algorithm for task scheduling and data assignment of data-intensive scientific workflows on clouds. Future Gener Comput Syst 76:1–17

    Article  Google Scholar 

  • Tong Z, Chen H, Deng X, Li K, Li K (2020) A scheduling scheme in the cloud computing environment using deep Q-learning. Inf Sci 512:1170–1191

    Article  Google Scholar 

  • Wang WX, Wang X, Ge XL, Deng L (2014) Multi-objective optimization model for multi-project scheduling on critical chain. Adv Eng Softw 68:33–39

    Article  Google Scholar 

  • Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 8(3):279–292

    Article  Google Scholar 

  • Xu B, Peng Z, Xiao F, Gates AM, Yu JP (2015) Dynamic deployment of virtual machines in cloud computing using multi-objective optimization. Soft Comput 19(8):2265–2273

    Article  Google Scholar 

  • Yao G, Ding Y, Jin Y, Hao K (2017) Endocrine-based coevolutionary multi-swarm for multi-objective workflow scheduling in a cloud system. Soft Comput 21(15):4309–4322

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Semnan Branch, Islamic Azad University, Semnan, Iran

    Ali Asghari & Mohammad Karim Sohrabi

Authors
  1. Ali Asghari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Karim Sohrabi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad Karim Sohrabi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Asghari, A., Sohrabi, M.K. Bi-objective cloud resource management for dependent tasks using Q-learning and NSGA-3. J Ambient Intell Human Comput 15, 197–217 (2024). https://doi.org/10.1007/s12652-022-03885-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-022-03885-y

Keywords

Search

Navigation