Advertisement

A multi-objective trade-off framework for cloud resource scheduling based on the Deep Q-network algorithm

Abstract

Cloud computing, now a mature computing service, provides an efficient and economical solution for processing big data. As such, it attracts a lot of attention in academia and plays an important role in industrial applications. With the recent increase in the scale of cloud computing data centers, and improvements in user service quality requirements, the structure of the whole cloud system has become more complex, which has also made the resource scheduling management of these systems more challenging. Thus, the goal of this research was to resolve the conflict between cloud service providers (CSPs) who aim to minimize energy costs and those who seek to optimize service quality. Based on the excellent environmental awareness and online adaptive decision-making ability of deep reinforcement learning (DRL), we proposed an online resource scheduling framework based on the Deep Q-network (DQN) algorithm. The framework could make a trade-off of the two optimization objectives of energy consumption and task makespan by adjusting the proportion of the reward of different optimization objectives. Experimental results showed that this framework could effectively be used to make a trade-off of the two optimization objectives of energy consumption and task makespan, and exhibited obvious optimization effects compared with the baseline algorithm. Therefore, our proposed framework can dynamically adjust the optimization objective of the system according to the different requirements of the cloud system.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Abbreviations

CSP:

Cloud service provider

DRL:

Deep reinforcement learning

RL:

Reinforcement learning

DQN:

Deep Q-network

QoS:

Quality of service

MoPSO:

Multi-objective particle swarm optimization

GA:

Genetic algorithm

ACO:

Ant colony optimization

MOO:

Multi-objective optimization

VM:

Virtual machine

AWT:

Average waiting time

SLA:

Server level agreement

ETE:

End-to-end

DAG:

Directed acyclic graphs

RR:

Round-robin

MEC:

Mobile edge computing

References

  1. 1.

    Lin, W.W., Qi, D.Y., et al.: Review of cloud computing resource scheduling. Comput. Sci. 39(10), 1–6 (2012)

  2. 2.

    Ramezani, F., Lu, J., Hussain, F.: Task scheduling optimization in cloud computing applying multi-objective particle swarm optimization. In: Service-Oriented Computing. ICSOC 2013. Lecture Notes in Computer Science, vol. 8274, pp. 237–251. Springer (2013)

  3. 3.

    Liu, J., Luo, X.G., Zhang, X.M., et al.: Job scheduling algorithm for cloud computing based on particle swarm optimization. Adv. Mater. Res. 662, 957–960 (2013)

  4. 4.

    Alkayal, E.S., Jennings, N.R., Abulkhair, M.F.: Efficient task scheduling multi-objective particle swarm optimization in cloud computing. In: 2016 IEEE 41st Conference on Local Computer Networks Workshops (LCN Workshops), pp. 17–24 (2016)

  5. 5.

    Verma, A., Kaushal, S.: A hybrid multi-objective particle swarm optimization for scientific workflow scheduling. Parallel Comput. 62, 1–19 (2017)

  6. 6.

    Agarwal, Mohit, Srivastava, G.M.S.: Genetic algorithm enabled particle swarm optimization (PSOGA) based task scheduling in cloud computing environment. Int. J. Inf. Technol. Decis. Mak. 17(3), 1237–1267 (2018)

  7. 7.

    Barrett, E., Howley, E., Duggan, J.: Applying reinforcement learning towards automating resource allocation and application scalability in the cloud. Concurr. Comput. 25(12), 1656–1674 (2013)

  8. 8.

    Farahnakian, F., Liljeberg, P., Plosila, J.: Energy-efficient virtual machines consolidation in cloud data centers using reinforcement learning. In: 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, pp. 500–507 (2014)

  9. 9.

    Cui, D., Ke, W., Peng, Z., Zuo, J.: Multiple DAGs workflow scheduling algorithm based on reinforcement learning in cloud computing. In: Computational Intelligence and Intelligent Systems. ISICA 2015. Communications in Computer and Information Science, vol. 575, pp. 305–311 (2016)

  10. 10.

    Sharma, A.R., Kaushik, P., et al.: Deep and reinforcement learning in natural language processing. In: Proceedings of the 2017 International Conference on Computing, Communication and Automation (ICCCA), pp. 350–354 (2017)

  11. 11.

    Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Playing atari with deep reinforcement learning. In: Proceedings of Workshops at the 26th Neural Information Processing Systems. Computer Science, pp. 201–220 (2013)

  12. 12.

    Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

  13. 13.

    Phaniteja, S., Dewangan P.: A deep reinforcement learning approach for dynamically stable inverse kinematics of humanoid robots. In: Proceedings of the 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 1818–1823 (2017)

  14. 14.

    Bitsakos, C., Konstantinou, I., et al.: DERP: A deep reinforcement learning cloud system for elastic resource provisioning. In: 2018 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), vol. 1, pp. 21–29 (2018)

  15. 15.

    Huang, L., Feng, X., Zhang, C., et al.: Deep reinforcement learning-based joint task offloading and bandwidth allocation for multi-user mobile edge computing. Digit. Commun. Netw. 5(1), 10–17 (2019)

  16. 16.

    Huang, L., Feng, X., Feng, A., et al.: Distributed deep learning-based offloading for mobile edge computing networks. Mobile Networks and Applications, pp. 1–8 (2018)

  17. 17.

    Huang, L., Feng, X., et al.: Multi-server multi-user multi-task computation offloading for mobile edge computing networks. Sensors 19(6), 1446 (2019)

  18. 18.

    Tsai, C.W., Huang, W.C., Chiang, M.H., et al.: A hyper-heuristic scheduling algorithm for cloud. IEEE Trans. Cloud Comput. 2(2), 236–250 (2014)

  19. 19.

    Salza, P., Ferrucci, F., Sarro, F.: Develop, deploy and execute parallel genetic algorithms in the cloud. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, ACM, pp. 121–122 (2016)

  20. 20.

    Li, H.H., Chen, Z.G., Zhan, Z.H., et al.: Renumber coevolutionary multiswarm particle swarm optimization for multi-objective workflow scheduling on cloud computing environment. In: Companion Publication of the Conference on Genetic & Evolutionary Computation, ACM, pp. 1419–1420 (2015)

  21. 21.

    Yu, Z., Fang, L.I., Tao, Z., et al.: Task scheduling algorithm based on genetic ant colony algorithm in cloud computing environment. Comput. Eng. Appl. 50(6), 51–55 (2014)

  22. 22.

    Zhang, H., Li, P., Zhou, Z., Yu, X.: A PSO-based hierarchical resource scheduling strategy on cloud computing. In: International Conference on Trustworthy Computing and Services, pp. 325–332. Springer (2013)

  23. 23.

    Xue, S., Li, M., Xu, X., Chen, J., Xue, S.: An ACO-LB algorithm for task scheduling in the cloud environment. J. Softw. 9(2), 466–473 (2014)

  24. 24.

    Gao, Y., Guan, H., Qi, Z., et al.: A multi-objective ant colony system algorithm for virtual machine placement in cloud computing. J. Comput. Syst. Sci. 79(8), 1230–1242 (2013)

  25. 25.

    Zuo, L., Shu, L., Dong, S., et al.: A multi-objective optimization scheduling method based on the ant colony algorithm in cloud computing. IEEE Access 3, 2687–2699 (2015)

  26. 26.

    Chen, S., Wu, J., Lu, Z.: A cloud computing resource scheduling policy based on genetic algorithm with multiple fitness. In: IEEE International Conference on Computer & Information Technology, pp. 177–184 (2012)

  27. 27.

    Duan, H., Chen, C., Min, G., Wu, Y.: Energy-aware scheduling of virtual machines in heterogeneous cloud computing systems. Future Gener. Comput. Syst. 74, 142–150 (2017)

  28. 28.

    Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013)

  29. 29.

    Peng, Z., Cui, D., Zuo, J., et al.: Random task scheduling scheme based on reinforcement learning in cloud computing. Clust. Comput. 18(4), 1595–1607 (2015)

  30. 30.

    Peng, Z., Cui, D., Zuo, J., et al.: Research on cloud computing resources provisioning based on reinforcement learning. Math. Probl. Eng. 9, 1–12 (2015)

  31. 31.

    Liu, J.Z., Zhang, Y.X., Zhou, Y.Z., et al.: Aggressive resource provisioning for ensuring QoS in virtualized environments. IEEE Trans. Cloud Comput. 3(2), 119–131 (2015)

  32. 32.

    Peng, Z., Cui, D., Ma, Y., et al.: A reinforcement learning-based mixed job scheduler scheme for cloud computing under SLA constraint. In: 2016 IEEE 3rd International Conference on Cyber Security and Cloud Computing (CSCloud), pp. 142–147 (2016)

  33. 33.

    Liu, Q., Zhan, J.W., Zhang, Z.Z., et al.: A survey on deep reinforcement learning. Chin. J. Comput. 41(1), 1–27 (2018)

  34. 34.

    Mao, H., Alizadeh, M., Menache, I., et al.: Resource management with deep reinforcement learning. In: ACM Workshop on Hot Topics in Networks. ACM, pp. 50–56 (2016)

  35. 35.

    Lin, J.P., Peng, Z.Z., Cui, D.D.: Deep reinforcement learning for multi-resource cloud job scheduling. In: Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science, Springer, vol. 11303, pp. 289–302 (2018)

  36. 36.

    Liu, N., Li, Z., Xu, Z., et al.: A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning. In: 37th IEEE International Conference on Distributed Computing (ICDCS 2017), Computer Science, pp. 34–56 (2017)

  37. 37.

    Cheng, M., Li, J., Nazarian, S.: Deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers. In: 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 129–134 (2018)

  38. 38.

    Quan, L., Wang, Z., Ren, F.: A novel two-layered reinforcement learning for task offloading with tradeoff between physical machine utilization rate and delay. Future Internet 10, 60 (2018)

  39. 39.

    Gao, Y., Wang, Y., Gupta, S.K., Pedram, M.: An energy and deadline aware resource provisioning, scheduling and optimization framework for cloud systems. In: Proceedings of the Ninth IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis, pp. 1–10. IEEE Press (2013)

  40. 40.

    Pedram, M.: Energy-efficient datacenters. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 31, 1465–1484 (2012)

  41. 41.

    Google cluster data. [Online]. https://github.com/google/cluster-data

Download references

Acknowledgements

The work presented in this paper was supported by National Natural Science Foundation of China (61772145, 61672174). Jianpeng Lin and Delong Cui are corresponding authors.

Author information

Correspondence to Jianpeng Lin or Delong Cui.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Peng, Z., Lin, J., Cui, D. et al. A multi-objective trade-off framework for cloud resource scheduling based on the Deep Q-network algorithm. Cluster Comput (2020). https://doi.org/10.1007/s10586-019-03042-9

Download citation

Keywords

  • Cloud computing
  • Resource scheduling
  • Multi-objective
  • Deep reinforcement learning