Skip to main content
SpringerLink
Log in

Cost-efficient reactive scheduling for real-time workflows in clouds

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Workflow comprising of many tasks and data dependencies among tasks is an attractive programming paradigm for processing big data in clouds, and workflow scheduling plays essential roles in improving the cost and resource efficiency for cloud platforms. Up to now, large numbers of scheduling approaches have been proposed and improved. However, the majority of them focused on scheduling a single workflow and have not adequately exploited the idle time slots on resources to reduce the cost for executing workflow applications. To cover the above issue, we suggest to schedule tasks from different workflows in a hybrid way to take full advantage of idle time slots to improve the cost and resource efficiency, while guaranteeing the deadlines of workflows. To achieve the above idea, we first introduce a reactive scheduling architecture for real-time workflows. Then, a novel cost-efficient reactive scheduling algorithm (CERSA) is proposed to deploy multiple workflows with deadlines to cloud platforms. Finally, on the basis of real-world workflow traces, extensive experiments are conducted to compare CERSA with five existing algorithms. The experimental results demonstrate that CERSA is better than those algorithms with respect to monetary cost and resource efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://github.com/HuangkeChen/Uncertainty-Aware-Workflow-Scheduling.

  2. https://confluence.pegasus.isi.edu/display/pegasus/WorkflowGenerator.

References

  1. Mell P, Grance T (2011) The nist definition of cloud computing (draft). NIST Spec Publ 800:145

    Google Scholar 

  2. Armbrust M, Fox A, Griffith R, Joseph AD, Katz R, Konwinski A, Lee G, Patterson D, Rabkin A, Stoica I (2010) A view of cloud computing. Commun ACM 53(4):50–58

    Article  Google Scholar 

  3. Chen H, Zhu X, Guo H, Zhu J, Qin X, Wu J (2015) Towards energy-efficient scheduling for real-time tasks under uncertain cloud computing environment. J Syst Softw 99:20–35

    Article  Google Scholar 

  4. Sfrent A, Pop F (2015) Asymptotic scheduling for many task computing in big data platforms. Inf Sci 319:71–91

    Article  MathSciNet  Google Scholar 

  5. Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Future Gener Comput Syst 29(3):682–692

    Article  Google Scholar 

  6. Boutin E, Ekanayake J, Lin W, Shi B, Zhou J, Qian Z, Wu M, Zhou L (2014) Apollo: scalable and coordinated scheduling for cloud-scale computing. In: Proceedings of the 11th USENIX conference on operating systems design and implementation. USENIX Association, pp 285–300

  7. Dalman T, Wiechert W, Nöh K (2016) A scientific workflow framework for 13 c metabolic flux analysis. J Biotechnol 232:12–24

    Article  Google Scholar 

  8. Chen H, Zhu X, Liu G, Pedrycz W (2018) Uncertainty-aware online scheduling for real-time workflows in cloud service environment. IEEE Trans Serv Comput. https://doi.org/10.1109/TSC.2018.2866421

  9. Alkhanak EN, Lee SP, Rezaei R, Parizi RM (2016) Cost optimization approaches for scientific workflow scheduling in cloud and grid computing: a review, classifications, and open issues. J Syst Softw 113:1–26

    Article  Google Scholar 

  10. Chauhan MA, Babar MA, Benatallah B (2017) Architecting cloud-enabled systems: a systematic survey of challenges and solutions. Softw Pract Exp 47(4):599–644

    Google Scholar 

  11. Chen H, Zhu X, Qiu D, Liu L, Du Z (2017) Scheduling for workflows with security-sensitive intermediate data by selective tasks duplication in clouds. IEEE Trans Parallel Distrib Syst. https://doi.org/10.1109/TPDS.2017.2678507

    Article  Google Scholar 

  12. Zhu Z, Zhang G, Li M, Liu X (2016) Evolutionary multi-objective workflow scheduling in cloud. IEEE Trans Parallel Distrib Syst 27(5):1344–1357

    Article  Google Scholar 

  13. Calheiros RN, Buyya R (2014) Meeting deadlines of scientific workflows in public clouds with tasks replication. IEEE Trans Parallel Distrib Syst 25(7):1787–1796

    Article  Google Scholar 

  14. Lee YC, Han H, Zomaya AY, Yousif M (2015) Resource-efficient workflow scheduling in clouds. Knowl Based Syst 80:153–162

    Article  Google Scholar 

  15. Abrishami S, Naghibzadeh M, Epema DH (2013) Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds. Future Gener Comput Syst 29(1):158–169

    Article  Google Scholar 

  16. Meneguzzo DM, Liknes GC, Nelson MD (2013) Mapping trees outside forests using high-resolution aerial imagery: a comparison of pixel-and object-based classification approaches. Environ Monit Assess 185(8):6261–6275

    Article  Google Scholar 

  17. Zhu Z, Qi G, Chai Y, Li P (2017) A geometric dictionary learning based approach for fluorescence spectroscopy image fusion. Appl Sci 7(2):161

    Article  Google Scholar 

  18. Abduljabbar ZA, Jin H, Ibrahim A, Hussien ZA, Hussain MA, Abbdal SH, Zou D (2016) Sepim: secure and efficient private image matching. Appl Sci 6(8):213

    Article  Google Scholar 

  19. Topcuoglu H, Hariri S, Wu M-Y (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260–274

    Article  Google Scholar 

  20. Durillo JJ, Nae V, Prodan R (2014) Multi-objective energy-efficient workflow scheduling using list-based heuristics. Future Gener Comput Syst 36:221–236

    Article  Google Scholar 

  21. Li K, Tang X, Veeravalli B, Li K (2015) Scheduling precedence constrained stochastic tasks on heterogeneous cluster systems. IEEE Trans Comput 64(1):191–204

    Article  MathSciNet  Google Scholar 

  22. Abrishami S, Naghibzadeh M, Epema DH (2012) Cost-driven scheduling of grid workflows using partial critical paths. IEEE Trans Parallel Distrib Syst 23(8):1400–1414

    Article  Google Scholar 

  23. Poola D, Garg SK, Buyya R, Yang Y, Ramamohanarao K (2014) Robust scheduling of scientific workflows with deadline and budget constraints in clouds. In: Proceedings of the 28th International Conference on Advanced Information Networking and Applications (AINA). IEEE, pp 858–865

  24. Rodriguez MA, Buyya R (2014) Deadline based resource provisioningand scheduling algorithm for scientific workflows on clouds. IEEE Trans Cloud Comput 2(2):222–235

    Article  Google Scholar 

  25. Su H-Y, Hsu Y-L, Chen Y-C (2016) Pso-based voltage control strategy for loadability enhancement in smart power grids. Appl Sci 6(12):449

    Article  Google Scholar 

  26. Mezmaz M, Melab N, Kessaci Y, Lee YC, Talbi E-G, Zomaya AY, Tuyttens D (2011) A parallel bi-objective hybrid metaheuristic for energy-aware scheduling for cloud computing systems. J Parallel Distrib Comput 71(11):1497–1508

    Article  Google Scholar 

  27. Taheri J, Lee YC, Zomaya AY, Siegel HJ (2013) A bee colony based optimization approach for simultaneous job scheduling and data replication in grid environments. Comput Oper Res 40(6):1564–1578

    Article  MathSciNet  Google Scholar 

  28. Xu Y, Li K, Hu J, Li K (2014) A genetic algorithm for task scheduling on heterogeneous computing systems using multiple priority queues. Inf Sci 270:255–287

    Article  MathSciNet  Google Scholar 

  29. Jakob W, Strack S, Quinte A, Bengel G, Stucky K-U, Süß W (2013) Fast rescheduling of multiple workflows to constrained heterogeneous resources using multi-criteria memetic computing. Algorithms 6(2):245–277

    Article  Google Scholar 

  30. Yao G, Ding Y, Jin Y, Hao K (2017) Endocrine-based coevolutionary multi-swarm for multi-objective workflow scheduling in a cloud system. Soft Comput 21(15):4309–4322

    Article  Google Scholar 

  31. Mahmood A, Khan SA (2017) Hard real-time task scheduling in cloud computing using an adaptive genetic algorithm. Computers 6(2):15

    Article  Google Scholar 

  32. Arabnejad H, Barbosa JG (2017) Multi-qos constrained and profit-aware scheduling approach for concurrent workflows on heterogeneous systems. Future Gener Comput Syst 68:211–221

    Article  Google Scholar 

  33. Xie G, Liu L, Yang L, Li R (2016) Scheduling trade-off of dynamic multiple parallel workflows on heterogeneous distributed computing systems. Concurr Comput Pract Exp 29:1–18

    Google Scholar 

  34. Arabnejad H, Barbosa JG (2017) Maximizing the completion rate of concurrent scientific applications under time and budget constraints. J Comput Sci 23:120–129

    Article  MathSciNet  Google Scholar 

  35. Rimal BP, Maier M (2017) Workflow scheduling in multi-tenant cloud computing environments. IEEE Trans Parallel Distrib Syst 28(1):290–304

    Article  Google Scholar 

  36. Sharif S, Taheri J, Zomaya AY, Nepal S (2014) Online multiple workflow scheduling under privacy and deadline in hybrid cloud environment. In: Proceedings of the IEEE International Conference on Cloud Computing Technology and Science, pp 455–462

  37. Sîrbu A, Pop C, Şerbănescu C, Pop F (2016) Predicting provisioning and booting times in a metal-as-a-service system. Future Gener Comput Syst. https://doi.org/10.1016/j.future.2016.07.001

    Article  Google Scholar 

Download references

Acknowledgements

This research is supported by the National Natural Science Foundation of China under Grants (No. 61572511 and 61603404) and the Scientific Research Project of National University of Defense Technology under Grants (No. ZK16-03-09 and ZK16-03-30).

Author information

Authors and Affiliations

  1. College of Systems Engineering, National University of Defense Technology, Changsha, 410073, Hunan, People’s Republic of China

    Huangke Chen, Jianghan Zhu & Lisu Huo

  2. School of Traffic and Transportation Engineering, Central South University, Changsha, 410073, Hunan, People’s Republic of China

    Guohua Wu

Authors
  1. Huangke Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianghan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guohua Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lisu Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guohua Wu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, H., Zhu, J., Wu, G. et al. Cost-efficient reactive scheduling for real-time workflows in clouds. J Supercomput 74, 6291–6309 (2018). https://doi.org/10.1007/s11227-018-2561-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-018-2561-9

Keywords

Search

Navigation