Skip to main content
SpringerLink
Log in

DDBWS: a dynamic deadline and budget-aware workflow scheduling algorithm in workflow-as-a-service environments

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Workflow scheduling has been excessively studied in different environments like clusters, grids, and clouds. Cloud is a scalable, cost-effective environment that allows users to access an unlimited amount of resources and offers a pay-as-you-go model. An increase in the users’ desire to run their workflow applications on clouds leads to the development of multi-tenant environments like workflow-as-a-service platforms (WaaS). By leveraging cloud features, WaaS offers an environment where users can submit their workflows for execution with different quality of service (QoS) attributes at different. The problem of finding an appropriate scheduling algorithm considering factors like resource heterogeneity and QoS requirements is an NP-complete problem. Most of the existing algorithms in the literature are designed to schedule a single instance of a workflow or have a static behavior. Using static scheduling in dynamic environments like WaaS can lead to a low planning success rate. Besides, while it is possible to share resources between users, for simplicity purposes a majority of proposed algorithms schedule at most one task on a computing resource at any given point in time. They also consider either the time or cost as a hard constraint during scheduling. To cover these limitations in this study, we propose DDBWS, a Dynamic, Deadline and Budget-aware, Workflow Scheduling algorithm that is designed specifically for the WaaS environments. DDBWS schedules workflows by solving a multi-resource packing problem. Unlike several existing algorithms, it considers both CPU and memory demands for tasks simultaneously. Also, it leverages containers to run multiple tasks on a VM concurrently. It uses a bi-factor to control the tradeoff between cost and resource utilization during the mapping of tasks to resources. Based on real-world workflow traces, we have generated 6 different datasets of synthetic workflows. To compare the performance of the proposed scheduling algorithm, we chose two of the state-of-the-art dynamic concurrent workflow scheduling algorithms called EPSM and MW-HBDCS. We have conducted several experiments on these datasets. The results of the performed experiments show that DDBWS achieves at least 96% planning success rate on 6 different workloads which is a comparable PSR. The proposed algorithm decreases the total leased VM numbers considerably. It also outperforms its rivals in terms of the total execution cost and decreases the overall execution cost by at least 8.03% and on average 32.08%. The 95% confidence interval for this decrease is 32.08 ± 14.1 based on 12 samples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Ullman JD (1975) NP-complete scheduling problems. J Comput Syst Sci 10(3):384–393

    Article  MathSciNet  Google Scholar 

  2. Deldari A, Naghibzadeh M, Abrishami S (2017) CCA: a deadline-constrained workflow scheduling algorithm for multicore resources on the cloud. J Supercomput 73(2):756–781

    Article  Google Scholar 

  3. Sahni J, Vidyarthi DP (2015) A cost-effective deadline-constrained dynamic scheduling algorithm for scientific workflows in a cloud environment. IEEE Trans Cloud Comput 6(2):2–18

    Google Scholar 

  4. Alejandra Rodriguez M, Buyya R (2017) Budget-driven scheduling of scientific workflows in IaaS clouds with fine-grained billing periods. ACM Trans Auton Adapt Syst (TAAS) 12(2):1–22

    Article  Google Scholar 

  5. Wu Q, Ishikawa F, Zhu Q, Xia Y, Wen J (2017) Deadline-constrained Cost optimization approaches for workflow scheduling in clouds. IEEE Trans Parallel Distrib Syst 28(12):3401–3412

    Article  Google Scholar 

  6. Caniou Y, Caron E, Kong Win Chang A, Robert Y (2018) Budget-aware scheduling algorithms for scientific workflows with stochastic task weights on heterogeneous IaaS Cloud platforms. In: Proceedings of the 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Vancouver, BC, Canada, pp 15–26

  7. Ghasemzadeh M, Arabnejad H, Barbosa JG (2017) Deadline-budget constrained scheduling algorithm for scientific workflows in a cloud environment. In: .

  8. Arabnejad H, Barbosa JG, Prodan R (2016) Low-time complexity budget–deadline constrained workflow scheduling on heterogeneous resources. Future Gener Comput Syst 55:29–40

    Article  Google Scholar 

  9. Zheng W, Sakellariou R (2013) Budget-deadline constrained workflow planning for admission control. J Grid Comput 11(4):633–651

    Article  Google Scholar 

  10. Asghari A, Sohrabi MK, Yaghmaee F (2020) Online scheduling of dependent tasks of cloud’s workflows to enhance resource utilization and reduce the makespan using multiple reinforcement learning-based agents. Soft Comput 24(21):16177–16199

    Article  Google Scholar 

  11. Hu Y, Laat CD, Zhao Z (2019) Multi-objective container deployment on heterogeneous clusters. In: Proceedings of the 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), Larnaca, Cyprus, pp 592–599

  12. Wang Y, Liu H, Zheng W, Xia Y, Li Y, Chen P, Guo K, Xie H (2019) Multi-objective workflow scheduling with deep-Q-network-based multi-agent reinforcement learning. IEEE Access 7:39974–39982

    Article  Google Scholar 

  13. Zheng W, Yan W, Bugingo E, Zhang D (2018) Online scheduling to maximize resource utilization of deadline-constrained workflows on the cloud. In: Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design ((CSCWD), Nanjing, China, pp 98–103A

  14. Li Z, Ge J, Hu H, Song W, Hu H, Luo B (2018) Cost and energy aware scheduling algorithm for scientific workflows with deadline constraint in clouds. IEEE Trans Serv Comput 11(4):713–726

    Article  Google Scholar 

  15. Chen H, Zhu X, Liu G, Pedrycz W (2018) Uncertainty-aware online scheduling for real-time workflows in cloud service environment. IEEE Trans Serv Comput (Early Access ), pp 1–13

  16. Chen H, Zhu J, Wu G, Huo L (2018) Cost-efficient reactive scheduling for real-time workflows in clouds. J Supercomput 74(11):6291–6309

    Article  Google Scholar 

  17. Liu J, Ren J, Dai W, Zhang D, Zhou P, Zhang Y, Min G, Najjari N (2019) Online multi-workflow scheduling under uncertain task execution time in IaaS clouds. IEEE Trans Cloud Comput ( Early Access ), pp 1–15

  18. Malawski M, Juve G, Deelman E, Nabrzyski J (2015) Algorithms for cost-and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds. J Internet Serv Appl 48:1–18

    Google Scholar 

  19. Rodriguez MA, Buyya R (2018) Scheduling dynamic workloads in multi-tenant scientific workflow as a service platforms. Future GenerComput Syst 79:739–750

    Article  Google Scholar 

  20. Zhou N, Li F, Xu K, Qi D (2018) Concurrent workflow budget- and deadline-constrained scheduling in heterogeneous distributed environments. Soft Comput 22(23):7705–7718

    Article  Google Scholar 

  21. Hu Y, Laat CD, Zhao Z (2019) Learning workflow scheduling on multi-resource clusters. In: Proceedings of the 2019 IEEE International Conference on Networking, Architecture and Storage (NAS), EnShi, China, pp 1–8

  22. Zhu Z, Tang Z (2019) Deadline-constrained workflow scheduling in IaaS clouds with multi-resource packing. Future Gener Comput Syst 101:880–893

    Article  Google Scholar 

  23. Rubab S, Hassan MF, Mahmood AK, Mehmood Shah SN (2019) QoS based multi constraints bin packing job scheduling heuristic for heterogeneous volunteer grid resources. Int Arab J Inf Technol 16(4):661–668

    Google Scholar 

  24. Topcuoglu H, Hariri S, Wu M-Y (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260–274

    Article  Google Scholar 

  25. Verma A, Kaushal S (2015) Cost-time efficient scheduling plan for executing workflows in the cloud. J Grid Comput 13(4):495–506

    Article  MathSciNet  Google Scholar 

  26. Meng S, Wang S, Wu T, Li D, Huang T, Wu X, Xu X, Dou W (2016) An uncertainty-aware evolutionary scheduling method for cloud service provisioning. In: Proceedings of the 2016 IEEE International Conference on Web Services (ICWS), San Francisco, CA, USA, pp 506–5013

  27. Anwar N, Deng H (2018) Elastic scheduling of scientific workflows under deadline constraints in cloud computing environments. Futur Internet 10(5):5

    Article  Google Scholar 

  28. Poola D, Garg SK, Buyya R, Yang Y, Ramamohanarao K (2014) Robust scheduling of scientific workflows with deadline and budget constraints in clouds. In: Proceedings of the 2014 IEEE 28th International Conference on Advanced Information Networking and Applications, Victoria, BC, Canada, pp 858–865

  29. Yana H, Zhuab X, Chena H, Guoc H, Zhoua W, Baoa W (2019) DEFT: dynamic fault-tolerant elastic scheduling for tasks with uncertain runtime in cloud. Inf Sci 477:30–46

    Article  Google Scholar 

  30. Verma A, Kaushal S (2017) A hybrid multi-objective particle swarm optimization for scientific workflow scheduling. Parallel Comput 62:1–19

    Article  MathSciNet  Google Scholar 

  31. Arabnejad V, Bubendorfer K, Ng B (2019) Dynamic multi-workflow scheduling: a deadline and cost-aware approach for commercial clouds. Futur Gener Comput Syst 100:98–108

    Article  Google Scholar 

  32. Deelman E, Vahi K, Juve G, Rynge M, Callaghan S, Maechling PJ, Mayani R, Chen W, da Silva RF, Livny M, Wenger K (2015) Pegasus, a workflow management system for science automation. Futur Gener Comput Syst 46:17–35

    Article  Google Scholar 

  33. Arabnejad H, Barbosa JG (2017) Maximizing the completion rate of concurrent scientific applications under time and budget constraints. J Comput Sci 23:120–129

    Article  MathSciNet  Google Scholar 

  34. Yu J, Buyya R (2006) Scheduling scientific workflow applications with deadline and budget constraints using genetic algorithms. Sci Program 14(3):217–230

    Google Scholar 

  35. J Empowering App Development for Developers | Docker. https://www.docker.com/. Accessed 29 Nov 2020

  36. Amazon EC2 Instance Types—Amazon Web Services. https://aws.amazon.com/ec2/instance-types/. Accessed 29 Nov 2019

  37. V. Arabnejad, K. Bubendorfer, B. Ng, "Deadline Distribution Strategies for Scientific Workflow Scheduling in Commercial Clouds," in Proceedings of the 9th International Conference on Utility and Cloud Computing, December-2016, Shanghai, China, pp. 70-78.9.

  38. Sharif S, Taheri J, Zomaya AY (2016) Online multiple workflow scheduling under privacy and deadline in hybrid cloud environment. In: Proceedings of the 2014 IEEE 6th International Conference on Cloud Computing Technology and Science, Singapore, Singapore, pp 455–462

  39. Sun T, Xiao C, Xu X, Tian G (2017) An improved budget-deadline constrained workflow scheduling algorithm on heterogeneous resources. In: Proceedings of the 2017 IEEE 4th International Conference on Cyber Security and Cloud Computing (CSCloud), New York, NY, USA, pp 40–45

  40. Rizvi N, Ramesh D (2020) Fair budget constrained workflow scheduling approach for heterogeneous clouds. Cluster Comput 23(4):1–17

    Article  Google Scholar 

  41. Yu Z, Shi W (2008) A planner-guided scheduling strategy for multiple workflow applications. In: Proceedings of the 2008 International Conference on Parallel Processing-Workshops, Portland, OR, USA, pp 1–8

  42. Arabnejad H, Barbosa JG (2012) Fairness resource sharing for dynamic workflow scheduling on heterogeneous systems. In: Proceedings of the 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications, Leganes, Spain, pp 633–639

  43. Hsu CC, Huang KC, Wang FJ (2011) Online scheduling of workflow applications in grid environments. Futur Gener Comput Syst 27(6):860–870

    Article  Google Scholar 

  44. Calheiros RN, Ranjan R, Beloglazov A, De Rose CA, Buyya R (2011) Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw Pract Exp 41(1):23–50

    Article  Google Scholar 

  45. Piraghaj SF, Dastjerdi AV, Calheiros RN, Buyya R (2017) Containercloudsim: An environment for modeling and simulation of containers in cloud data centers. Softw Pract Exp 47(4):505–521

    Article  Google Scholar 

  46. Schad J, Dittrich J, Quiané-Ruiz JA (2010) Runtime measurements in the cloud: observing, analyzing, and reducing variance. In: Proceedings of the VLDB Endowment, Vol. 3, no.1–2, pp 460–471

  47. Mao M, Humphrey M A performance study on the VMstartup time in the cloud. In: Proceedings of the 2012 IEEE 5th International Conference on Cloud Computing, June 2012, Honolulu, HI, USA, pp 423–430

  48. Bharathi S, Chervenak A, Deelman E, Mehta G, Su M, Vahi K Characterization of scientific workflows. In: Proceedings of the 2008 Workshop on Workflows in Support of Large-Scale Science, December-2008, Austin, TX, USA, pp 1–10

  49. Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Futur Gener Comput Syst 29(3):682–692

    Article  Google Scholar 

  50. Singh V, Indrajeet G, Prasanta KJ (2018) A novel cost-efficient approach for deadline-constrained workflow scheduling by dynamic provisioning of resources. Futur Gener Comput Syst 79:95–110

    Article  Google Scholar 

  51. da Silva RF, Chen W, Juve G, Vahi K, Deelman E Community Resources for enabling research in distributed scientific workflows. In:

  52. da Silva RF, Pottier L, Coleman T, Deelman E, Casanova H (2020) WorkflowHub: community framework for enabling scientific workflow research and development. arXiv:2009.00250v1

  53. pegasus-isi/1000genome-workflow. https://github.com/pegasus-isi/1000genome-workflow. Accessed 29 Nov 2020

  54. Soybean Knowledge Base (SoyKB) Pipeline. https://pegasus.isi.edu/application-showcase/soykb. Accessed 29 Nov 2020

  55. pegasus-traces/cycles. https://github.com/workflowhub/pegasus-traces/tree/master/cycles. Accessed 4 Feb 2021

Download references

Funding

This study has received no funding from any organization.

Author information

Authors and Affiliations

  1. School of Computer Engineering, Iran University of Science and Technology, Hengam St., Resalat Sq., 16846-13114, Tehran, Iran

    Ehsan Saeedizade & Mehrdad Ashtiani

Authors
  1. Ehsan Saeedizade
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehrdad Ashtiani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehrdad Ashtiani.

Ethics declarations

Conflict of interest

All of the authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saeedizade, E., Ashtiani, M. DDBWS: a dynamic deadline and budget-aware workflow scheduling algorithm in workflow-as-a-service environments. J Supercomput 77, 14525–14564 (2021). https://doi.org/10.1007/s11227-021-03858-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-03858-6

Keywords

Search

Navigation