Skip to main content
SpringerLink
Log in

A flexible deadline-driven resource provisioning and scheduling algorithm for multiple workflows with VM sharing protocol on WaaS-cloud

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Maximized deployment of workflows in the research organizations has motivated the emergence of multi-tenant environment that offer these workflows deployment as a service. Hence, Workflow-as-a-Service (WaaS) model has been made by research organizations to handle the future model of Workflow Management System (WMS) that would serve a variety of users from a specific single point of provision. These mechanisms vary from widely used WMS in that they process a variety of scientific workflows at the time of execution. A widely used WMS is mostly used to run a single workflow in a dedicated set-up meanwhile WaaS-cloud platforms improving the set-up by adopting multiple workflows deployment in a multi-tenant service model. In this study, we leverage an advanced Virtual Machine (VM) sharing protocol to optimize VM utilization and to achieve specific Quality of Service conditions from a variety of users in WaaS cloud platforms. We present a Flexible Deadline-driven resource Provisioning and scheduling algorithm for Multiple workflows (FDPM) that can minimize the computing expenses by adapting VM sharing to make the workflow’s execution cost lesser while achieving a user-assigned deadline. Our analysis proves that the FDPM algorithm can leverage the VM sharing protocol to make better performance with regard to minimizing the execution cost evaluated to the modified deadline-driven scheduling algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Deelman E, Peterka T, Altintas I, Carothers CD, van Dam KK, Moreland K, Parashar M, Ramakrishnan L, Taufer M, Vetter J (2018) The future of scientific workflows. Int J High Perform Comput Appl 32(1):159–175

    Article  Google Scholar 

  2. Deelman E, Singh G, Livny M, Berriman B, Good J (2008) The cost of doing science on the cloud: the montage example. In: SC'08: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing. IEEE, pp 1–12

  3. Leitner P, Cito J (2016) Patterns in the chaos—a study of performance variation and predictability in public IaaS clouds. ACM Trans Internet Technol (TOIT) 16(3):1–23

    Article  Google Scholar 

  4. Rodriguez MA, Buyya R (2017) A taxonomy and survey on scheduling algorithms for scientific workflows in IaaS cloud computing environments. Concurr Comput Pract Exp 29(8):e4041

    Article  Google Scholar 

  5. Alkhanak EN, Lee SP, Rezaei R, Parizi RM (2016) Cost optimization approaches for scientific workflow scheduling in cloud and grid computing: a review, classifications, and open issues. J Syst Softw 113:1–26

    Article  Google Scholar 

  6. Rajasekar P, Palanichamy Y (2021) Scheduling multiple scientific workflows using containers on IaaS cloud. J Ambient Intell Humaniz Comput 12(7):7621–7636

    Article  Google Scholar 

  7. Hilman MH, Rodriguez MA, Buyya R (2017) Task-based budget distribution strategies for scientific workflows with coarse-grained billing periods in IaaS clouds. In: 2017 IEEE 13th International Conference on e-Science (e-Science). IEEE, pp 128–137

  8. Jones M, Arcand B, Bergeron B, Bestor D, Byun C, Milechin L, Gadepally V, Hubbell M, Kepner J, Michaleas P, Mullen J (2016) Scalability of VM provisioning systems. In: 2016 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, pp 1–5

  9. Lopes RV, Menascé D (2016) A taxonomy of job scheduling on distributed computing systems. IEEE Trans Parallel Distrib Syst 27(12):3412–3428

    Article  Google Scholar 

  10. Yu J, Buyya R (2005) A taxonomy of workflow management systems for grid computing. J Grid Comput 3(3–4):171–200

    Article  Google Scholar 

  11. Wieczorek M, Hoheisel A, Prodan R (2008) Taxonomies of the multi-criteria grid workflow scheduling problem. In: Grid middleware and services. Springer, Boston, pp 237–264

  12. Tsafrir D, Etsion Y, Feitelson DG (2007) Backfilling using system-generated predictions rather than user runtime estimates. IEEE Trans Parallel Distrib Syst 18(6):789–803

    Article  Google Scholar 

  13. Verma A, Cherkasova L, Campbell RH (2011) Aria: automatic resource inference and allocation for mapreduce environments. In: Proceedings of the 8th ACM International Conference on Autonomic Computing, pp 235–244

  14. Rimal BP, Maier M (2016) Workflow scheduling in multi-tenant cloud computing environments. IEEE Trans Parallel Distrib Syst 28(1):290–304

    Article  Google Scholar 

  15. Chen H, Zhu J, Wu G, Huo L (2018) Cost-efficient reactive scheduling for real-time workflows in clouds. J Supercomput 74(11):6291–6309

    Article  Google Scholar 

  16. Stavrinides GL, Duro FR, Karatza HD, Blas JG, Carretero J (2017) Different aspects of workflow scheduling in large-scale distributed systems. Simul Model Pract Theory 70:120–134

    Article  Google Scholar 

  17. Arabnejad H, Barbosa JG (2017) Maximizing the completion rate of concurrent scientific applications under time and budget constraints. J Comput Sci 23:120–129

    Article  MathSciNet  Google Scholar 

  18. Zhou N, Li F, Xu K, Qi D (2018) Concurrent workflow budget-and deadline-constrained scheduling in heterogeneous distributed environments. Soft Comput 22(23):7705–7718

    Article  Google Scholar 

  19. Arabnejad H, Barbosa JG (2017) Multi-QoS constrained and profit-aware scheduling approach for concurrent workflows on heterogeneous systems. Futur Gener Comput Syst 68:211–221

    Article  Google Scholar 

  20. Chen H, Zhu J, Zhang Z, Ma M, Shen X (2017) Real-time workflows oriented online scheduling in uncertain cloud environment. J Supercomput 73(11):4906–4922

    Article  Google Scholar 

  21. Chen H, Zhu X, Liu G, Pedrycz W (2018) Uncertainty-aware online scheduling for real-time workflows in cloud service environment. IEEE Trans Serv Comput

  22. Liu L, Zhang M, Buyya R, Fan Q (2017) Deadline-constrained coevolutionary genetic algorithm for scientific workflow scheduling in cloud computing. Concurr Comput Pract Exp 29(5):e3942

    Article  Google Scholar 

  23. Xu X, Xiao C, Tian G, Sun T (2017) Expansion slot backfill scheduling for concurrent workflows with deadline on heterogeneous resources. Clust Comput 20(1):471–483

    Article  Google Scholar 

  24. Ghafouri R, Movaghar A (2021) An adaptive and deadline-constrained workflow scheduling algorithm in infrastructure as a service clouds. Iran J Comput Sci https://doi.org/10.1007/s42044-021-00082-6

  25. Sun T, Xiao C, Xu X (2019) A scheduling algorithm using sub-deadline for workflow applications under budget and deadline constrained. Clust Comput 22(3):5987–5996

    Article  Google Scholar 

  26. KhojastehToussi G, Naghibzadeh M (2021) A divide and conquer approach to deadline constrained cost-optimization workflow scheduling for the cloud. Cluster Comput 24:1711–1733. https://doi.org/10.1007/s10586-020-03223-x

    Article  Google Scholar 

  27. Alworafi MA, Mallappa S (2020) A collaboration of deadline and budget constraints for task scheduling in cloud computing. Clust Comput 23(2):1073–1083

    Article  Google Scholar 

  28. Chen W, Xie G, Li R, Li K (2021) Execution cost minimization scheduling algorithms for deadline-constrained parallel applications on heterogeneous clouds. Clust Comput 24(2):701–715

    Article  Google Scholar 

  29. Zhou AC, He B, Liu C (2015) Monetary cost optimizations for hosting workflow-as-a-service in IaaS clouds. IEEE Trans Cloud Comput 4(1):34–48

    Article  Google Scholar 

  30. Rajasekar P, Palanichamy Y (2021) Adaptive resource provisioning and scheduling algorithm for scientific workflows on IaaS cloud. SN Comput Sci 2:456. https://doi.org/10.1007/s42979-021-00852-w

    Article  Google Scholar 

  31. Saeedizade E, Ashtiani M (2021) DDBWS: a dynamic deadline and budget-aware workflow scheduling algorithm in workflow-as-a-service environments. J Supercomput. https://doi.org/10.1007/s11227-021-03858-6

    Article  Google Scholar 

  32. Ahmad W, Alam B, Atman A (2021) An energy-efficient big data workflow scheduling algorithm under budget constraints for heterogeneous cloud environment. J Supercomput 77:11946–11985. https://doi.org/10.1007/s11227-021-03733-4

    Article  Google Scholar 

  33. Iranmanesh A, Naji HR (2021) DCHG-TS: a deadline-constrained and cost-effective hybrid genetic algorithm for scientific workflow scheduling in cloud computing. Clust Comput 24(2):667–681

    Article  Google Scholar 

  34. Chen W, Xie G, Li R, Bai Y, Fan C, Li K (2017) Efficient task scheduling for budget constrained parallel applications on heterogeneous cloud computing systems. Futur Gener Comput Syst 74:1–11

    Article  Google Scholar 

  35. Shea R, Wang F, Wang H, Liu J (2014) A deep investigation into network performance in virtual machine based cloud environments. In: IEEE INFOCOM 2014-IEEE Conference on Computer Communications. IEEE, pp 1285–1293

  36. Hilman MH, Rodriguez MA, Buyya R (2018) Task runtime prediction in scientific workflows using an online incremental learning approach. In: 2018 IEEE/ACM 11th International Conference on Utility and Cloud Computing (UCC). IEEE, pp 93–102

  37. Pham TP, Durillo JJ, Fahringer T (2017) Predicting workflow task execution time in the cloud using a two-stage machine learning approach. IEEE Trans Cloud Comput 8(1):256–268

    Article  Google Scholar 

  38. Kozhirbayev Z, Sinnott RO (2017) A performance comparison of container-based technologies for the cloud. Futur Gener Comput Syst 68:175–182

    Article  Google Scholar 

  39. Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Futur Gener Comput Syst 29(3):682–692

    Article  Google Scholar 

  40. Da Silva RF, Chen W, Juve G, Vahi K, Deelman E (2014) Community resources for enabling research in distributed scientific workflows. In: 2014 IEEE 10th International Conference on e-Science, vol 1. IEEE, pp 177–184

  41. Calheiros RN, Ranjan R, Beloglazov A, De Rose CA, Buyya R (2011) CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw Pract Exp 41(1):23–50

    Article  Google Scholar 

  42. Ullrich M, Lässig J, Sun J, Gaedke M, Aida K (2018) A benchmark model for the creation of compute instance performance footprints. In: International Conference on Internet and Distributed Computing Systems. Springer, Cham, pp 221–234

Download references

Funding

The first author would like to thank “Anna Centenary Research Fellowship, Anna University” for supporting the proposed research work financially in the form of scholarship.

Author information

Authors and Affiliations

  1. Department of Information Science and Technology, College of Engineering, Anna University, Guindy, Chennai, Tamil Nadu, India

    P. Rajasekar & Yogesh Palanichamy

Authors
  1. P. Rajasekar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yogesh Palanichamy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to P. Rajasekar.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rajasekar, P., Palanichamy, Y. A flexible deadline-driven resource provisioning and scheduling algorithm for multiple workflows with VM sharing protocol on WaaS-cloud. J Supercomput 78, 8025–8055 (2022). https://doi.org/10.1007/s11227-021-04225-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-04225-1

Keywords

Search

Navigation