Skip to main content
SpringerLink
Log in

Dynamic provisioning with structure inspired selection and limitation of VMs based cost-time efficient workflow scheduling in the cloud

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Workflow scheduling in cloud computing environments is nowadays a hot topic as scientific workflows application are gradually taking advantage of commercial cloud assets. Common users’ quality of service (QoS) requirements are the respect of defined budget and deadline when executing their workflow job. Since execution cost minimization and completion time minimization are contradictory objectives, addressing such issue through trade-off function approaches have proved to be an efficient way. This paper presents the Cost-Time Trade-off efficient Workflow Scheduling with Dynamic provisioning (CTTWSDP) algorithm. CTTWSDP relies on dynamic VMs provisioning with a limited number of leased VMs, and a cost-time trade-off function over heterogeneous instances to determine the most viable schedule. CTTWSDP also proposed an improved Implicit Requested Instance Types Range (IRITR) evaluation, which is a scheduling concept introduced in our previous work. The IRITR evaluation aims at determining a range of VMs instance types that best suits the workflow execution, in order to avoid overbidding and underbidding that may lead to budget and deadline violation respectively. The results of simulations prove the effectiveness of the proposal. CTTWSDP achieves a 17.09–76.06% higher success rate when compared to four state-of-the-art algorithms. Furthermore, ANOVA along with Tukey–Kramer post-hoc tests have been conducted, revealing the effectiveness of CTTWSDP over three of the baseline algorithm, while for the fourth one the outperformance of CTTWSDP is not statistically significant. An analysis of the standard deviation of the success rate proves that CTTWSDP is more stable in its performance no matter the type and the workload of the workflow. With a standard deviation of 6.73, smaller than the ones obtained by the other algorithms from 18.66 to 34.10.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

Notes

  1. https://www.excel-easy.com/examples/anova.html

  2. http://davidmlane.com/hyperstat/sr_table.html

References

  1. Taylor, I.J., Deelman, E., Gannon, D.B., Shields, M., et al.: Workflows for e-Science: Scientific Workflows for Grids, vol. 1. Springer, New York (2007)

    Book  Google Scholar 

  2. Gil, Y., Deelman, E., Ellisman, M., Fahringer, T., Fox, G., Gannon, D., Goble, C., Livny, M., Moreau, L., Myers, J.: Examining the challenges of scientific workflows. Computer 40(12), 24–32 (2007)

    Article  Google Scholar 

  3. Ullman, J.D.: Np-complete scheduling problems. J. Comput. Syst. Sci. 10(3), 384–393 (1975)

    Article  MathSciNet  Google Scholar 

  4. Balis, B., Figiela, K., Jopek, K., Malawski, M., Pawlik, M.: Porting hpc applications to the cloud: a multi-frontal solver case study. J. Comput. Sci. 18, 106–116 (2017)

    Article  Google Scholar 

  5. Madduri, R., Chard, K., Chard, R., Lacinski, L., Rodriguez, A., Sulakhe, D., Kelly, D., Dave, U., Foster, I.: The globus galaxies platform: delivering science gateways as a service. Concurr. Comput. Pract. Expe. 27(16), 4344–4360 (2015)

    Article  Google Scholar 

  6. Vöckler, J.S., Juve, G., Deelman, E., Rynge, M., Berriman, B.: Experiences using cloud computing for a scientific workflow application. In: Proceedings of the 2nd international workshop on Scientific cloud computing, ACM, pp 15–24 (2011)

  7. Jackson, K.R., Ramakrishnan, L., Muriki, K., Canon, S., Cholia, S., Shalf, J., Wasserman, H.J., Wright, N.J.: Performance analysis of high performance computing applications on the amazon web services cloud. In: Proceedings of the 2nd IEEE international conference on cloud computing technology and science, IEEE, pp 159–168 (2010)

  8. Juve, G., Deelman, E., Vahi, K., Mehta, G., Berriman, B., Berman, B.P., Maechling, P.: Scientific workflow applications on amazon ec2. In: Proceedings of the 2009 5th IEEE international conference on e-science workshops, IEEE, pp 59–66 (2009)

  9. Deelman, E., Singh, G., Livny, M., Berriman, B., Good, J.: The cost of doing science on the cloud: the montage example. In: SC’08: Proceedings of the 2008 ACM/IEEE conference on Supercomputing, IEEE, pp 1–12 (2008)

  10. Rodriguez, M.A., Buyya, R.: Scheduling dynamic workloads in multi-tenant scientific workflow as a service platforms. Future Gen. Comput. Syst. 79, 739–750 (2018)

    Article  Google Scholar 

  11. Arabnejad, V., Bubendorfer, K., Ng, B.: Budget and deadline aware e-science workflow scheduling in clouds. IEEE Trans. Parallel Distrib. Syst. 30(1), 29–44 (2019)

    Article  Google Scholar 

  12. Ndamlabin Mboula, J.E., Kamla, V.C., Tayou Djamegni, C.: Cost-time trade-off efficient workflow scheduling in cloud. Simul. Model. Pract. Theory, p. 102107 (2020)

  13. Hilman, M.H., Rodriguez, M.A., Buyya, R.: Multiple workflows scheduling in multi-tenant distributed systems: a taxonomy and future directions. ACM Comput. Surv. (CSUR) 53(1), 1–39 (2020)

    Article  Google Scholar 

  14. Ghobaei-Arani, M., Jabbehdari, S., Pourmina, M.A.: An autonomic resource provisioning approach for service-based cloud applications: a hybrid approach. Future Gen. Comput. Syst. 78, 191–210 (2018)

    Article  Google Scholar 

  15. Hilman, M.H., Rodriguez, M.A., Buyya, R.: Task runtime prediction in scientific workflows using an online incremental learning approach. In: Proceedings of the 2018 IEEE/ACM 11th International Conference on Utility and Cloud Computing (UCC), IEEE, pp 93–102 (2018)

  16. Juve, G., Chervenak, A., Deelman, E., Bharathi, S., Mehta, G., Vahi, K.: Characterizing and profiling scientific workflows. Future Gen. Comput. Syst. 29(3), 682–692 (2013)

    Article  Google Scholar 

  17. Singh, V., Gupta, I., Jana, P.K.: A novel cost-efficient approach for deadline-constrained workflow scheduling by dynamic provisioning of resources. Future Gen. Comput. Syst. 79, 95–110 (2018)

    Article  Google Scholar 

  18. Garg, R., Mittal, M., et al.: Reliability and energy efficient workflow scheduling in cloud environment. Clust. Comput. 22(4), 1283–1297 (2019)

    Article  Google Scholar 

  19. Faragardi, H.R., Sedghpour, M.R.S., Fazliahmadi, S., Fahringer, T., Rasouli, N.: Grp-heft: a budget-constrained resource provisioning scheme for workflow scheduling in iaas clouds. IEEE Trans. Parallel Distrib. Syst. 31, 1239–1254 (2019)

    Article  Google Scholar 

  20. Arabnejad, V., Bubendorfer, K., Ng, B.: Scheduling deadline constrained scientific workflows on dynamically provisioned cloud resources. Future Gen. Comput. Syst. 75, 348–364 (2017)

    Article  Google Scholar 

  21. Ahmad, W., Alam, B., Ahuja, S., Malik, S.: A dynamic vm provisioning and de-provisioning based cost-efficient deadline-aware scheduling algorithm for big data workflow applications in a cloud environment. Clust. Comput., pp 1–30 (2020)

  22. Chen, W., Deelman, E.: Workflowsim: A toolkit for simulating scientific workflows in distributed environments. In: Proceedings of the 2012 IEEE 8th International Conference on E-Science, IEEE, pp 1–8 (2012)

  23. Bharathi, S., Chervenak, A., Deelman, E., Mehta, G., Su, M.H., Vahi, K.: Characterization of scientific workflows. In: Proceedings of the 2008 third workshop on workflows in support of large-scale science, IEEE, pp 1–10 (2008)

  24. Topcuoglu, H., Hariri, S., My, Wu.: Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans. Parallel Distrib. Syst. 13(3), 260–274 (2002)

    Article  Google Scholar 

  25. Hu, X.H., Ouyang, J.C., Yang, Z.H., Chen, Z.H.: An ipso algorithm for grid task scheduling based on satisfaction rate. In: Proceedings of the 2009 International Conference on Intelligent Human-Machine Systems and Cybernetics, IEEE, vol. 1, pp. 262–265 (2009)

  26. Chen, W.N., Zhang, J.: An ant colony optimization approach to a grid workflow scheduling problem with various qos requirements. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 39(1), 29–43 (2008)

    Article  MathSciNet  Google Scholar 

  27. Chuang, L.Y., Tsai, S.W., Yang, C.H. Catfish particle swarm optimization. In: Proceedings of the 2008 IEEE Swarm Intelligence Symposium, IEEE, pp 1–5. https://doi.org/10.1109/SIS.2008.4668277 (2008)

  28. Verma, A., Kaushal, S.: A hybrid multi-objective particle swarm optimization for scientific workflow scheduling. Parallel Comput. 62, 1–19 (2017)

    Article  MathSciNet  Google Scholar 

  29. Elsherbiny, S., Eldaydamony, E., Alrahmawy, M., Reyad, A.E.: An extended intelligent water drops algorithm for workflow scheduling in cloud computing environment. Egypt. Inf. J. 19(1), 33–55 (2018)

    Google Scholar 

  30. Casas, I., Taheri, J., Ranjan, R., Wang, L., Zomaya, A.Y.: Ga-eti: an enhanced genetic algorithm for the scheduling of scientific workflows in cloud environments. J. Comput. Sci. 26, 318–331 (2018)

    Article  Google Scholar 

  31. Zhou, J., Wang, T., Cong, P., Lu, P., Wei, T., Chen, M.: Cost and makespan-aware workflow scheduling in hybrid clouds. J. Syst. Architect. 100, 101631 (2019)

    Article  Google Scholar 

  32. Biswas, T., Kuila, P., Ray, A.K., Sarkar, M.: Gravitational search algorithm based novel workflow scheduling for heterogeneous computing systems. Simul. Model. Pract. Theory 96, 101932 (2019)

    Article  Google Scholar 

  33. Zheng, W., Sakellariou, R.: Budget-deadline constrained workflow planning for admission control. J. Grid Comput. 11(4), 633–651 (2013)

    Article  Google Scholar 

  34. Verma, A., Kaushal, S.: Cost-time efficient scheduling plan for executing workflows in the cloud. J. Grid Comput. 13(4), 495–506 (2015)

    Article  MathSciNet  Google Scholar 

  35. Poola, D., Garg, S.K., Buyya, R., Yang, Y., Ramamohanarao, K.: Robust scheduling of scientific workflows with deadline and budget constraints in clouds. In: Proceedings of the 2014 IEEE 28th international conference on advanced information networking and applications, IEEE, pp. 858–865 (2014)

  36. Khan, M.A.: Scheduling for heterogeneous systems using constrained critical paths. Parallel Comput. 38(4–5), 175–193 (2012)

    Article  Google Scholar 

  37. Amazon EC2 (????a) Amazon EC2 Instance Types. https://aws.amazon.com/ec2/instance-types/, online; accessed 06 July 2019

  38. Amazon EC2 (????b) Amazon EC2 pricing. https://aws.amazon.com/ec2/pricing/on-demand/, online; accessed 06 July 2019

  39. Mao, M., Humphrey, M.: A performance study on the vm startup time in the cloud. In: Proceedings of the 2012 IEEE Fifth International Conference on Cloud Computing, IEEE, pp. 423–430 (2012)

  40. Ghobaei-Arani, M., Rahmanian, A.A., Aslanpour, M.S., Dashti, S.E.: Csa-wsc: cuckoo search algorithm for web service composition in cloud environments. Soft. Comput. 22(24), 8353–8378 (2018)

    Article  Google Scholar 

  41. Ghobaei-Arani, M., Rahmanian, A.A., Souri, A., Rahmani, A.M.: A moth-flame optimization algorithm for web service composition in cloud computing: simulation and verification. Softw. Pract. Exp. 48(10), 1865–1892 (2018)

    Google Scholar 

  42. Singh, V., Gupta, I., Jana, P.K.: An energy efficient algorithm for workflow scheduling in iaas cloud. J. Grid Comput. pp. 1–20 (2019)

  43. Choudhary, A., Gupta, I., Singh, V., Jana, P.K.: A gsa based hybrid algorithm for bi-objective workflow scheduling in cloud computing. Future Gen. Comput. Syst. 83, 14–26 (2018)

    Article  Google Scholar 

  44. Statology. How to Perform a Tukey–Kramer Post Hoc Test in Excel. https://www.statology.org/tukey-kramer-post-hoc-test-excel/, online; accessed 9 January 2021 (2020)

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Science, Faculty of Science, University of Ngaoundere, P.O. Box 454, Ngaoundere, Cameroon

    Jean Etienne Ndamlabin Mboula

  2. Department of Mathematics and Computer Science, ENSAI, University of Ngaoundere, P.O. Box 455, Ngaoundere, Cameroon

    Vivient Corneille Kamla

  3. Department of Mathematics and Computer Science, Faculty of Science, University of Dschang, P.O. Box 67, Dschang, Cameroon

    Clémentin Tayou Djamégni

Authors
  1. Jean Etienne Ndamlabin Mboula
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vivient Corneille Kamla
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Clémentin Tayou Djamégni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jean Etienne Ndamlabin Mboula.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ndamlabin Mboula, J.E., Kamla, V.C. & Tayou Djamégni, C. Dynamic provisioning with structure inspired selection and limitation of VMs based cost-time efficient workflow scheduling in the cloud. Cluster Comput 24, 2697–2721 (2021). https://doi.org/10.1007/s10586-021-03289-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-021-03289-1

Keywords

Search

Navigation