Skip to main content
SpringerLink
Log in

Towards decomposition based multi-objective workflow scheduling for big data processing in clouds

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

A workflow is a group of tasks that are processed in a particular order to complete an application. Also, it is a popular paradigm used to model complex big-data applications. Executing complex applications in a distributed system such as cloud or cluster implicates optimization of several conflicting objectives such as monetary cost, energy consumption, total execution time of the application (makespan). Regardless of this trend, most of the workflow scheduling approaches focused on single or bi-objective optimization problem. In this paper, we considered the problem of scheduling workflows in a cloud environment as a multi-objective optimization problem, and hence proposed a multi-objective workflow-scheduling algorithm based on decomposition. The proposed algorithm is capable of finding optimal solutions with a single run. Our evaluation results show that, by a single run, the proposed approach manages to obtain the Pareto Front solutions which are at least as good as schedules produced by running a single-objective scheduling algorithm with constraints for multiple times.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Hu, Z., Li, D., Guo, D.: Balance resource allocation for spark jobs based on prediction of the optimal resource. Tsinghua Sci. Technol. 25(04), 487–497 (2020)

    Article  Google Scholar 

  2. Garey, M.R., Johnson, D.S.: Computers and Intractability. A Guide to the Theory of NP-Completeness. W. H. Freeman & Co., New York (1990)

    Google Scholar 

  3. Cloudsigma.: Cloudsigma. (2009). https://www.cloudsigma.com/, Accessed 27 Jan 2020

  4. Elasticshosts.: Elastichosts. (2008). https://www.elastichosts.com/. Accessed 27 Jan 2020

  5. Pietri, I., Sakellariou, R.: Cost-efficient cpu provisioning for scientific workflows on clouds. In: Altmann, J., Silaghi, G.C., Rana, O.F. (eds.) Economics of Grids, Clouds, Systems, and Services. Springer International Publishing, Cham (2016)

    Google Scholar 

  6. Zhang, Q., Li, H.: Moea/d: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans. Evolut. Comput. 11(6), 712–731 (2007)

    Article  Google Scholar 

  7. Alla, H.B., Alla, S.B., Touhafi, A., Ezzati, A.: A novel task scheduling approach based on dynamic queues and hybrid meta-heuristic algorithms for cloud computing environment. Clust. Comput. 21(3), 1797–1820 (2018)

    Article  Google Scholar 

  8. Hosseinzadeh, M., Ghafour, M.Y., Hama, H.K., Vo, B., Khoshnevis, A.: Multi-objective task and workflow scheduling approaches in cloud computing: a comprehensive review. J. Grid Comput. 18, 327–356 (2020)

    Article  Google Scholar 

  9. Abazari, F., Analoui, M., Takabi, H., Fu, S.: Mows: multi-objective workflow scheduling in cloud computing based on heuristic algorithm. Simul. Modell. Pract. Theory 93, 119–132 (2019)

    Article  Google Scholar 

  10. Hu, H., Li, Z., Hu, H., Chen, J., Ge, J., Li, C., Chang, V.: Multi-objective scheduling for scientific workflow in multicloud environment. J. Netw. Comput. Appl. 114, 108–122 (2018)

    Article  Google Scholar 

  11. Zhou, X., Zhang, G., Sun, J., Zhou, J., Wei, T., Hu, S.: Minimizing cost and makespan for workflow scheduling in cloud using fuzzy dominance sort based heft. Future Gener. Comput. Syst. 93, 278–289 (2019)

    Article  Google Scholar 

  12. Bugingo, E., Zheng, W., Zhang, D., Qin, Y., Zhang, D.: (2019) Decomposition based multi-objective workflow scheduling for cloud environments. In: 2019 Seventh International Conference on Advanced Cloud and Big Data (CBD), pp. 37–42

  13. Iranmanesh, A., Naji, H.R.: DCHG-TS: a deadline-constrained and cost-effective hybrid genetic algorithm for scientific workflow scheduling in cloud computing. Clust. Comput. (2019). https://doi.org/10.1007/s10586-020-03145-8

    Article  Google Scholar 

  14. Emmanuel, B., Qin, Y., Wang, J., Zhang, D., Zheng, W.: Cost optimization heuristics for deadline constrained workflow scheduling on clouds and their comparative evaluation. Concurr. Comput. 30(20), e4762 (2018)

    Article  Google Scholar 

  15. Topcuoglu, H., Hariri, S., Min-You, Wu: Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans. Parall. Distribut. Syst. 13(3), 260–274 (2002)

    Article  Google Scholar 

  16. Ahmad, W., Alam, B., Ahuja, S., Malik, S.: A dynamic VM provisioning and de-provisioning based cost-efficient deadline-aware scheduling algorithm for Big Data workflow applications in a cloud environment. Clust. Comput. (2020). https://doi.org/10.1007/s10586-020-03100-7

  17. Ijaz, S., Munir, E.U.: MOPT: list-based heuristic for scheduling workflows in cloud environment. J. Supercomput. 75(7), 3740–3768 (2020)

    Article  Google Scholar 

  18. Zhou, N., Lin, W., Feng, W., Shi, F., Pang, X.: Budget-deadline constrained approach for scientific workflows scheduling in a cloud environment. Clust. Comput. (2020). https://doi.org/10.1007/s10586-020-03176-1

  19. Zheng, W., Qin, Y., Bugingo, E., Zhang, D., Chen, J.: Cost optimization for deadline-aware scheduling of big-data processing jobs on clouds. Future Gener. Comput. Syst. 82, 244–255 (2018)

    Article  Google Scholar 

  20. Choudhary, A., Gupta, I., Singh, V., Jana, P.K.: A GSA based hybrid algorithm for bi-objective workflow scheduling in cloud computing. Future Gener. Comput. Syst. 83, 14–26 (2018)

    Article  Google Scholar 

  21. Xue, C., Lin, C., Hu, J.: Scalability analysis of request scheduling in cloud computing. Tsinghua Sci. Technol. 24(03), 249–261 (2019)

    Article  Google Scholar 

  22. Zhang, H., Xie, J., Ge, J., Shi, J., Zhang, Z.: Hybrid particle swarm optimization algorithm based on entropy theory for solving DAR scheduling problem. Tsinghua Sci. Technol. 24(03), 281–290 (2019)

    Google Scholar 

  23. Zhang, M., Li, H., Liu, L., Buyya, R.: An adaptive multi-objective evolutionary algorithm for constrained workflow scheduling in clouds. Distribut. Parall. Databases 36(2), 339–368 (2018)

    Article  Google Scholar 

  24. Singh, V., Gupta, I., Jana, P.K.: An energy efficient algorithm for workflow scheduling in IAAS cloud. J. Grid Comput. 18, 357–376 (2020)

    Article  Google Scholar 

  25. Li, F., Liu, J., Huang, P., Shi, H.: (2018) An indicator and decomposition based steady-state evolutionary algorithm for many-objective optimization. Math. Probl. Eng. (2018)

  26. Miettinen, K., Mustajoki, J., Stewart, T.J.: Interactive multiobjective optimization with nimbus for decision making under uncertainty. OR Spectrum 36(1), 39–56 (2014)

    Article  MathSciNet  Google Scholar 

  27. Miettinen, K., Mäkelä, M.M.: Synchronous approach in interactive multiobjective optimization. Eur. J. Operat. Res. 170(3), 909–922 (2006)

    Article  Google Scholar 

  28. Zheng, W., Emmanuel, B., Wang, C., Qin ,Y., Zhang, D.: Cost optimization for scheduling scientific workflows on clouds under deadline constraints. In: 2017 Fifth International Conference on Advanced Cloud and Big Data (CBD), pp. 51–56 (2017)

  29. Juve, G.: Workflowgenerator. (2014). https://confluence.pegasus.isi.edu/display/pegasus/WorkflowGenerator. Accessed 27 Jan 2020

  30. Juve, G., Chervenak, A., Deelman, E., Bharathi, S., Mehta, G., Vahi, K.: Characterizing and profiling scientific workflows. Future Gener. Comput. Syst. 29(3), 682–692 (2013b)

    Article  Google Scholar 

  31. Sun, T., Xiao, C., Xu, X.: A scheduling algorithm using sub-deadline for workflow applications under budget and deadline constrained. Clust. Comput. 22(3), 5987–5996 (2019)

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Science Foundation of Fujian Province of China (No. 2018J01107), and was also jointly supported by the National Natural Science Foundation of China (NSFC, Grant No. 61672439).

Author information

Authors and Affiliations

  1. Department of Computer Science, School of Informatics, Xiamen University, Xiamen, China

    Emmanuel Bugingo, Defu Zhang, Zhaobin Chen & Wei Zheng

Authors
  1. Emmanuel Bugingo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Defu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhaobin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Zheng.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bugingo, E., Zhang, D., Chen, Z. et al. Towards decomposition based multi-objective workflow scheduling for big data processing in clouds. Cluster Comput 24, 115–139 (2021). https://doi.org/10.1007/s10586-020-03208-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-020-03208-w

Keywords

Search

Navigation