Skip to main content
SpringerLink
Log in

Adaptive multiple-workflow scheduling with task rearrangement

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Large-scale distributed computing systems like grids and more recently clouds are a platform of choice for many resource-intensive applications. Workflow applications account for the majority of these applications, particularly in science and engineering. A workflow application consists of multiple precedence-constrained tasks with data dependencies. Since resources in those systems are shared by many users and applications deployed there are very diverse, scheduling is complicated. Often, the actual execution of applications differs from the original schedule following delays such as those caused by resource contention and other issues in performance prediction. These delays have further impact when running multiple workflow applications due to inter-task dependencies. In this paper, we investigate the problem of scheduling multiple workflow applications concurrently, explicitly taking into account scheduling robustness. We present a dynamic task rearrangement and rescheduling algorithm that exploits the scheduling flexibility from precedence constraints among tasks. The algorithm optimizes resource allocation among multiple workflows, and it often stops the influence of delayed execution passing to subsequent tasks. The experimental results demonstrate that our approach can significantly improve performance in multiple-workflow scheduling.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Oinn T et al (2004) Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics 20(17):3045–3054

    Article  Google Scholar 

  2. Maechling P et al (2005) Simplifying construction of complex workflows for non-expert users of the Southern California Earthquake Center community modeling environment. ACM SIGMOD Rec 34(3):24–30

    Article  Google Scholar 

  3. Plale B et al (2005) Towards dynamically adaptive weather analysis and forecasting in LEAD. In: Proceedings of int’l conf. on computational science, workshop on dynamic data driven applications, pp 624–631

  4. Greenberg A, Hamilton J, Maltz DA, Patel P (2008) The cost of a cloud: research problems in data center networks. ACM SIGCOMM Comput Commun Rev 39(1):68–73

    Article  Google Scholar 

  5. Topcuoglu H, Hariri S, Wu M (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260–274

    Article  Google Scholar 

  6. Mandal A et al (2005) Scheduling strategies for mapping application workflows onto the grid. In: Proceedings of IEEE int’l symp. on high performance distributed computing, pp 125–134

  7. Rahman M, Venugopal S, Buyya R (2007) A dynamic critical path algorithm for scheduling scientific workflow applications on global grids. In: Proceedings of IEEE int’l conf. on e-science and grid computing, pp 35–42

  8. Cao H, Jin H, Wu X, Wu S, Shi X (2008) DAGMap: efficient scheduling for DAG grid workflow job. In: Proceedings of ACM/IEEE int’l conf. on grid computing, pp 17–24

  9. Blythe J et al (2005) Task scheduling strategies for workflow-based applications in grids. In: Proceedings of IEEE int’l symp. on cluster computing and the grid, pp 759–767

  10. Yu J, Kirley M, Buyya R (2007) Multi-objective planning for workflow execution on grids. In: Proceedings of IEEE/ACM int’l conf. on grid computing, pp 10–17

  11. Lee YC, Subrata R, Zomaya AY (2009) On the performance of a dual-objective optimization model for workflow applications on grid platforms. IEEE Trans Parallel Distrib Syst 20(9):1273–1284

    Article  Google Scholar 

  12. Fahringer T et al (2005) ASKALON: a tool set for cluster and grid computing. Concurr Comput Pract Exp 17(2–4):143–169

    Article  Google Scholar 

  13. Decker J, Schneider J (2007) Heuristic scheduling of grid workflows supporting co-allocation and advance reservation. In: Proceedings of IEEE int’l symp. on cluster computing and the grid, pp 335–342

  14. Wieczorek M et al (2006) Applying Advance reservation to increase predictability of workflow execution on the grid. In: Proceedings of IEEE international conference on e-science and grid computing, pp 82–90

  15. Smith W, Foster I, Taylor V (2000) Scheduling with advanced reservations. In: Proceedings of IEEE international symposium on parallel and distributed processing, pp 127–132

  16. Curino C et al (2014) Reservation-based scheduling: if you’re late don’t blame us!. In: Proceedings of ACM symp. on cloud, computing, pp 1–14

  17. Majumdar S (2009) The any-schedulability criterion for providing QoS guarantees through advance reservation requests. In: Proceedings of IEEE/ACM international symposium on cluster computing and the grid, pp 490–495

  18. Chen W, Fekete A, Lee YC (2010) Exploiting Deadline Flexibility in Grid Workflow Rescheduling” deadline flexibility in grid workflow rescheduling. In: Proceedings of ACM/IEEE int’l conf. on grid computing, pp 105–112

  19. Chen W (2012) High performance multiple-workflow scheduling using task rearrangement. PhD thesis, University of Sydney

  20. Smith W, Foster I, Taylor V (2004) Predicting application run times with historical information. J Parallel Distrib Comput 64(9):1007–1016

    Article  MATH  Google Scholar 

  21. Duan R, Nadeem F, Wang J, Zhang Y, Prodan R, Fahringer T (2009) A hybrid intelligent method for performance modeling and prediction of workflow activities in grids. In: Proceedings of IEEE int’l symp. on cluster computing and the grid, pp 339–347

  22. Kleinberg J, Tardos E (2006) Algorithm design. Pearson/Addison-Wesley, USA

  23. Mu’alem AW, Feitelson DG (2001) Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling. IEEE Trans Parallel Distrib Syst 12(6):529–543

    Article  Google Scholar 

  24. Netto MAS, Buyya R (2008) Rescheduling co-allocation requests based on flexible advance reservations and processor remapping. In: Proceedings of IEEE/ACM int’l conf. on grid computing,break pp 144–151

  25. Zhao H, Sakellariou R (2006) Scheduling multiple DAGs onto heterogeneous systems. In: Proceedings of the 15th heterogeneous computing workshop

  26. Yu Z, Shi W (2008) A planner-guided scheduling strategy for multiple workflow applications. In: Proceedings of int’l conf. on parallel processing-workshops, pp 1–8

  27. Mao M, Humphrey M (2011) Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In: Proceedings of 2011 int’l conf. for high performance computing, networking, storage and analysis (SC), pp 49:1–49:12

  28. Malawski M, Juve G, Deelman E, Nabrzyski J (2012) Cost-and deadline-constrained provisioning for scientific workflow ensembles in iaas clouds. In: Proc. of int’l conf. on high performance computing, networking, storage and analysis, p 22

  29. Ali S, Maciejewski AA, Siegel HJ, Kim J-K (2004) Measuring the robustness of a resource allocation. IEEE Trans Parallel Distrib Syst 15(7):630–641

    Article  Google Scholar 

  30. Shestak V, Smith J, Maciejewski AA, Siegel HJ (2008) Stochastic robustness metric and its use for static resource allocations. J Parallel Distrib Comput 68(8):1157–1173

    Article  MATH  Google Scholar 

  31. Shi Z, Jeannot E, Dongarra JJ (2006) Robust task scheduling in non-deterministic heterogeneous computing systems. In: Proceedings of IEEE int’l conf. on cluster computing, pp 1–10

  32. Sakellariou R, Zhao H (2004) A low-cost rescheduling policy for efficient mapping of workflows on grid systems. Sci Program 12(4):253–262

    Google Scholar 

  33. Yu Z, Shi W (2007) An adaptive rescheduling strategy for grid workflow applications. In: Proc. of IEEE int’l symp. on parallel and distributed processing

  34. Zhang Y, Koelbel C, Cooper K (2009) Hybrid re-scheduling mechanisms for workflow applications on multi-cluster. In: Proc. of IEEE int’l symp. on cluster computing and the grid, pp 116–123

Download references

Acknowledgments

The work of A. Zomaya is supported by the Australian Research Council Discovery Grant DP1097111. This paper revises and incorporates material from [18, 19].

Author information

Authors and Affiliations

  1. School of Information Technologies, The University of Sydney, Sydney, NSW, 2006, Australia

    Wei Chen, Young Choon Lee, Alan Fekete & Albert Y. Zomaya

Authors
  1. Wei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Young Choon Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alan Fekete
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Albert Y. Zomaya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Young Choon Lee.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, W., Lee, Y.C., Fekete, A. et al. Adaptive multiple-workflow scheduling with task rearrangement. J Supercomput 71, 1297–1317 (2015). https://doi.org/10.1007/s11227-014-1361-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-014-1361-0

Keywords

Search

Navigation