The Journal of Supercomputing

, Volume 71, Issue 4, pp 1297–1317 | Cite as

Adaptive multiple-workflow scheduling with task rearrangement

  • Wei Chen
  • Young Choon Lee
  • Alan Fekete
  • Albert Y. Zomaya


Large-scale distributed computing systems like grids and more recently clouds are a platform of choice for many resource-intensive applications. Workflow applications account for the majority of these applications, particularly in science and engineering. A workflow application consists of multiple precedence-constrained tasks with data dependencies. Since resources in those systems are shared by many users and applications deployed there are very diverse, scheduling is complicated. Often, the actual execution of applications differs from the original schedule following delays such as those caused by resource contention and other issues in performance prediction. These delays have further impact when running multiple workflow applications due to inter-task dependencies. In this paper, we investigate the problem of scheduling multiple workflow applications concurrently, explicitly taking into account scheduling robustness. We present a dynamic task rearrangement and rescheduling algorithm that exploits the scheduling flexibility from precedence constraints among tasks. The algorithm optimizes resource allocation among multiple workflows, and it often stops the influence of delayed execution passing to subsequent tasks. The experimental results demonstrate that our approach can significantly improve performance in multiple-workflow scheduling.


Scheduling Workflow applications Workflow scheduling  Rescheduling 



The work of A. Zomaya is supported by the Australian Research Council Discovery Grant DP1097111. This paper revises and incorporates material from [18, 19].


  1. 1.
    Oinn T et al (2004) Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics 20(17):3045–3054CrossRefGoogle Scholar
  2. 2.
    Maechling P et al (2005) Simplifying construction of complex workflows for non-expert users of the Southern California Earthquake Center community modeling environment. ACM SIGMOD Rec 34(3):24–30CrossRefGoogle Scholar
  3. 3.
    Plale B et al (2005) Towards dynamically adaptive weather analysis and forecasting in LEAD. In: Proceedings of int’l conf. on computational science, workshop on dynamic data driven applications, pp 624–631Google Scholar
  4. 4.
    Greenberg A, Hamilton J, Maltz DA, Patel P (2008) The cost of a cloud: research problems in data center networks. ACM SIGCOMM Comput Commun Rev 39(1):68–73CrossRefGoogle Scholar
  5. 5.
    Topcuoglu H, Hariri S, Wu M (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260–274CrossRefGoogle Scholar
  6. 6.
    Mandal A et al (2005) Scheduling strategies for mapping application workflows onto the grid. In: Proceedings of IEEE int’l symp. on high performance distributed computing, pp 125–134Google Scholar
  7. 7.
    Rahman M, Venugopal S, Buyya R (2007) A dynamic critical path algorithm for scheduling scientific workflow applications on global grids. In: Proceedings of IEEE int’l conf. on e-science and grid computing, pp 35–42Google Scholar
  8. 8.
    Cao H, Jin H, Wu X, Wu S, Shi X (2008) DAGMap: efficient scheduling for DAG grid workflow job. In: Proceedings of ACM/IEEE int’l conf. on grid computing, pp 17–24Google Scholar
  9. 9.
    Blythe J et al (2005) Task scheduling strategies for workflow-based applications in grids. In: Proceedings of IEEE int’l symp. on cluster computing and the grid, pp 759–767Google Scholar
  10. 10.
    Yu J, Kirley M, Buyya R (2007) Multi-objective planning for workflow execution on grids. In: Proceedings of IEEE/ACM int’l conf. on grid computing, pp 10–17Google Scholar
  11. 11.
    Lee YC, Subrata R, Zomaya AY (2009) On the performance of a dual-objective optimization model for workflow applications on grid platforms. IEEE Trans Parallel Distrib Syst 20(9):1273–1284CrossRefGoogle Scholar
  12. 12.
    Fahringer T et al (2005) ASKALON: a tool set for cluster and grid computing. Concurr Comput Pract Exp 17(2–4):143–169CrossRefGoogle Scholar
  13. 13.
    Decker J, Schneider J (2007) Heuristic scheduling of grid workflows supporting co-allocation and advance reservation. In: Proceedings of IEEE int’l symp. on cluster computing and the grid, pp 335–342Google Scholar
  14. 14.
    Wieczorek M et al (2006) Applying Advance reservation to increase predictability of workflow execution on the grid. In: Proceedings of IEEE international conference on e-science and grid computing, pp 82–90Google Scholar
  15. 15.
    Smith W, Foster I, Taylor V (2000) Scheduling with advanced reservations. In: Proceedings of IEEE international symposium on parallel and distributed processing, pp 127–132Google Scholar
  16. 16.
    Curino C et al (2014) Reservation-based scheduling: if you’re late don’t blame us!. In: Proceedings of ACM symp. on cloud, computing, pp 1–14Google Scholar
  17. 17.
    Majumdar S (2009) The any-schedulability criterion for providing QoS guarantees through advance reservation requests. In: Proceedings of IEEE/ACM international symposium on cluster computing and the grid, pp 490–495Google Scholar
  18. 18.
    Chen W, Fekete A, Lee YC (2010) Exploiting Deadline Flexibility in Grid Workflow Rescheduling” deadline flexibility in grid workflow rescheduling. In: Proceedings of ACM/IEEE int’l conf. on grid computing, pp 105–112Google Scholar
  19. 19.
    Chen W (2012) High performance multiple-workflow scheduling using task rearrangement. PhD thesis, University of SydneyGoogle Scholar
  20. 20.
    Smith W, Foster I, Taylor V (2004) Predicting application run times with historical information. J Parallel Distrib Comput 64(9):1007–1016CrossRefMATHGoogle Scholar
  21. 21.
    Duan R, Nadeem F, Wang J, Zhang Y, Prodan R, Fahringer T (2009) A hybrid intelligent method for performance modeling and prediction of workflow activities in grids. In: Proceedings of IEEE int’l symp. on cluster computing and the grid, pp 339–347Google Scholar
  22. 22.
    Kleinberg J, Tardos E (2006) Algorithm design. Pearson/Addison-Wesley, USAGoogle Scholar
  23. 23.
    Mu’alem AW, Feitelson DG (2001) Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling. IEEE Trans Parallel Distrib Syst 12(6):529–543CrossRefGoogle Scholar
  24. 24.
    Netto MAS, Buyya R (2008) Rescheduling co-allocation requests based on flexible advance reservations and processor remapping. In: Proceedings of IEEE/ACM int’l conf. on grid computing,break pp 144–151Google Scholar
  25. 25.
    Zhao H, Sakellariou R (2006) Scheduling multiple DAGs onto heterogeneous systems. In: Proceedings of the 15th heterogeneous computing workshopGoogle Scholar
  26. 26.
    Yu Z, Shi W (2008) A planner-guided scheduling strategy for multiple workflow applications. In: Proceedings of int’l conf. on parallel processing-workshops, pp 1–8Google Scholar
  27. 27.
    Mao M, Humphrey M (2011) Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In: Proceedings of 2011 int’l conf. for high performance computing, networking, storage and analysis (SC), pp 49:1–49:12Google Scholar
  28. 28.
    Malawski M, Juve G, Deelman E, Nabrzyski J (2012) Cost-and deadline-constrained provisioning for scientific workflow ensembles in iaas clouds. In: Proc. of int’l conf. on high performance computing, networking, storage and analysis, p 22Google Scholar
  29. 29.
    Ali S, Maciejewski AA, Siegel HJ, Kim J-K (2004) Measuring the robustness of a resource allocation. IEEE Trans Parallel Distrib Syst 15(7):630–641CrossRefGoogle Scholar
  30. 30.
    Shestak V, Smith J, Maciejewski AA, Siegel HJ (2008) Stochastic robustness metric and its use for static resource allocations. J Parallel Distrib Comput 68(8):1157–1173CrossRefMATHGoogle Scholar
  31. 31.
    Shi Z, Jeannot E, Dongarra JJ (2006) Robust task scheduling in non-deterministic heterogeneous computing systems. In: Proceedings of IEEE int’l conf. on cluster computing, pp 1–10Google Scholar
  32. 32.
    Sakellariou R, Zhao H (2004) A low-cost rescheduling policy for efficient mapping of workflows on grid systems. Sci Program 12(4):253–262Google Scholar
  33. 33.
    Yu Z, Shi W (2007) An adaptive rescheduling strategy for grid workflow applications. In: Proc. of IEEE int’l symp. on parallel and distributed processingGoogle Scholar
  34. 34.
    Zhang Y, Koelbel C, Cooper K (2009) Hybrid re-scheduling mechanisms for workflow applications on multi-cluster. In: Proc. of IEEE int’l symp. on cluster computing and the grid, pp 116–123Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Wei Chen
    • 1
  • Young Choon Lee
    • 1
  • Alan Fekete
    • 1
  • Albert Y. Zomaya
    • 1
  1. 1.School of Information TechnologiesThe University of SydneySydneyAustralia

Personalised recommendations