A Hybrid Mapping and Scheduling Algorithm for Distributed Workflow Applications in a Heterogeneous Computing Environment

  • Mengxia Zhu
  • Fei Cao
  • Jia Mi
Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 382)

Abstract

Computing intensive scientific workflows structured as a directed acyclic graph (DAG) are widely applied to various distributed science and engineering applications to enable efficient knowledge discovery by automated data processing. Effective mapping and scheduling the workflow modules to the underlying distributed computing environment with heterogeneous resources for optimal network performance has remained as a challenge and attracted research efforts with many simulations and real experiments carried out in the grid and cloud infrastructures. Due to the computing intractability of this type of optimization problem, heuristic algorithms are commonly proposed to achieve the minimum end-to-end delay (EED) or other objectives such as maximum reliability and stability. In this paper, a Hybrid mapping algorithm combining Recursive Critical Path search and layer-based Priority techniques (HRCPP) is designed and developed to achieve the minimum EED. Four representative mapping and scheduling algorithms for minimum EED are compared with HRCPP. Our simulation results illustrate that HRCPP consistently achieves the smallest EED with a low algorithm running time observed from many different scales of simulated test cases.

Keywords

Schedule Algorithm Directed Acyclic Graph Critical Path Task Graph Hybrid Mapping 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Kwok, Y., Ahmad, I.: Dynamic critical-path scheduling: An effective technique for allocating task graph to multiprocessors. IEEE Trans. on Parallel and Distributed Systems 7(5) (May 1996)Google Scholar
  2. 2.
    Wu, Q., Gu, Y.: Supporting distributed application workflows in heterogeneous computing environments. In: Proc. of the 14th IEEE Int. Conf. on Parallel and Distributed Systems, Melbourne, Australia, pp. 3–10 (December 2008)Google Scholar
  3. 3.
    Sekhar, A., Manoj, B., Murthy, C.: A state-space search approach for optimizing reliability and cost of execution in distributed sensor networks. In: Proc. of Int. Workshop on Distributed Computing, pp. 63–74 (2005)Google Scholar
  4. 4.
    Agarwalla, B., Ahmed, N., Hilley, D., Ramachandran, U.: Streamline: a scheduling heuristic for streaming application on the grid. In: The 13th Multimedia Computing and Networking Conf., San Jose, CA (2006)Google Scholar
  5. 5.
    Wu, M.Y., Gajski, D.D.: Hypertool: A programming aid for message-passing systems. IEEE Trans. on Parallel and Distributed Systems 1(3), 330–343 (1990)CrossRefGoogle Scholar
  6. 6.
    Wang, L., Siegel, H.J., Roychowdhury, V.P., Maciejewski, A.A.: Task matching and scheduling in heterogeneous computing environment using a genetic-algorithm-based approach. Journal Parallel and Distributed Computing 4, 175–187 (1997)Google Scholar
  7. 7.
    Carter, B.R., Watson, D.W., Freund, R.F., Keith, E., Mirabile, F., Siegel, H.J.: Generational scheduling for dynamic task management in heterogeneous computing systems. Information Science 106(3-4), 219–236 (1998)CrossRefGoogle Scholar
  8. 8.
    Ma, T., Buyya, R.: Critical-path and priority based algorithms for scheduling workflows with parameter sweep tasks on global grids. In: Proc. of the 17th Int. Symp. on Computer Architecture on High Performance Computing, pp. 251–258 (2005)Google Scholar
  9. 9.
    Kwok, Y.K., Ahmad, I.: Static scheduling algorithms for allocating directed task graphs to multiprocessors. ACM Computing Surveys 31(4), 406–471 (1999)CrossRefGoogle Scholar
  10. 10.
    Afrati, F.N., Papadimitriou, C.H., Papageorgiou, G.: Scheduling dags to minimize time and communication. In: Proc. of the 3rd Aegean Workshop on Computing: VLSI Algorithms and Architectures, pp. 134–138. Springer, Heidelberg (1988)Google Scholar
  11. 11.
    Topcuoglu, H., Hariri, S., Wu, M.: Performance effective and lowcomplexity task scheduling for heterogeneous computing. IEEE Trans. on Parallel and Distributed Systems 13(3) (2002)Google Scholar
  12. 12.
    Cordella, L., Foggia, P., Sansone, C., Vento, M.: An improved algorithm for matching large graphs. In: Proc. of the 3rd IAPR-TC-15 Int. Workshop on Graph-based Representations, Italy (2001)Google Scholar
  13. 13.
    Rahman, M., Venugopal, S., Buyya, R.: Ant colony system: A dynamic critical path algorithm for scheduling scientific workflow applications on global grids. In: Proc. of the 3rd IEEE Int. Conf. on e-Science and Grid Computing, pp. 35–42 (2007)Google Scholar
  14. 14.
    Boeres, C., Filho, J., Rebello, V.: A cluster-based strategy for scheduling task on heterogeneous processors. In: in Proc. of 16th Symp. on Computer Architecture and High Performance Computing, pp. 214–221 (2004)Google Scholar
  15. 15.
    Rahman, M., Venugopal, S., Buyya, R.: A dynamic critical path algorithm for scheduling scientific workflow applications on global grids. In: In Proc. of the 3rd IEEE Int. Conf. on e-Science and Grid Computing, pp. 35–42 (2007)Google Scholar
  16. 16.
    Wu, Q., Gu, Y., Lin, Y., Rao, N.S.V.: Latency Modeling and Minimization for Large-scale Scientific Workflows in Distributed Network Environments. In: Proc. of the 44th Annual Simulation Symposium (ANSS 2011), Boston, MA, USA, April 4-7 (2011)Google Scholar
  17. 17.
    Zhu, M., Wu, Q., Rao, N.S.V., Iyengar, S.S.: Adaptive visualization pipeline decomposition and mapping onto computer networks. In: Proc. of the IEEE Internatioal Conference on Image and Graphics, Hong Kong, China, December 18-20, pp. 402–405 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Mengxia Zhu
    • 1
  • Fei Cao
    • 1
  • Jia Mi
    • 1
  1. 1.Computer Science DepartmentSouthern Illinois UniversityCarbondaleUSA

Personalised recommendations