Journal of Grid Computing

, Volume 5, Issue 2, pp 197–212 | Cite as

A Tool for Prioritizing DAGMan Jobs and its Evaluation

  • Grzegorz MalewiczEmail author
  • Ian Foster
  • Arnold L. Rosenberg
  • Michael Wilde


It is often difficult to perform efficiently a collection of jobs with complex job dependencies due to temporal unpredictability of the Grid. One way to mitigate the unpredictability is to schedule job execution in a manner that constantly maximizes the number of jobs that can be sent to workers. A recently developed scheduling theory provides a basis to meet that optimization goal. Intuitively, when the number of such jobs is always large, high parallelism can be maintained, even if the number of workers changes over time in an unpredictable manner. In this paper we present the design, implementation, and evaluation of a practical scheduling tool inspired by the theory. Given a DAGMan input file with interdependent jobs, the tool prioritizes the jobs. The resulting schedule significantly outperforms currently used schedules under a wide range of system parameters, as shown by simulation studies. For example, a scientific data analysis application, AIRSN, was executed at least 13% faster with 95% confidence. An implementation of the tool was integrated with the Condor high-throughput computing system.

Key words

scheduling dags unpredictability systems Grid Condor applications of theory 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aho, A.V., Garey, M.R., Ullman, J.D.: The transitive reduction of a directed graph. SIAM J. Comput. 1, 131–137 (1972)zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Annis, J., Zhao, Y., Voeckler, J., Wilde, M., Kent, S., Foster, I.: Applying chimera virtual data concepts to cluster finding in the sloan sky survey. ACM/IEEE Conference on Supercomputing, pp. 1–14 (2002)Google Scholar
  3. 3.
    Berriman, B., Bergou, A., Deelman, E., Good, J., Jacob, J., Katz, D., Kesselman, C., Laity, A., Singh, G., Su, M.-H., Williams, R.: Montage: a Grid-enabled image mosaic service for the NVO. Astron. Data Anal. Softw. Syst. (2003)Google Scholar
  4. 4.
    Buyya, R., Abramson, A., Giddy, J.: A case for economy Grid architecture for service oriented Grid computing. In: 10th Heterogeneous Computing Wkshp. (2001)Google Scholar
  5. 5.
    Cohen, P.R.: Empirical Methods for Artificial Intelligence. MIT, Cambridge, MA (1995)zbMATHGoogle Scholar
  6. 6.
    Cordasco, G., Malewicz, G., Rosenberg, A.L.: Applying IC-scheduling theory to familiar classes of computations. In: Workshop on Large-scale and Volatile Desktop Grids (2007)Google Scholar
  7. 7.
    Cordasco, G., Malewicz, G., Rosenberg, A.L.: Extending IC-scheduling theory via the sweep algorithm. Typescript, Univ. Massachusetts (2006)Google Scholar
  8. 8.
    Cordasco, G., Malewicz, G., Rosenberg, A.L.: Advances in IC-scheduling theory: scheduling expansive and reductive dags and scheduling dags via duality. IEEE Trans. Parallel Distrib. Syst. (2007) (to appear)Google Scholar
  9. 9.
    Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms (2nd edn.). MIT, Cambridge, MA (2001)zbMATHGoogle Scholar
  10. 10.
    Deelman, E., Kesselman, C., Mehta, G., Meshkat, L., Pearlman, L., Blackburn, K., Ehrens, P., Lazzarini, A., Williams, R., Koranda, S.: GriPhyN and LIGO, building a virtual data Grid for gravitational wave scientists. In: Proc. IEEE Int. Symp. High Perform. Distrib. Comput. (HPDC’02), pp. 225–234 (2002)Google Scholar
  11. 11.
    Foster, I., Kesselman, C. (eds.): The Grid: Blueprint for a New Computing Infrastructure (2nd edn.). Morgan-Kaufmann, San Francisco (2004)Google Scholar
  12. 12.
    Foster, I. et al.: The Grid2003 production Grid: principles and practice. In: 13th IEEE International Symposium on High Performance Distributed Computing (HPDC’04), pp. 236–245 (2004)Google Scholar
  13. 13.
    Hall, R., Rosenberg, A.L., Venkataramani, A.: A comparison of dag-scheduling strategies for Internet-based computing. In: 21st Intl. Parallel and Distributed Processing Symp. (2007)Google Scholar
  14. 14.
    Horn, J.V., Dobson, J., Woodward, J., Wilde, M., Zhao, Y., Voeckler, J., Foster, I.: Grid-based computing and the future of Neuroscience computation. Methods in Mind, MIT, Cambridge, MA (2005)Google Scholar
  15. 15.
    Hsu, H.T.: An algorithm for finding a minimal equivalent graph of a digraph. J. ACM 22, 11–16 (1975)zbMATHCrossRefGoogle Scholar
  16. 16.
    Kondo, D., Casanova, H., Wing, E., Berman, F.: Models and scheduling mechanisms for global computing applications. In: 16th Intl. Parallel and Distributed Processing Symp. (2002)Google Scholar
  17. 17.
    Malewicz, G., Rosenberg, A.L.: On batch-scheduling dags for Internet-based computing. In: 11th European Conf. on Parallel Processing (Euro-Par’05), pp. 262–271 (2005)Google Scholar
  18. 18.
    Malewicz, G., Rosenberg, A.L., Yurkewych, M.: Toward a theory for scheduling dags in internet-based computing. IEEE Trans. Comput. 55(6), 757–768 (2006)CrossRefGoogle Scholar
  19. 19.
    Rosenberg, A.L.: On scheduling mesh-structured computations for Internet-based computing. IEEE Trans. Comput. 53, 1176–1186 (2004)CrossRefGoogle Scholar
  20. 20.
    Rosenberg, A.L., Yurkewych, M.: Guidelines for scheduling some common computation-dags for Internet-based computing. IEEE Trans. Comput. 54, 428–438 (2005)CrossRefGoogle Scholar
  21. 21.
    Szalay, A.S., Kunszt, P.Z., Thakar, A., Gray, J., Slutz, D., Brunner, R.J.: Designing and mining multi-terabyte astronomy archives: the sloan digital sky survey. In: ACM SIGMOD International Conference on Management of Data, pp. 451–462 (2000)Google Scholar
  22. 22.
    Sun, X.-H., Wu, M.: Grid harvest service: a system for long-term, application-level task scheduling. In: IEEE Intl. Parallel and Distributed Processing Symp. (2003)Google Scholar
  23. 23.
    Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurrency and Computation: Practice and Experience 17, 323–356 (2005)CrossRefGoogle Scholar

Copyright information

© Springer Science + Business Media B.V. 2007

Authors and Affiliations

  • Grzegorz Malewicz
    • 1
    Email author
  • Ian Foster
    • 2
    • 4
  • Arnold L. Rosenberg
    • 3
  • Michael Wilde
    • 4
  1. 1.Department of EngineeringGoogle, Inc.Mountain ViewUSA
  2. 2.Department of Computer ScienceUniversity of ChicagoChicagoUSA
  3. 3.Department of Computer ScienceUniversity of Massachusetts AmherstAmherstUSA
  4. 4.Division of Mathematics and Computer ScienceArgonne National LaboratoryArgonneUSA

Personalised recommendations