Journal of Grid Computing

, Volume 2, Issue 4, pp 353–367 | Cite as

The DataGrid Workload Management System: Challenges and Results

  • G. Avellino
  • S. Beco
  • B. Cantalupo
  • A. Maraschini
  • F. Pacini
  • M. Sottilaro
  • A. Terracina
  • D. Colling
  • F. Giacomini
  • E. Ronchieri
  • A. Gianelle
  • M. Mazzucato
  • R. Peluso
  • M. Sgaravatto
  • A. Guarise
  • R. Piro
  • A. Werbrouck
  • D. Kouřil
  • A. Křenek
  • L. Matyska
  • M. Mulač
  • J. Pospíšil
  • M. Ruda
  • Z. Salvet
  • J. Sitera
  • J. Škrabal
  • M. Voců
  • M. Mezzadri
  • F. Prelz
  • S. Monforte
  • M. Pappalardo
Article

Abstract

The workload management task of the DataGrid project was mandated to define and implement a suitable architecture for distributed scheduling and resource management in a Grid environment. The result was the design and implementation of a Grid Workload Management System, a super-scheduler with the distinguishing property of being able to take data access requirements into account when scheduling jobs to the available Grid resources. Many novel issues in various fields were faced such as resource management, resource reservation and co-allocation, Grid accounting. In this paper, the architecture and the functionality provided by the DataGrid Workload Management System are presented.

Keywords

Grid scheduling distributed resource management Grid workload management 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    “Home page of the DataGrid project”, http://www.edg.org
  2. 2.
    D. Abramson, J. Giddy and R. Buyya, “An Economy Driven Resource Management Architecture for Global Computational Power Grids”, in Proc. of the 7th International International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA 2000), Las Vegas, USA, 2000. Google Scholar
  3. 3.
    R. Alfieri et al., “VOMS, an Authorization System for Virtual Organizations”, in Grid Computing, First European Across Grids Conference, 2004. Google Scholar
  4. 4.
    S. Andreozzi, M. Sgaravatto and C. Vistoli, “Sharing a Conceptual Model of Grid Resources and Services”, in Proceedings of the 2003 Computing in High Energy and Nuclear Physics Conference (CHEP03), La Jolla, CA, USA, March 2003. Google Scholar
  5. 5.
    C. Anglano et al., “Integrating Grid Tools to Build a Computing Resource Broker: Activities of DataGrid WP1”, in Proceedings of the 2001 Computing in High Energy and Nuclear Physics Conference (CHEP01), Beijing, China, September 2001. Google Scholar
  6. 6.
    G. Avellino et al., “The EU DataGrid Workload Management System: Towards the Second Major Release”, in Proceedings of the 2003 Computing in High Energy and Nuclear Physics Conference (CHEP03), La Jolla, CA, USA, March 2003. Google Scholar
  7. 7.
    G. Avellino et al., “The First Deployment of Workload Management Services on the EU DataGrid Testbed: Feedback on Design and Implementation”, in Proceedings of the 2003 Computing in High Energy and Nuclear Physics Conference (CHEP03), La Jolla, CA, USA, March 2003. Google Scholar
  8. 8.
    A. Barmouta and R. Buyya, “GridBank: A Grid Accounting Services Architecture (GASA) for Distributed System Sharing and Integration”, in Proceedings of the 17th Annual International Parallel & Distributed Processing Symposium (IPDPS 2003) Workshop on Internet Computing and E-Commerce. Google Scholar
  9. 9.
    G. Bronevetsky et al., “Automated Aplication-Level Checkpointing of MPI Programs”, in ACM Symposium on Principles and Practice of Parallel Programming, 2003. Google Scholar
  10. 10.
    A. Cooke et al., “Relational Grid Monitoring Architecture (R-GMA)”, Presented at UK e-Science All-Hands meeting, Nottingham, UK, 2003. https://edms.cern.ch/file/400756/1/rgma.pdf
  11. 11.
    K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W. Smith and S. Tuecke, “A Resource Management Architecture for Metacomputing Systems”, Lecture Notes in Computer Science, Vol. 1459, 1998. Google Scholar
  12. 12.
    H. Dail et al., “Scheduling in the Grid Application Development Software Project”, in [24], pp. 73–98. Google Scholar
  13. 13.
    E. Deelman, J. Blythe, Y. Gil and C. Kesselman, “Workflow Management in GriPhyn”, in [24], pp. 99–118. Google Scholar
  14. 14.
    D.F. Ferguson et al., “Economic Models for Allocating Resources in Computer Systems”, in Market-Based Control: A Paradigm for Distributed Resource Allocation. World Scientific: Hong Kong, 1996. Google Scholar
  15. 15.
    I. Foster and C. Kesselman, “The Globus Toolkit”, in I. Foster and C. Kesselman (eds), The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann: San Francisco, CA, 1999, Chap. 11, pp. 259–278. Google Scholar
  16. 16.
    I. Foster, C. Kesselman, G. Tsudik and S. Tuecke, “A Security Architecture for Computational Grids”, in Proc. 5th ACM Conference on Computer and Communications Security Conference, 1998. Google Scholar
  17. 17.
    J. Frey, T. Tannenbaum, I. Foster, M. Livny and S. Tuecke, “Condor-G: A Computation Management Agent for Multi-Institutional Grids”, in Proceedings of the Tenth IEEE Symposium on High Performance Distributed Computing (HPDC). San Francisco, California, 2001, pp. 7–9. Google Scholar
  18. 18.
    F. Giacomini, F. Prelz, M. Sgaravatto, I. Terekhov, G. Garzoglio and T. Tannenbaum, “Planning on the Grid: A Status Report”, Technical Report INFN-TC-02/26, INFN-GRID Project, 2002. Google Scholar
  19. 19.
    S. Jackson et al., “Charter of the Global Grid Forum Usage Records Working Group”, Technical Report, 2003. Google Scholar
  20. 20.
    K. Kurowski, J. Nabrzyski, A. Oleksiak and J. Weglarz, “Multicriteria Aspects of the Grid Resource Management”, in [24], pp. 271–294. Google Scholar
  21. 21.
    A. Křenek and Z. Salvet, “L&B API Reference”, DataGrid-01-TED-0139, 2003, http://lindir.ics.muni.cz/dg_public/
  22. 22.
    M. Litzkow, M. Livny and M. Mutka, “Condor – A Hunter of Idle Workstations”, in Proceedings of the 8th International Conference of Distributed Computing Systems, 1988. Google Scholar
  23. 23.
    M. Milka, G. Waligora and J. Weglarz, “A Metaheuristic Approach to Scheduling Workflow Jobs on a Grid”, in [24], pp. 295–320. Google Scholar
  24. 24.
    J. Nabryzski, J. Schopf and J. Weglarz (eds), Grid Resource Management. Kluwer Academic: Norwell, MA, 2003. Google Scholar
  25. 25.
    J. Novotny, S. Tuecke and V. Welch, “An Online Credential Repository for the Grid: MyProxy”, in Proceedings of the Tenth International Symposium on High Performance Distributed Computing (HPDC-10), 2001. Google Scholar
  26. 26.
    R. Piro, A. Guarise and A. Werbrouck, “An Economy-Based Accounting Infrastructure for the DataGrid”, in Proc. of the 4th International Workshop on Grid Computing (Grid2003), Phoenix, Arizona, USA, 2003. Google Scholar
  27. 27.
    R. Piro, A. Guarise and A. Werbrouck, “Simulation of Price-Sensitive Resource Brokering and the Hybrid Pricing Model with DGAS-Sim”, in Proc. of the 13th International Workshops on Enabling Technologies: Infrastructures for Collaborative Enterprises (WETICE 2004), Modena, Italy, 2004. Google Scholar
  28. 28.
    R. Raman, M. Livny and M. Solomon, “Matchmaking: Distributed Resource Management for High Throughput Computing”, in Proceedings of the Seventh IEEE International Symposium on High Performance Distributed Computing (HPDC7), Chicago, IL, 1998. Google Scholar
  29. 29.
    R. Raman, M. Livny and M. Solomon, “Matchmaking: Distributed Resource Management for High Throughput Computing”, in Proceedings of the Seventh IEEE International Symposium on High Performance Distributed Computing (HPDC7), Chicago, IL, 1998. Google Scholar
  30. 30.
    R. Raman, M. Livny and M. Solomon, “Resource Management through Multilateral Matchmaking”, in Proceedings of the Ninth IEEE Symposium on High Performance Distributed Computing (HPDC9), Pittsburgh, PA, 2000, pp. 290–291. Google Scholar
  31. 31.
    J.M. Schopf, “Ten Actions when SuperScheduling”, Technical Report GFD-I.4, Global Grid Forum, Scheduling Working Group, 2001. Google Scholar
  32. 32.
    D. Simmel et al., “Charter of the Global Grid Forum Grid Checkpoint Recovery Working Group”, Technical Report, 2004. Google Scholar
  33. 33.
    D. Thain and M. Livny, “Bypass: A Tool for Building Split Execution Systems”, in Proceedings of the Ninth IEEE Symposium on High Performance Distributed Computing (HPDC9), Pittsburgh, PA, 2000, pp. 79–85. Google Scholar
  34. 34.
    M.P. Thomas et al., “The GridPort Toolkit: a System for Building Grid Portals”, in Proc. of the Tenth IEEE International Symposium on High Performance Distributed Computing, 2001. Google Scholar

Copyright information

© Springer 2005

Authors and Affiliations

  • G. Avellino
    • 1
  • S. Beco
    • 1
  • B. Cantalupo
    • 1
  • A. Maraschini
    • 1
  • F. Pacini
    • 1
  • M. Sottilaro
    • 1
  • A. Terracina
    • 1
  • D. Colling
    • 2
  • F. Giacomini
    • 3
  • E. Ronchieri
    • 3
  • A. Gianelle
    • 4
  • M. Mazzucato
    • 4
  • R. Peluso
    • 4
  • M. Sgaravatto
    • 4
  • A. Guarise
    • 5
  • R. Piro
    • 5
  • A. Werbrouck
    • 5
  • D. Kouřil
    • 6
  • A. Křenek
    • 6
  • L. Matyska
    • 6
  • M. Mulač
    • 6
  • J. Pospíšil
    • 6
  • M. Ruda
    • 6
  • Z. Salvet
    • 6
  • J. Sitera
    • 6
  • J. Škrabal
    • 6
  • M. Voců
    • 6
  • M. Mezzadri
    • 7
  • F. Prelz
    • 7
  • S. Monforte
    • 8
  • M. Pappalardo
    • 8
  1. 1.DATAMAT S.p.A.RomaItaly
  2. 2.Physics DepartmentImperial College LondonLondonUnited Kingdom
  3. 3.INFN CNAFBolognaItaly
  4. 4.INFN Sezione di PadovaPadovaItaly
  5. 5.INFN Sezione di TorinoTorinoItaly
  6. 6.CESNET z.s.p.o.Praha 6Czech Republic
  7. 7.INFN Sezione di MilanoMilanoItaly
  8. 8.INFN Sezione di CataniaCataniaItaly

Personalised recommendations