Advertisement

Integrating Local Job Scheduler – LSFTM with GfarmTM

  • Xiaohui Wei
  • Wilfred W. Li
  • Osamu Tatebe
  • Gaochao Xu
  • Liang Hu
  • Jiubin Ju
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3758)

Abstract

Applications that both access and generate large data sets increasingly draw our attention in high energy physics, astronomy, genomics and other disciplines. The Data Grids, like Gfarm, seek to harness geographically distributed resources for such large-scale data-intensive problems. However, scheduling is a challenging task in this context. In this paper, we discuss the integration of LSF with Gfarm. We will discuss how to enable LSF to support Gfarm applications requiring GSI authentication, the design and implementation of data aware scheduling and data management. The system is able to find data-affinity hosts for Gfarm jobs and to adjust the distribution of the data replicas dynamically according to the job load. Before job running, the system will setup the proper credential for it. Using the LSF scheduler plugin mechanism, we do not need to write a new scheduler from scratch or make a lot of changes to an existing scheduler.

Keywords

data grid data aware scheduling GSI LSF Gfarm 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Basney, J., Livny, M.: Managing Network Resources in Condor. In: Proceedings of the Ninth IEEE Symposium on High Performance Distributed Computing (HPDC9), Pittsburgh, Pennsylvania, August 2000, pp. 298–299 (2000)Google Scholar
  2. 2.
    Zhou, S., Zheng, X., Wang, J., et al.: Utopia: a Load Sharing Facility for Large, Heterogeneous Distributed Computer Systems. Software—Practice And Experience 23(12), 1305–1336 (1993)CrossRefGoogle Scholar
  3. 3.
    James, P.J.: Portable Batch System: Exterernal Reference Specification Altair PBS Pro 5.3 (March 2003), http://www.mta.ca/torch/pdf/pbspro54/pbsproers.pdf
  4. 4.
    Sun Microsystems, Inc. Sun Grid Engine 5.3 Administration and User’s Guide (April 2002), http://gridengine.sunsource.net/project/gridengine-download/SGE53AdminUserDoc.pdf
  5. 5.
    Frey, J., Tannenbaum, T., Foster, I., et al.: Condor-G: A Computation Management Agent for Multi-Institutional Grids. Journal of Cluster Computing 5, 237–246 (2002)CrossRefGoogle Scholar
  6. 6.
    Platform Computing Co. Open source metascheduling for Virtual Organizations with the Community Scheduler Framework, CSF (2004), http://www.cs.virginia.edu/~grimshaw/CS851-2004/Platform/CSF_architecture.pdf
  7. 7.
    MONARC Collaboration. Models of Networked Analysis at Regional Centres for LHC experiments: Phase 2 report. Technical Report CERN/LCB-001, CERN (2000), http://wwwcern.ch/MONARC/
  8. 8.
    Tatebe, O., Morita, Y., Matsuoka, S., et al.: Grid Datafarm Architecture for Petascale Data Intensive Computing. In: Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, pp. 102–110 (2002)Google Scholar
  9. 9.
    Ranganathan, K., Foster, I.: Decoupling Computation and Data Scheduling in Distributed Data-Intensive Applications. In: Proceedings of 11th IEEE International Symposium on High Performance Distributed Computing (HPDC-11), Edinburgh, Scotland (July 2002), http://www.globus.org/research/papers/decouple.pdf
  10. 10.
    Schintke, F., Schutt, T., Alexander: A Framework for Self-Optimizing Grids Using P2P Components. In: Proceedings of the 14th International Workshop on Database and Expert Systems Applications, DEXA 2003 (2003), http://www.zib.de/reinefeld/Publications/dexa03.pdf
  11. 11.
    Blythe, J., Deelman, E., Gil, Y., et al.: The Role of Planning in Grid Computing. In: 13th International Conference on Automated Planning and Scheduling (ICAPS), Trento, Italy (June 2003), http://www.isi.edu/~gil/papers/icaps03-submission.pdfGoogle Scholar
  12. 12.
    Sakae, Y., et al.: Preliminary Evaluation of Dynamic Load Balancing Using Loop Re-partitioning on Omni/SCASH. In: The 3rd International Symposium on Cluster Computing and the Grid, Tokyo, Japan, May 2003, pp. 463–471 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Xiaohui Wei
    • 1
  • Wilfred W. Li
    • 2
  • Osamu Tatebe
    • 3
  • Gaochao Xu
    • 1
  • Liang Hu
    • 1
  • Jiubin Ju
    • 1
  1. 1.College of Computer Science and TechnologyJilin UniversityChangchunPRC
  2. 2.University of CaliforniaSan DiegoUSA
  3. 3.Grid Technology Research Center, AISTTsukuba, IbarakiJapan

Personalised recommendations