Abstract
In data-intensive applications, such as high-energy physics, bio-informatics, we encounter applications involving numerous jobs that access and generate large datasets. Effective scheduling such applications is challenging, due to a need to consider for both computational resources and data storage resources. In this paper, we describe an adaptive scheduling model that consider availability of computational, storage and network resources. Based on this model we implement a scheduler used in our campus grid. The results achieved by our scheduler have been analyzed by comparing Greedy algorithm that is widely used in computational grids and some data grids.
This paper is supported by National Science Foundation under grant 60125208 and 60273076.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Foster, I., Kesselman, C., Tuecke, S.: The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International J. Supercomputer Applications (2001)
Chervenak, A., Foster, I., Kesselman, C., Salisbury, C., Tuecke, S.: The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Datasets. Journal of Network and Computer Applications (2001)
Ranganathan, K., Foster, I.: Decoupling Computation and Data Scheduling in Distributed Data-Intensive Applications. In: Proceedings of 11th IEEE International Symposium on High Performance Distributed Computing (HPDC-11), Edinburgh, Scotland (July 2002)
Jin, H., Zou, D., Wu, S., Chen, H.: Grid Fault-Tolerant Architecture and Practice. Journal of Computer Science and Technology (JCST) (4) (2003)
Hoschek, W., Jaen-Martinez, J., Samar, A., Stockinger, H., Stockinger, K.: Data management in an international data grid project. In: Buyya, R., Baker, M. (eds.) GRID 2000. LNCS, vol. 1971, pp. 77–90. Springer, Heidelberg (2000)
Thain, D., Bent, J., Arpaci-Dusseau, A., Arpaci-Dusseau, R., Livny, M.: Gathering at the Well: Creating Communities for Grid I/O. In: Proceedings of Supercomputing 2001, Denver, Colorado (November 2001)
Basney, J., Livny, M., Mazzanti, P.: Utilizing Widely Distributed Computational Resources Efficiently with Execution Domains. Computer Physics Communications (2000)
Berman, F.: High Performance Schedulers. In: Foster, I., Kesselman, C. (eds.) The Grid: Blueprint for a New Computing Infrastructure, pp. 279–309. Morgan-Kaufmann, San Francisco (1999)
Park, S.-M., Kim, J.-H.: Chameleon: A Resource Scheduler in A Data Grid Environment. In: Proceedings of the 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, Tokyo (2003)
Wolski, R., Spring, N., Hayes, J.: The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing. Journal of Future Generation Computing Systems 15 (October 1999)
Stockinger, H., Stockinger, K., Schikuta, E., Willers, I.: Towards a Cost Model for Distributed and Replicated Data Stores. In: Proceedings of 9th Euromicro Workshop on Parallel and Distributed Processing PDP 2001, Mantova (2001)
Takefusa, A., Casanova, H., Matsuoka, S., Berman, F.: A Study of Deadline Scheduling for Client-Server Systems on the Computational Grid. In: Proceedings of 10th IEEE International Symposium on High Performance Distributed Computing, HPDC-10 (2001)
Condor, http://www.cs.wisc.edu/condor/
Data Grid Project WP1, Definition of Architecture, Technical Plan and Evaluation Criteria for Scheduling, Resource Management, Security and Job Description, Datagrid document DataGrid-01-D1.2-0112-0-3, 14/09/2001
Application Level Scheduling (AppLeS), http://apples.ucsd.edu/
Buyya, R., Abramson, D., Giddy, J.: Nimrod-G Resource Broker for Service-Oriented Grid Computing. IEEE Distributed Systems Online 2(7) (November 2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Shi, X., Jin, H., Qiang, W., Zou, D. (2004). An Adaptive Meta-scheduler for Data-Intensive Applications. In: Li, M., Sun, XH., Deng, Q., Ni, J. (eds) Grid and Cooperative Computing. GCC 2003. Lecture Notes in Computer Science, vol 3033. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24680-0_132
Download citation
DOI: https://doi.org/10.1007/978-3-540-24680-0_132
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-21993-4
Online ISBN: 978-3-540-24680-0
eBook Packages: Springer Book Archive