Abstract
With the increasing number of scientific applications manipulating huge amounts of data, effective high-level data management is an increasingly important problem. Unfortunately, so far the solutions to the high‐level data management problem either require deep understanding of specific storage architectures and file layouts (as in high-performance file storage systems) or produce unsatisfactory I/O performance in exchange for ease-of-use and portability (as in relational DBMSs). In this paper we present a novel application development environment which is built around an active meta-data management system (MDMS) to handle high-level data in an effective manner. The key components of our three-tiered architecture are user application, the MDMS, and a hierarchical storage system (HSS). Our environment overcomes the performance problems of pure database-oriented solutions, while maintaining their advantages in terms of ease-of-use and portability. The high levels of performance are achieved by the MDMS, with the aid of user-specified, performance-oriented directives. Our environment supports a simple, easy-to-use yet powerful user interface, leaving the task of choosing appropriate I/O techniques for the application at hand to the MDMS. We discuss the importance of an active MDMS and show how the three components of our environment, namely the application, the MDMS, and the HSS, fit together. We also report performance numbers from our ongoing implementation and illustrate that significant improvements are made possible without undue programming effort.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
C. Baru, R. Moore, A. Rajasekar and M. Wan, The SDSC storage resource broker, in: Proc. of CASCON'98 Conference, Toronto, Canada (December 1998).
R. Bordawekar, J.M. del Rosario and A. Choudhary, Design and implementation of primitives for Parallel I/O, in: Proc. of Supercomputing' 93 (November 1993).
R. Bordawekar, A. Choudhary, K. Kennedy, C. Koelbel and M. Paleczny, A model and compilation strategy for out-of-core data parallel programs, Proc. of the ACM Symposium on Principles and Practice of Parallel Programming (July 1995) pp. 1-10.
P. Brown, R. Troy, D. Fisher, S. Louis, J.R. McGraw and R. Musick, Meta-data sharing for balanced performance, in: Proc. of the 1st IEEE Meta-data Conference, Silver Spring, MD (1996).
P. Cao, E. Felten and K. Li, Application-controlled file caching policies, in: Proc. of the 1994 Summer USENIX Technical Conference (June 1994) pp. 171-182.
A. Choudhary, R. Bordawekar, M. Harry, R. Krishnaiyer, R. Ponnusamy, T. Singh and R. Thakur, PASSION: parallel and scalable software for input-output, NPAC Technical Report SCCS-636 (September 1994).
A. Choudhary and M. Kandemir, System-level meta-data for high performance data management, Proc. of the 3rd IEEE Meta-Data Conference, Bethesda, MD (April 6-7, 1999).
P.F. Corbett, D.G. Feitelson, J.-P. Prost and S.J. Baylor, Parallel access to files in the Vesta file system, in: Proc. of Supercomputing'93 (November 1993) pp. 472-481.
P. Corbett, D. Fietelson, S. Fineberg, Y. Hsu, B. Nitzberg, J. Prost, M. Snir, B. Traversat and P. Wong, Overview of the MPI-IO parallel I/O interface, in: Proc. of 3rd Workshop on I/O in Parallel and Distributed Systems, IPPS'95, Santa Barbara, CA (April 1995).
T.H. Cormen and D.M. Nicol, Out-of-core FFTs with parallel disks, ACM SIGMETRICS Performance Evaluation Review 25(3) (December 1997) 3-12.
R.A. Coyne, H. Hulen and R. Watson, The high performance storage system, in: Proc. of Supercomputing 93, Portland, OR (November 1993).
P.E. Crandall, R.A. Aydt, A.A. Chien and D.A. Reed, Input/output characteristics of scalable parallel applications, in: Proc. of Supercomputing' 95.
J.R. Davis, Datalinks: Managing external data with DB2 universal database, IBM Corporation White Paper (August 1997).
J. del Rosario, R. Bordawekar and A. Choudhary, Improved parallel I/O via a two-phase run-time access strategy, in: Proc. of the 1993 IPPS Workshop on Input/Output in Parallel Computer Systems (April 1993).
J. del Rosario and A. Choudhary, High performance I/O for parallel computers: problems and prospects, IEEE Computer (March 1994).
M. Drewry, H. Conover, S. McCoy and S. Graves, Meta-data: quality vs. quantity, in: Proc. of the 2nd IEEE Meta-data Conference, Silver Spring, MD (1997).
C.S. Ellis and D. Kotz, Prefetching in file systems for MIMD multiprocessors, in: Proc. of the 1989 International Conference on Parallel Processing, St. Charles, IL, August 1989 (Pennsylvania State Univ. Press, 1989) pp. I:306-314.
M. Kandaswamy, M. Kandemir, A. Choudhary and D. Bernholdt, Performance implications of architectural and software techniques on I/O-intensive applications, in: Proc. of the International Conference on Parallel Processing (ICPP'98), Minneapolis, MN (August 1998).
J.F. Karpovich, A.S. Grimshaw and J.C. French, Extensible file systems (ELFS): An object-oriented approach to high performance file I/O, in: Proc. of the Ninth Annual Conference on Object-Oriented Programming Systems, Languages, and Applications (October 1994) pp. 191-204.
D. Kotz, Disk-directed I/O for MIMD multiprocessors, in: Proc. of the 1994 Symposium on Operating Systems Design and Implementation, USENIX Association (November 1994) pp. 61-74.
D. Kotz, Multiprocessor file system interfaces, in: Proc. of the 2nd International Conference on Parallel and Distributed Information Systems, IEEE Computer Society Press (1993) pp. 194-201.
T. Madhyastha and D. Reed, Intelligent, adaptive file system policy selection, in: Proc. of Frontiers of Massively Parallel Computing, Annapolis, MD (October 1996) pp. 172-179.
G. Memik, M. Kandemir and A. Choudhary, A run-time library for tape resident data, Technical Report CPDC-TR-9909-014, Center for A. Choudhary et al. / Data management for large-scale scientific computations 59 Parallel and Distributed Computing, Northwestern University (September 1999).
R.H. Patterson, G.A. Gibson and M. Satyanarayanan, A status report on research in transparent informed prefetching, ACM Operating Systems Review V 27(2) (April 1993) pp. 21-34.
B. Rullman, Paragon parallel file system, External Product Specification, Intel Supercomputer Systems Division.
K.E. Seamons and M. Winslett, Multidimensional array I/O in Panda 1.0, Journal of Supercomputing 10(2) (1996) 191-211.
M. Stonebraker, Object-Relational DBMSs: Tracking the Next Great Wave (Morgan Kaufman, San Mateo, CA, 1998).
R. Thakur, W. Gropp and E. Lusk, An experimental evaluation of the parallel I/O systems of the IBM SP and Intel Paragon using a production application, in: Proc. of the 3rd Int'l Conf. of the Austrian Center for Parallel Computation (ACPC) with special emphasis on Parallel Databases and Parallel I/O, September 1996, Lecture Notes in Computer Science, Vol. 1127 (Springer-Verlag, 1996) pp. 24-35.
R. Thakur, W. Gropp and E. Lusk, Data sieving and collective I/O in ROMIO, Proc. of the 7th Symposium on the Frontiers of Massively Parallel Computation (February 1999).
S. Toledo and F.G. Gustavson, The design and implementation of SOLAR, a portable library for scalable out-of-core linear algebra computations, in: Proc. of 4th Annual Workshop on I/O in Parallel and Distributed Systems (May 1996).
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Choudhary, A., Kandemir, M., No, J. et al. Data management for large‐scale scientific computations in high performance distributed systems. Cluster Computing 3, 45–60 (2000). https://doi.org/10.1023/A:1019063700437
Issue Date:
DOI: https://doi.org/10.1023/A:1019063700437