Skip to main content
SpringerLink
Log in
Book cover

Data Engineering pp 143–168Cite as

Parallel File Systems

  • Chapter
  • First Online:

  • 2782 Accesses

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 132))

Abstract

The success of a CDI Grid is dependent upon the design of its storage infrastructure. As seen in Chapter 7, processing in this environment revolves around the simultaneous movement and transformation of data on many compute elements. Effective storage solutions combine hardware and software to meet these needs. The storage hardware selected must provide enough raw throughput for the expected workloads. Typical storage hardware architectures also often provide some redundancy to help in creating a fault tolerant system. Storage software, specifically file systems, must organize this storage hardware into a single logical space, provide efficient mechanisms for accessing that space, and hide common hardware failures from compute elements.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  • Barroso L, Dean J, Holzle U (2003) Web search for a planet: The Google cluster architecture. In: IEEE Micro, vol. 23, issue 2, pp 22-28.

    Article  Google Scholar 

  • Culler D, Singh J, Gupta A (1999) Parallel Computer Architecture: A Hardware/Software Approach. Morgan Kaufmann, San Francisco, CA.

    Google Scholar 

  • Dean J, Ghemawat S (2004) MapReduce: Simplified data processing on large clusters. In: OSDI’04, 6th Symposium on Operating Systems Design and Implementation, Sponsored by USENIX in cooperation with ACM SIGOPS, pp 137–150.

    Google Scholar 

  • EMC Corporation (2006) Deploying Celerra MPFSi in high-performance computing environments. EMC White Paper.

    Google Scholar 

  • Fridella S, Jiang X, Black D (2003) Elements of a scalable network file system protocol. In: NFS Extensions for Parallel Storage Workshop.

    Google Scholar 

  • Ghemawat S, Gobioff H, Leung S (2003) The Google File System. In: ACM SIGOPS Operating Systems Review Volume 37, Issue 5, pp 29-43.

    Article  Google Scholar 

  • Gropp W, Lusk E, Skjellum A (1999) Using MPI: Portable Parallel Programming with the Message-Passing Interface. MIT Press, Cambridge, MA.

    Google Scholar 

  • The Hadoop Distributed File System, http://lucene.apache.org/hadoop/hdfs_design.html.

  • Hildebrand D, Honeyman P (2005) Exporting storage systems in a scalable manner with pNFS. In: Proceedings of the 22nd IEEE - 13th NASA Goddard (MSST2005) Conference on Mass Storage Systems and Technologies, Monterey, California.

    Google Scholar 

  • IEEE/ANSI Std. 1003.1 (1996) Portable operating system interface (POSIX) part 1: System application program interface (API) [C Language].

    Google Scholar 

  • Message Passing Interface Forum (1997) MPI-2: Extensions to the Message-Passing Interface.

    Google Scholar 

  • Patterson D, Gibson G, Katz R (1988) A case for redundant arrays of inexpensive disks (RAID). In: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp 108-116.

    Google Scholar 

  • Purakayastha A, Ellis C, Kotz D, Nieuwejaar N, Best M (1995) Characterizing parallel file-access patterns on a large-scale multiprocessor. In: Proceedings of the Ninth International Parallel Processing Symposium. pp 165-172.

    Google Scholar 

  • Schroeder B, Gibson G (2007) Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? In: 5th USENIX Conference on File and Storage Technologies.

    Google Scholar 

  • Shepler S, Callaghan B, Robinson D, Thurlow R, Sun Microsystems, Inc., Beame C, Hummingbird Ltd. Eisler M, Noveck D, Network Appliance, Inc. (2003) Network File System (NFS) version 4 protocol. Network Working Group RFC 3530, Internet Engineering Task Force.

    Google Scholar 

  • Thakur R, Gropp W, Lusk E (1999) On implementing MPI-IO portably and with high performance. In: Proceedings of the Sixth Workshop on I/O in Parallel and Distributed Systems, Atlanta, Georgia, pp 23-32.

    Google Scholar 

  • Welch B, Halevy B, Goodson G, Black D, Adamson A (2005) pNFS operations. Internet Engineering Task Force Internet-Draft.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL, USA

    Robert Ross & Philip Carns

  2. Acxiom Corporation Conway, Conway, AR, USA

    David Metheny

Authors
  1. Robert Ross
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Philip Carns
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Metheny
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Dept. Systems Engineering, Donaghey College of Info Sci., University of Arkansas, South University Avenue 2801, Little Rock, 72204-1099, Arkansas, USA

    Yupo Chan

  2. Dept. Information Science, University of Arkansas, Little Rock, South University Ave. 2801, Little Rock, 72204-1099, USA

    John Talburt

  3. Acxiom Corporation, Conway, USA

    Terry M. Talley

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Ross, R., Carns, P., Metheny, D. (2009). Parallel File Systems. In: Chan, Y., Talburt, J., Talley, T. (eds) Data Engineering. International Series in Operations Research & Management Science, vol 132. Springer, Boston, MA. https://doi.org/10.1007/978-1-4419-0176-7_8

Download citation

  • DOI: https://doi.org/10.1007/978-1-4419-0176-7_8

  • Published:

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4419-0175-0

  • Online ISBN: 978-1-4419-0176-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation