Skip to main content

Advertisement

SpringerLink
Log in
Menu
Find a journal Publish with us
Search
Cart
Book cover

IFIP International Conference on Distributed Applications and Interoperable Systems

DAIS 2012: Distributed Applications and Interoperable Systems pp 31–44Cite as

  1. Home
  2. Distributed Applications and Interoperable Systems
  3. Conference paper
Scalability of Replicated Metadata Services in Distributed File Systems

Scalability of Replicated Metadata Services in Distributed File Systems

  • Dimokritos Stamatakis18,
  • Nikos Tsikoudis18,
  • Ourania Smyrnaki18 &
  • …
  • Kostas Magoutis18 
  • Conference paper
  • 840 Accesses

  • 2 Citations

Part of the Lecture Notes in Computer Science book series (LNCCN,volume 7272)

Abstract

There has been considerable interest recently in the use of highly-available configuration management services based on the Paxos family of algorithms to address long-standing problems in the management of large-scale heterogeneous distributed systems. These problems include providing distributed locking services, determining group membership, electing a leader, managing configuration parameters, etc. While these services are finding their way into the management of distributed middleware systems and data centers in general, there are still areas of applicability that remain largely unexplored. One such area is the management of metadata in distributed file systems. In this paper we show that a Paxos-based approach to building metadata services in distributed file systems can achieve high availability without incurring a performance penalty. Moreover, we demonstrate that it is easy to retrofit such an approach to existing systems (such as PVFS and HDFS) that currently use different approaches to availability. Our overall approach is based on the use of a general-purpose Paxos-compatible component (the embedded Oracle Berkeley database) along with a methodology for making it interoperate with existing distributed file system metadata services.

Keywords

  • Directory Object
  • Hadoop Distribute File System
  • Distribute File System
  • Metadata Server
  • Operating System Principle

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Download conference paper PDF

References

  1. Lamport, L.: The Part-Time Parliament. ACM Transactions on Computer Systems (TOCS) 16(2), 133–169 (1998)

    CrossRef  Google Scholar 

  2. Oki, B.M., Liskov, B.H.: Viewstamped Replication: A New Primary Copy Method to Support Highly Available Distributed Systems. In: Proc. of the 7th ACM Symposium on Principles of Distributed Computing (PODC 1988), Toronto, Canada (1988)

    Google Scholar 

  3. Lampson, B.W.: How to Build a Highly Available System using Consensus. In: Babaoğlu, Ö., Marzullo, K. (eds.) WDAG 1996. LNCS, vol. 1151, pp. 1–17. Springer, Heidelberg (1996)

    CrossRef  Google Scholar 

  4. Burrows, M.: The Chubby Lock Service for Loosely-Coupled Distributed Systems. In: Proceedings of OSDI 2006, Seattle, WA (2006)

    Google Scholar 

  5. Junqueira, F., Reed, B.C., Serafini, M.: Zab: High-performance Broadcast for Primary-Backup Systems. In: Proc. of IEEE/IFIP International Conference on Dependable Systems and Networks, Hong Kong, China (2011)

    Google Scholar 

  6. Olson, M.A., Bostic, K., Seltzer, M.I.: Berkeley DB. In: Proceedings of USENIX Annual Technical Conference, FREENIX Track, Monterey, CA (1999)

    Google Scholar 

  7. Perl, S.E., Seltzer, M.I.: Data Management for Internet-Scale Single-Sign-On. In: Proc. of USENIX WORLDS 2006, Seattle, WA (2006)

    Google Scholar 

  8. Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. ACM Transactions on Computer Systems (TOCS) 26(2), 1–26 (2008)

    CrossRef  MATH  Google Scholar 

  9. Redstone, J., Chandra, T., Griesemer, R.: Paxos Made Live: An Engineering Perspective. In: Proc. of the 26th Annual ACM Symposium on Principles of Distributed Computing (PODC 2007), Portland, OR (2007)

    Google Scholar 

  10. Lee, E., Thekkath, C.: Petal: Distributed Virtual Disks. In: Proc. of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Cambridge, MA (1996)

    Google Scholar 

  11. Liskov, B., Ghemawat, S., Gruber, R., Johnson, P., Shrira, L.: Replication in the Harp File System. In: Proc. of the 13th ACM Symposium on Operating Systems Principles, Pacific Grove, CA (1991)

    Google Scholar 

  12. MacCormick, J., Murphy, N., Najork, M., Thekkath, C.A., Zhou, L.: Boxwood: Abstractions as the Foundation for Storage Infrastructure. In: Proc. of 6th Symposium on Operating Systems Design & Implementation (OSDI 2004), San Francisco, CA (2004)

    Google Scholar 

  13. MacCormick, J., Thekkath, C.A., Jager, M., Roomp, K., Zhou, L., Peterson, R.: Niobe: A Practical Replication Protocol. ACM Trans. on Storage 3(4), 1–43 (2008)

    CrossRef  Google Scholar 

  14. Shepler, S., et al.: Parallel NFS, RFC 5661-5664, http://tools.ietf.org/html/rfc5661

  15. Ligon, M., Ross, R.: Overview of the Parallel Virtual Fle System. In: Proceedings of USENIX Extreme Linux Workshop, Monterey, CA (1999)

    Google Scholar 

  16. Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The Hadoop Distributed File System. In: Proc. of IEEE Conference on Mass Storage Systems and Technologies (MSST), Lake Tahoe, NV (2010)

    Google Scholar 

  17. Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google File System. In: Proc. of 19th ACM Symposium on Operating Systems Principles (SOSP-19), Bolton Landing, New York (2003)

    Google Scholar 

  18. Bhide, A.K., Elnozahy, E.N., Morgan, S.P.: A Highly Available Network File Server. In: Proc. of the USENIX Winter Conference, Nashville, TE (January 1991)

    Google Scholar 

  19. Anderson, T., Dahlin, M., Neefe, J., Patterson, D., Roselli, D., Wang, R.: Serverless Network File Systems. In: Proc. of 15th Symposium on Operating Systems Principles, Copper Mountain, CO (1996)

    Google Scholar 

  20. Weil, S., Brandt, S., Miller, E.L., Long, D., Maltzahn, C.: Ceph: A Scalable, High-Performance Distributed File System. In: Proc. of 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2006), Seattle, WA (2006)

    Google Scholar 

  21. Vahalia, U.: Unix Internals: The New Frontiers. Prentice Hall (2008)

    Google Scholar 

  22. Garcia-Molina, H., Salem, K.: Main Memory Database Systems: An Overview. IEEE Transactions on Knowledge and Data Engineering 4(6), 509–516 (1992)

    CrossRef  Google Scholar 

  23. Pacemaker. A Scalable High-Availability Cluster Resource Manager, http://clusterlabs.org

  24. Katcher, J.: PostMark: A New File System Benchmark. Technical report, Network Appliance TR-3022 (October 1997)

    Google Scholar 

  25. Schneider, F.B.: Implementing Fault-Tolerant Services Using the State Machine Approach: a Tutorial. ACM Computing Surveys 22(4), 299–319 (1990)

    CrossRef  Google Scholar 

  26. Gray, J.: Why Do Computers Stop and What Can be Done About it? Technical report, Tandem TR 85-7 (1985)

    Google Scholar 

  27. Gifford, D.: Weighted Voting for Replicated Data. In: Proc. of the 7th ACM Symposium on Operating Systems Principles (SOSP), Pacific Grove, CA (1979)

    Google Scholar 

  28. Thekkath, C., Mann, T., Lee, E.: Frangipani: a Scalable Distributed File System. In: Proc. of the 16th Symp. on Operating Systems Principles, S. Malo, France (1997)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Computer Science (ICS), Foundation for Research and Technology Hellas (FORTH), Heraklion, GR-70013, Greece

    Dimokritos Stamatakis, Nikos Tsikoudis, Ourania Smyrnaki & Kostas Magoutis

Authors
  1. Dimokritos Stamatakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikos Tsikoudis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ourania Smyrnaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kostas Magoutis
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute of Information Systems, Vienna University of Technology, Argentinierstrasse 8/184-1, 1040, Vienna, Austria

    Karl Michael Göschka

  2. Swedish Institute of Computer Science, Isafjordsgatan 22, 164 29, Kista, Sweden

    Seif Haridi

Rights and permissions

Reprints and Permissions

Copyright information

© 2012 IFIP International Federation for Information Processing

About this paper

Cite this paper

Stamatakis, D., Tsikoudis, N., Smyrnaki, O., Magoutis, K. (2012). Scalability of Replicated Metadata Services in Distributed File Systems. In: Göschka, K.M., Haridi, S. (eds) Distributed Applications and Interoperable Systems. DAIS 2012. Lecture Notes in Computer Science, vol 7272. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-30823-9_3

Download citation

  • .RIS
  • .ENW
  • .BIB
  • DOI: https://doi.org/10.1007/978-3-642-30823-9_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-30822-2

  • Online ISBN: 978-3-642-30823-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Search

Navigation

  • Find a journal
  • Publish with us

Discover content

  • Journals A-Z
  • Books A-Z

Publish with us

  • Publish your research
  • Open access publishing

Products and services

  • Our products
  • Librarians
  • Societies
  • Partners and advertisers

Our imprints

  • Springer
  • Nature Portfolio
  • BMC
  • Palgrave Macmillan
  • Apress
  • Your US state privacy rights
  • Accessibility statement
  • Terms and conditions
  • Privacy policy
  • Help and support

167.114.118.210

Not affiliated

Springer Nature

© 2023 Springer Nature