Distributed Paged Hash Tables

  • José Rufino
  • António Pina
  • Albano Alves
  • José Exposto
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2565)

Abstract

In this paper we present the design and implementation of DPH, a storage layer for cluster environments. DPH is a Distributed Data Structure (DDS) based on the distribution of a paged hash table. It combines main memory with file system resources across the cluster in order to implement a distributed dictionary that can be used for the storage of very large data sets with key based addressing techniques. The DPH storage layer is supported by a collection of cluster-aware utilities and services. Access to the DPH interface is provided by a user-level API. A preliminary performance evaluation shows promising results.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    Al Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchek, and V. Sunderam. PVM: Parallel Virtual Machine. A User’s Guide and Tutorial for Networked Parallel Computing. Scientific and Engineering Computation. MIT Press, 1994. 679Google Scholar
  2. [2]
    M. Snir, S. Otto, S. Huss-Lederman, David Walker, and J. Dongarra. MPI-The Complete Reference. Scientific and Engineering Computation. MIT Press, 1998. 679Google Scholar
  3. [3]
    W. Litwin, M.-A. Neimat, and D.A. Schneider. LH*: Linear Hashing for Distributed Files. In Proceedings of the ACM SIGMOD-International Conference on Management of Data, pages 327–336, 1993. 679, 680Google Scholar
  4. [4]
    R. Devine. Design and implementation of DDH: a distributed dynamic hashing algorithm. In Proceedings of the 4th Int. Conf. on Foundations of Data Organization and Algorithms, pages 101–114, 1993. 679, 680Google Scholar
  5. [5]
    V. Hilford, F.B. Bastani, and B. Cukic. EH*-Extendible Hashing in a Distributed Environment. In Proceedings of the COMPSAC’ 97-21st International Computer Software and Applications Conference, 1997. 679, 681Google Scholar
  6. [6]
    R. Vingralek, Y. Breitbart, and G. Weikum. Distributed File Organization with Scalable Cost/Performance. In Proceedings of the ACM SIGMOD-International Conference on Management of Data, 1994. 679Google Scholar
  7. [7]
    B. Kroll and P. Widmayer. Distributing a Search Tree Among a Growing Number of Processors. In Proceedings of the ACM SIGMOD-International Conference on Management of Data, pages 265–276, 1994. 679, 681Google Scholar
  8. [8]
    T. Johnson and A. Colbrook. A Distributed, Replicated, Data-Balanced Search Structure. Technical Report TR03-028, Dept. of CISE, University of Florida, 1995. 679, 681Google Scholar
  9. [9]
    S.D. Gribble, E.A. Brewer, J.M. Hellerstein, and D. Culler. Scalable, Distributed Data Structures for Internet Service Construction. In Proceedings of the Fourth Symposium on Operating Systems Design and Implementation, 2000. 679, 681Google Scholar
  10. [10]
    W.K. Preslan et all. A 64-bit, Shared Disk File System for Linux. In Proceedings of the 7h NASA Goddard Conference on Mass Storage Systems and Tech. in cooperation with the Sixteenth IEEE Symposium on Mass Storage Systems, 1999. 679Google Scholar
  11. [11]
    P.H. Carns, W. B. Ligon, R.B. Ross, and R. Thakur. PVFS: A Parallel File System for Linux Clusters. In Proceedings of the 4th Annual Linux Showcase and Conference, pages 317–327. USENIX Association, 2000. 679Google Scholar
  12. [12]
    J. S. Vitter. Online Data Structures in External Memory. In Proceedings of the 26th Annual Intern. Colloquium on Automata, Languages, and Programming, 1999. 679, 691Google Scholar
  13. [13]
    J. Rufino, A. Pina, A. Alves, and J. Exposto. Distributed Hash Tables. International Workshop on Performance-oriented Application Development for Distributed Architectures (PADDA 2001), 2001. 679Google Scholar
  14. [14]
    D.E. Knuth. The Art of Computer Programming-Volume 3: Sorting and Searching. Addison-Wesley, 2nd edition, 1998. 680, 682Google Scholar
  15. [15]
    R. J. Enbody and H.C. Du. Dynamic Hashing Schemes. ACM Computing Surveys, (20):85–113, 1988. 680, 691CrossRefGoogle Scholar
  16. [16]
    W. Litwin. Linear hashing: A new tool for file and table addressing. In Proceedings of the 6th Conference on Very Large Databases, pages 212–223, 1980. 680Google Scholar
  17. [17]
    R. Fagin, J. Nievergelt, N. Pippenger, and H.R. Strong. Extendible hashing: a fast access method for dynamic files. ACM Transactions on Database Systems, (315–344), 1979. 680, 681Google Scholar
  18. [18]
    T. Stornetta and F. Brewer. Implementation of an Efficient Parallel BDD Package. In Proceedings of the 33rd ACM/IEEE Design Automation Conference, 1996. 681Google Scholar
  19. [19]
    P. Bagwell. Ideal Hash Trees. Technical report, Computer Science Department, Ecole Polytechnique Federale de Lausanne, 2000. 681Google Scholar
  20. [20]
    A. Pina, V. Oliveira, C. Moreira, and A. Alves. pCoR-a Prototype for Resource Oriented Computing. (to appear in HPC 2002), 2002. 682Google Scholar
  21. [21]
    A. Pina. MC2-Modelo de ComputaÇão Celular. Origem e EvoluÇão. PhDthesis, Dep. de Informática, Univ. do Minho, Braga, Portugal, 1997. 682Google Scholar
  22. [22]
    Myricom. The GM Message Passing System, 2000. 682, 685Google Scholar
  23. [23]
    B. Jenkins. A Hash Function for Hash Table Lookup. Dr. Doob’s, 1997. 682, 686, 691Google Scholar
  24. [24]
    A.V. Aho, R. Sethi, and J.D. Ullman. Compilers: Principles, Techniques and Tools. Addison-Wesley, 1985. 682Google Scholar
  25. [25]
    R. C. Uzgalis. General Hash Functions. Technical Report TR 91-01, University of Hong Kong, 1991. 682Google Scholar
  26. [26]
    W. Pugh. SkipList s: A Probabilistic Alternative to Balanced Trees. Communications of the ACM, 33(6):668–676, 1990. 683CrossRefMathSciNetGoogle Scholar
  27. [27]
    D. Kargeer, A. Sherman, A. Berkheimer, B. Bogstad, R. Dhanidina, K. Iwamoto, B. Kim, L. Matkins, and Y. Yerushalmi. Web Caching with Consistent Hashing. In Proceedings of the 8th International WWW Conference, 1999. 691Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • José Rufino
    • 1
  • António Pina
    • 2
  • Albano Alves
    • 1
  • José Exposto
    • 1
  1. 1.Polytechnic Institute of BraganÇaBraganÇaPortugal
  2. 2.University of MinhoBragaPortugal

Personalised recommendations