Skip to main content
SpringerLink
Log in

Dynamic Space Efficient Hashing

  • Published:
Algorithmica Aims and scope Submit manuscript

Abstract

We consider space efficient hash tables that can grow and shrink dynamically and are always highly space efficient, i.e., their space consumption is always close to the lower bound even while growing and when taking into account storage that is only needed temporarily. None of the traditionally used hash tables have this property. We show how known approaches like linear probing and bucket cuckoo hashing can be adapted to this scenario by subdividing them into many subtables or using virtual memory overcommitting. However, these rather straightforward solutions suffer from slow amortized insertion times due to frequent reallocation in small increments. Our main result is Dynamic Space Efficient Cuckoo Table (DySECT ) which avoids these problems. DySECT consists of many subtables which grow by doubling their size. The resulting inhomogeneity in subtable sizes is counterbalanced by the flexibility available in bucket cuckoo hashing where each element can go to several buckets each of which containing several cells. Experiments indicate that DySECT works well with loads up to 98%. With up to 1.9 times better performance than the next best solution. Additionally, we give a tight theoretical analysis for the possible load threshold of DySECT, i.e., a bound where with high probability the table can be filled up to that load but not above said load. This load also matches our experimental findings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. www.xxhash.com

  2. We change notation slightly. In [17] X and Y were called A and B, the operator \(\min \) was denoted by \(\wedge \), \(n_{\mathsf {max}}^*\) was simply M and m was n.

  3. Experiments on a desktop machine and a server with Intel Xeon E5-2670 CPUs yielded similar results.

References

  1. Aldous, D., Michael Steele, J.: The Objective Method: Probabilistic Combinatorial Optimization and Local Weak Convergence, pp. 1–72. Springer, Berlin (2004)

    MATH  Google Scholar 

  2. Arbitman, Y., Naor, M., Segev, G.: De-amortized cuckoo hashing: provable worst-case performance and experimental results. In: 36th International Conference on Automata, Languages and Programming (ICALP), number 5555 in LNCS, pp. 107–118. Springer (2009)

  3. Celis, P., Larson, P.-A., Ian Munro, J: Robin hood hashing. In: 26th IEEE Symposium on Foundations of Computer Science (FOCS), pp. 281–288 (1985)

  4. Devroye, L., Morin, P.: Cuckoo hashing: further analysis. Inf. Process. Lett. 86(4), 215–219 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  5. Dietzfelbinger, M., Goerdt, A., Mitzenmacher, M., Montanari, A., Pagh, R., Rink, M.: Tight thresholds for cuckoo hashing via XORSAT. In: 27th International Conference on Automata, Languages and Programming (ICALP), pp. 213–225 (2010)

  6. Dietzfelbinger, M., Karlin, A., Mehlhorn, K., Meyer auf der Heide, F., Rohnert, H., Tarjan, R.E.: Dynamic perfect hashing: upper and lower bounds. SIAM J. Comput. 23(4), 738–761 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  7. Dietzfelbinger, M., Mitzenmacher, M., Rink, M.: Cuckoo hashing with pages. In: 19th European Symposium on Algorithms (ESA), number 6942 in LNCS, pp. 615–627. Springer (2011)

  8. Dietzfelbinger, M., Weidling, C.: Balanced allocation and dictionaries with tightly packed constant size bins. In: International Colloquium on Automata, Languages, and Programming, pp. 166–178. Springer (2005)

  9. Dietzfelbinger, M., Weidling, C.: Balanced allocation and dictionaries with tightly packed constant size bins. Theor. Comput. Sci. 380(1), 47–68 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  10. Fotakis, D., Pagh, R., Sanders, P., Spirakis, P.: Space efficient hash tables with worst case constant access time. Theory Comput. Syst. 38(2), 229–248 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  11. Fountoulakis, N., Panagiotou, K., Steger, A.: On the insertion time of cuckoo hashing. SIAM J. Comput. 42(6), 2156–2181 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  12. Frieze, A., Melsted, P., Mitzenmacher, M.: An analysis of random-walk cuckoo hashing. SIAM J. Comput. 40(2), 291–308 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  13. Guibas, L.J., Szemeredi, E.: The analysis of double hashing. J. Comput. Syst. Sci. 16(2), 226–274 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  14. Kirsch, A., Mitzenmacher, M.: Less hashing, same performance: Building a better bloom filter. In 14th European Symposium on Algorithms (ESA), number 4168 in LNCS, pp. 456–467. Springer (2006)

  15. Kirsch, A., Mitzenmacher, M.: Using a queue to de-amortize cuckoo hashing in hardware. In: 45th Annual Allerton Conference on Communication, Control, and Computing, vol. 75 (2007)

  16. Knuth, D.E.: The Art of Computer Programming Sorting and Searching, vol. 3. Addison Wesley Longman Publishing Co., Inc., Boston (1998)

    MATH  Google Scholar 

  17. Lelarge, M.: A new approach to the orientation of random hypergraphs. In: Proc. 23rd SODA, pp. 251–264 (2012)

  18. Li, X., Andersen, D.G., Kaminsky, M., Freedman, M.J.: Algorithmic improvements for fast concurrent cuckoo hashing. In: 9th European Conference on Computer Systems, EuroSys, pp. 27:1–27:14. ACM (2014)

  19. Maier, T., Sanders, P.: Dynamic space efficient hashing. In: 25th Annual European Symposium on Algorithms (ESA 2017), vol.  87 of Leibniz International Proceedings in Informatics (LIPIcs). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik (2017)

  20. Mitzenmacher, M.: The power of two choices in randomized load balancing. IEEE Trans Parallel Distrib. Syst. 12(10), 1094–1104 (2001)

    Article  Google Scholar 

  21. Mitzenmacher, M.: Some open questions related to cuckoo hashing. In: 17th European Symposium on Algorithms (ESA), Vol. 5757 of LNCS, pp. 1–10. Springer (2009)

  22. Mitzenmacher, M., Panagiotou, K., Walzer, S.: Load thresholds for cuckoo hashing with double hashing. In: 16th Scandinavian Symposium and Workshops on Algorithm Theory (SWAT 2018), volume 101 of Leibniz International Proceedings in Informatics (LIPIcs), pp. 29:1–29:9. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik (2018)

  23. Nguyen, N., Tsigas, P.: Lock-free cuckoo hashing. In: 34th IEEE International Conference on Distributed Computing Systems (ICDCS), pp. 627–636 (2014)

  24. Pagh, R., Rodler, F.F.: Cuckoo hashing. J. Algorithms 51(2), 122–144 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  25. Walzer, S.: Load thresholds for cuckoo hashing with overlapping blocks. CoRR, abs/1707.06855 (2017)

Download references

Author information

Authors and Affiliations

  1. Karlsruhe Institute of Technology, Karlsruhe, Germany

    Tobias Maier & Peter Sanders

  2. Technische Universität Ilmenau, Ilmenau, Germany

    Stefan Walzer

Authors
  1. Tobias Maier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Sanders
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Walzer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tobias Maier.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Maier, T., Sanders, P. & Walzer, S. Dynamic Space Efficient Hashing. Algorithmica 81, 3162–3185 (2019). https://doi.org/10.1007/s00453-019-00572-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00453-019-00572-x

Keywords

Search

Navigation