Abstract
We suggest a method for holding a dictionary data structure, which maps keys to values, in the spirit of Bloom Filters. The space requirements of the dictionary we suggest are much smaller than those of a hashtable. We allow storing n keys, each mapped to value which is a string of k bits. Our suggested method requires nk + o(n) bits space to store the dictionary, and O(n) time to produce the data structure, and allows answering a membership query in O(1) memory probes. The dictionary size does not depend on the size of the keys. However, reducing the space requirements of the data structure comes at a certain cost. Our dictionary has a small probability of a one sided error. When attempting to obtain the value for a key that is stored in the dictionary we always get the correct answer. However, when testing for membership of an element that is not stored in the dictionary, we may get an incorrect answer, and when requesting the value of such an element we may get a certain random value. Our method is based on solving equations in GF(2k) and using several hash functions.
Another significant advantage of our suggested method is that we do not require using sophisticated hash functions. We only require pairwise independent hash functions. We also suggest a data structure that requires only nk bits space, has O(n 2) preprocessing time, and has a O(logn) query time. However, this data structures requires a uniform hash functions.
In order replace a Bloom Filter of n elements with an error proability of 2− k, we require nk + o(n) memory bits, O(1) query time, O(n) preprocessing time, and only pairwise independent hash function. Even the most advanced previously known Bloom Filter would require nk + O(n) space, and a uniform hash functions, so our method is significantly less space consuming especially when k is small.
Our suggested dictionary can replace Bloom Filters, and has many applications. A few application examples are dictionaries for storing bad passwords, differential files in databases, Internet caching and distributed storage systems.
A video presentation can be found in http://www.youtube.com/ watch?v=947gWqwkhu0.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bloom, B.H.: Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13(7), 422–426 (1970)
Broder, A., Mitzenmacher, M.: Network applications of bloom filters: A survey (2002)
Brodnik, A., Ian Munro, J.: Membership in constant time and almost-minimum space. SIAM J. Comput. 28(5), 1627–1640 (1999)
Charles, D., Chellapilla, K.: Bloomier filters: A second look. In: Halperin, D., Mehlhorn, K. (eds.) ESA 2008. LNCS, vol. 5193, pp. 259–270. Springer, Heidelberg (2008)
Chazelle, B., Kilian, J., Rubinfeld, R., Tal, A.: The bloomier filter: an efficient data structure for static support lookup tables. In: SODA 2004: Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms, Philadelphia, PA, USA, pp. 30–39. Society for Industrial and Applied Mathematics (2004)
Cohen, S., Matias, Y.: Spectral bloom filters. In: SIGMOD 2003: Proceedings of the 2003 ACM SIGMOD international conference on Management of data, pp. 241–252. ACM, New York (2003)
Dietzfelbinger, M., Pagh, R.: Succinct data structures for retrieval and approximate membership (extended abstract). In: Aceto, L., Damgård, I., Goldberg, L.A., Halldórsson, M.M., Ingólfsdóttir, A., Walukiewicz, I. (eds.) ICALP 2008, Part I. LNCS, vol. 5125, pp. 385–396. Springer, Heidelberg (2008)
Fan, L., Cao, P., Almeida, J., Broder, A.Z.: Summary cache: a scalable wide-area web cache sharing protocol. IEEE/ACM Trans. Netw. 8(3), 281–293 (2000)
Gremillion, L.L.: Designing a bloom filter for differential file access. Commun. ACM 25(9), 600–604 (1982)
Manber, U., Wu, S.: An algorithm for approximate membership checking with application to password security. Inf. Process. Lett. 50(4), 191–197 (1994)
McIlroy, M.D.: Development of a spelling list. IEEE Transactions on Communications 30(1), 91–99 (1982)
Mitzenmacher, M.: Compressed bloom filters. IEEE/ACM Trans. Netw. 10(5), 604–612 (2002)
Mullin, J.K., Margoliash, D.J.: A tale of three spelling checkers. Softw. Pract. Exper. 20(6), 625–630 (1990)
Mullin, J.K.: A second look at bloom filters. Commun. ACM 26(8), 570–571 (1983)
Pagh, A., Pagh, R., Srinivasa Rao, S.: An optimal bloom filter replacement. In: SODA 2005: Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms, Philadelphia, PA, USA, pp. 823–829. Society for Industrial and Applied Mathematics (2005)
Spafford, E.H.: Opus: Preventing weak password choices. Computer and Security 10, 273–278 (1992)
Villard, G.: A study of coppersmith’s block wiedemann algorithm using matrix polynomials
Wiedemann, D.H.: Solving sparse linear equations over finite fields. IEEE Trans. Information Theory IT-32(1), 54–62 (1986)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Porat, E. (2009). An Optimal Bloom Filter Replacement Based on Matrix Solving. In: Frid, A., Morozov, A., Rybalchenko, A., Wagner, K.W. (eds) Computer Science - Theory and Applications. CSR 2009. Lecture Notes in Computer Science, vol 5675. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03351-3_25
Download citation
DOI: https://doi.org/10.1007/978-3-642-03351-3_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03350-6
Online ISBN: 978-3-642-03351-3
eBook Packages: Computer ScienceComputer Science (R0)