Cache-, Hash- and Space-Efficient Bloom Filters
A Bloom filter is a very compact data structure that supports approximate membership queries on a set, allowing false positives.
We propose several new variants of Bloom filters and replacements with similar functionality. All of them have a better cache-efficiency and need less hash bits than regular Bloom filters. Some use SIMD functionality, while the others provide an even better space efficiency. As a consequence, we get a more flexible trade-off between false positive rate, space-efficiency, cache-efficiency, hash-efficiency, and computational effort. We analyze the efficiency of Bloom filters and the proposed replacements in detail, in terms of the false positive rate, the number of expected cache-misses, and the number of required hash bits. We also describe and experimentally evaluate the performance of highly-tuned implementations. For many settings, our alternatives perform better than the methods proposed so far.
Unable to display preview. Download preview PDF.
- Bloom, B.H.: Space-time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7) (1970)Google Scholar
- Broder, A., Mitzenmacher, M.: Network applications of bloom filters: A survey. Internet Mathematics, 1(4) (2004)Google Scholar
- Dillinger, P.C., Manolios, P.: Bloom filters in probabilistic verification. In: Hu, A.J., Martin, A.K. (eds.) FMCAD 2004. LNCS, vol. 3312, pp. 367–381. Springer, Heidelberg (2004)Google Scholar
- Dillinger, P.C., Manolios, P.: Fast and accurate bitstate verification for SPIN. In: Graf, S., Mounier, L. (eds.) Model Checking Software. LNCS, vol. 2989, pp. 57–75. Springer, Heidelberg (2004)Google Scholar
- Mitzenmacher, M.: Compressed Bloom filters. In: PODC 2001, pp. 144–150 (2001)Google Scholar
- Moffat, A., Turpin, A.: Compression and Coding Algorithms. Kluwer Academic Publishers, Dordrecht (2002)Google Scholar
- Pagh, A., Pagh, R., Rao, S.S.: An optimal Bloom filter replacement. In: SODA 2005, pp. 823–829 (2005)Google Scholar
- Sanders, P., Transier, F.: Intersection in integer inverted indices. In: ALENEX 2007 (2007)Google Scholar