TBF: A High-Efficient Query Mechanism in De-duplication Backup System

  • Bin Zhou
  • Hai Jin
  • Xia Xie
  • PingPeng Yuan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7296)


For the big data, the fingerprints of the data chunks are very huge and cannot be stored in the memory completely. Accordingly, a new query mechanism namely Two-stage Bloom Filter mechanism is proposed. First, each bit of the second grade bloom filter represents the chunks having the identical fingerprints which reducing the rate of false positives. Second, a two-dimensional list is created corresponding to the two grade bloom filter to gather the absolute addresses of the data chunks with the identical fingerprints. Finally, we suggest a new hash function class with the strong global random characteristic. Two-stage Bloom Filter decreases the number of accessing disks, improves the speed of detecting the redundant data chunks, and reduces the rate of false positive. Our experiments indicate that Two-stage Bloom Filter reduces about 30~40% storage accessing of false positive with the same length of the first grade Bloom Filter.


Two-stage Bloom Filter Standard Bloom Filter De-duplication False positive Fingerprint 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    The International Data Corporation website,
  2. 2.
    Zhu, B., Li, K., Patterson, H.: Avoiding the Disk Bottleneck in the Data Domain Deduplication File System. In: Proceedings of 6th USENIX Conference on File and Storage Technologies, pp. 1–14. USENIX Association, San Jose (2008)Google Scholar
  3. 3.
    Bobbarjung, D.R., Jaqannathan, S., Dubnicki, C.: Improving Duplicate Elimination in Storage Systems. ACM Transactions on Storage 2, 424–448 (2006)CrossRefGoogle Scholar
  4. 4.
    Lillibridge, M.: Sparse Indexing, Large Scale, Inline Deduplication Using Sampling and Locality. In: Proceedings of 7th USENIX Conference on File and Storage Technologies, pp. 111–123. USENIX Association, San Francisco (2009)Google Scholar
  5. 5.
    Thewl, T.T., Thein, N.L.: An Efficient Indexing Mechanism for Data Deduplication. In: Proceedings of 2009 International Conference on the Current Trends in Information Technology, pp. 1–5. IEEE Press, Dubai (2009)CrossRefGoogle Scholar
  6. 6.
    Bhagwat, D., Eshghi, K., Long, D.D.E., Lillibridge, M.: Extreme Binning: Scalable, Parallel Deduplication for Chunk-based File Backup. In: Proceedings of 17th IEEE/ACM International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, pp. 1–9. IEEE Press, London (2009)CrossRefGoogle Scholar
  7. 7.
    Kruus, E., Ungureanu, C., Dubnicki, C.: Bimodal Content Defined Chunking for Backup Streams. In: Proceedings of 8th USENIX Conference on File and Storage Technologies, pp. 239–252. USENIX Association, Berkeley (2010)Google Scholar
  8. 8.
    Lu, G.L., Jin, Y., Du, D.H.C.: Frequency Based Chunking for Data De-Duplication. In: Proceedings of 18th IEEE/ACM International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, pp. 287–296. IEEE Press, Miami (2010)CrossRefGoogle Scholar
  9. 9.
    Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. In: Proceedings of 7th USENIX Symposium on Operating Systems Design and Implementation, pp. 205–218. USENIX Association, Berkeley (2006)Google Scholar
  10. 10.
    Jain, N., Dahlin, M., Tewari, R.: TAPER: Tiered Approach for Eliminating Redundancy in Replica Synchronization. In: Proceedings of 4th USENIX Conference on File And Storage Technologies, pp. 281–294. USENIX Association, Berkeley (2005)Google Scholar
  11. 11.
    Bhattacherjee, S., Naranq, A., Garq, V.K.: High Throughput Data Redundancy Removal Algorithm with Scalable Performance. In: Proceedings of 6th International Conference on High Performance and Embedded Architectures and Compilers, pp. 87–96. ACM, New York (2011)Google Scholar
  12. 12.
    Debnath, B., Sengupta, S., Li, J., Lilja, D.J., Du, D.: BloomFlash: Bloom Filter on Flash-Based Storage. In: Proceedings of 31th International Conference on Distributed Computing Systems, pp. 635–644. IEEE Computer Society, Washington (2011)Google Scholar
  13. 13.
    Bender, M.A., Farach-Colton, M., Johnson, R., Kuszmaul, B.C., Medjedovic, D., Montes, P., Shetty, P., Spillane, R.P., Zadok, E.: Don’t Thrash: How to Cache Your Hash on Flash. In: Proceedings of 3rd USENIX Conference on Hot Topics in Storage and File Systems, p. 1. USENIX Association, Berkeley (2011)Google Scholar
  14. 14.
    Guo, D., Wu, J., Chen, H.H., Yuan, Y., Luo, X.S.: The Dynamic Bloom Filters. IEEE Transactions on Knowledge and Data Engineering 22, 120–133 (2010)CrossRefGoogle Scholar
  15. 15.
    Song, H.Y., Dharmapurikar, S., Turner, J., Lockwood, J.: Fast Hash Table Lookup Using Extended Bloom Filter: An Aid to Network Processing. In: Proceedings of the 2005 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, pp. 181–192. ACM, New York (2005)Google Scholar
  16. 16.
    Guo, D., Chen, H.H., Luo, X.S.: Theory and Network Application of Dynamic Bloom Filters. In: Proceedings of 25th Annual Joint Conference of the IEEE Computer and Communications Societies, pp. 1–12. IEEE Press, Spain (2006)Google Scholar
  17. 17.
    Ahmadi, M., Wong, S.: Modified Collision Packet Classification Using Counting Bloom Filter in Tuple Space. In: Proceedings of 25th IASTED International Conference on Parallel and Distributed Computing and Networks, pp. 70–76. ACTA Press, Anaheim (2007)Google Scholar
  18. 18.
    Ahmadi, M., Wong, S.: A Memory-optimized Bloom Filter Using an Additional Hashing Function. In: Proceedings of Global Telecommunications Conference, pp. 1–5. IEEE Press, New Orleans (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Bin Zhou
    • 1
    • 2
  • Hai Jin
    • 1
  • Xia Xie
    • 1
  • PingPeng Yuan
    • 1
  1. 1.Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and TechnologyHuazhong University of Science and TechnologyWuhanChina
  2. 2.School of Computer Science and TechnologySouth-Central University for NationalitiesWuhanChina

Personalised recommendations