Skip to main content
SpringerLink
Log in

Design and Implementation of Various File Deduplication Schemes on Storage Devices

  • Published:
Mobile Networks and Applications Aims and scope Submit manuscript

Abstract

As smart devices are revolutionized in recent years, people may generate enormous amount of various sized data and store them in the local or remote file system in their daily lives. With cheaper and easy to use private cloud storage appliances helping to handle the increasing demand of storing and sharing big volume of data, effective file deduplication schemes can greatly increase the space efficiency in private cloud storage systems as well as preserve network bandwidth. In the paper, we aim at designing and implementing several file deduplication schemes built in the private cloud storage appliance, based on different duplication checking rules, including file name, file size, and file partial/full content hash value. Experiment results show using partial content hashing based file deduplication scheme achieves a reasonably balanced performance without overutilized limited local computational resources.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Tate J, Beck P, Ibarra HH, Kumaravel S, Miklas L (2012) Introduction to storage area networks and system networking. IBM Redbooks

  2. Hong B, Plantenberg D, Long DD, & Sivan-Zimet M (2004) “Duplicate Data Elimination in a SAN File System”. In MSST (pp. 301–314)

  3. Bobbarjung DR, Jagannathan S, Dubnicki C (2006) Improving duplicate elimination in storage systems. ACM Trans Storage (TOS) 2(4):424–448

    Article  Google Scholar 

  4. Min J, Yoon D, Won Y (2011) Efficient deduplication techniques for modern backup operation. Comput IEEE Trans on 60(6):824–840

    Article  MathSciNet  Google Scholar 

  5. Li J, Li YK, Chen X, Lee PP, Lou W (2015) A hybrid cloud approach for secure authorized deduplication. Parallel and Distrib Sys IEEE Trans on 26(5):1206–1216

    Article  Google Scholar 

  6. Stanek J, Sorniotti A, Androulaki E, Kencl L (2014) A secure data deduplication scheme for cloud storage in financial cryptography and data security. Springer, Berlin Heidelberg, pp 99–118

    Google Scholar 

  7. Meyer DT, Bolosky WJ (2012) A study of practical deduplication. ACM Trans Storage (TOS) 7(4):14

    Google Scholar 

  8. Harnik D, Pinkas B, Shulman-Peleg A (2010) Side channels in cloud services: deduplication in cloud storage. Security & Privacy IEEE 8(6):40–47

    Article  Google Scholar 

  9. Paulo J, Pereira J (2014) A survey and classification of storage deduplication systems. ACM Comput Surveys (CSUR) 47(1):11

    Article  Google Scholar 

  10. Meister D., & Brinkmann A (2009) Multi-level comparison of data deduplication in a backup scenario. In Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference (p. 8). ACM

  11. Henson V (2003) An Analysis of Compare-by-hash. In HotOS (pp. 13–18)

  12. Malhotra J, & Bakal J (2015) A survey and comparative study of data deduplication techniques. In Pervasive Computing (ICPC), 2015 International Conference on (pp. 1–5). IEEE

  13. Rivest R (1992) The MD5 message-digest algorithm. RFC 1321

  14. Banachowski L, Kreczmar A, Rytter W (1991) Analysis of Algorithms and Data Structures

  15. Quinlan S, & Dorward S (2002) Venti: A New Approach to Archival Storage. In FAST (Vol. 2, pp. 89–101).2

Download references

Acknowledgments

The authors gratefully acknowledge the financial support from the “Aiming For the Top University Program” funded by Ministry of Education, Taiwan.

Author information

Authors and Affiliations

  1. Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan

    Kuan-Wu Su, Jenq-Shiou Leu, Min-Chieh Yu & Yong-Ting Wu

  2. QNAP Inc, Taipei, Taiwan

    Eau-Chung Lee

  3. Department of Electrical and Electronic Engineering, School of Engineering, Tokushima University, Tokushima, Japan

    Tian Song

Authors
  1. Kuan-Wu Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jenq-Shiou Leu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Min-Chieh Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yong-Ting Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eau-Chung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tian Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jenq-Shiou Leu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Su, KW., Leu, JS., Yu, MC. et al. Design and Implementation of Various File Deduplication Schemes on Storage Devices. Mobile Netw Appl 22, 40–50 (2017). https://doi.org/10.1007/s11036-016-0677-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11036-016-0677-9

Keywords

Search

Navigation