Skip to main content

Distributed Storage Hash Algorithm (DSHA) for File-Based Deduplication in Cloud Computing

Part of the Lecture Notes on Data Engineering and Communications Technologies book series (LNDECT,volume 44)

Abstract

Increasing volume of digital data in cloud storage demands more storage space and efficient technique to handle these data. Duplicate is unavoidable while handling huge volume of data. Data Deduplication is an efficient approach in cloud storage environment that utilizes different techniques to deal with duplicate data. Existing systems generate the hash value by using any kind of cryptographic hash algorithms such as MD5 or Secure hash algorithms to implement the De-duplication approach. These algorithms produce fixed length of 128 bit or 160 bit as output respectively in order to identify the presence of duplication. So, an additional memory space is used to store this hash value. In this paper, an efficient Distributed Storage Hash Algorithm (DSHA) has been proposed to lessen the memory space occupied by the hash value which is utilized to identify and discard redundant data in cloud. Experimental analysis shows that the proposed strategy, reduces memory utilization of hash value and improves data read/write performance.

Keywords

  • Deduplication
  • Chunking
  • MD5
  • SHA-1
  • Bloom filter

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-37051-0_64
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   219.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-37051-0
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   279.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

References

  1. El-Shimi, A., Kalach, R., Kumar, A., et al.: Primary data deduplication–large scale study and system design. In: Proceedings of the 2012 Conference on USENIX Annual Technical Conference (USENIX’12), USENIX Association, Boston, MA, USA, June 2012, pp. 1–12 (2012)

    Google Scholar 

  2. Bloom, B.H.: Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13(7), 422–426 (1970)

    CrossRef  Google Scholar 

  3. Zhu, B., Li, K., Patterson, R.H.: Avoiding the disk bottleneck in the data domain deduplication file system. In: Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST’08), pp. 269–282. USENIX Association, Berkeley, CA, USA (2008)

    Google Scholar 

  4. Yeo, C.S., Buyya, R.: Service level agreement based allocation of cluster resources: handling penalty to enhance utility. In: 7th IEEE International Conference on Cluster Computing (Cluster 2005), September 2005

    Google Scholar 

  5. Bhagwat, D., Eshghi, K., Long et D.D., et al.: Extreme binning: Scalable, parallel deduplication for chunk-based file backup. In: Proceedings of IEEE International Symposium on Modeling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS’09), pp. 1–9. IEEE Computer Society Press, London, UK, September 2009

    Google Scholar 

  6. Eastlake, D.: Us secure hash algorithm 1 (sha1). http://tools.ietf.org/html/rfc3174, September 2001

  7. Meyer, D., Bolosky, W.: A study of practical deduplication. In: Proceedings of the USENIX Conference on File and Storage Technologies (FAST’11), pp. 229–241. USENIX Association, San Jose, CA, USA, February 2011

    Google Scholar 

  8. MacDonald, J.: File system support for delta compression. Masters thesis, Department of Electrical Engineering and Computer Science, University of California at Berkeley (2000)

    Google Scholar 

  9. Dave, J., Faruki, P., Laxmi, V., Bezawada, B., Gaur, M.: Secure and efficient proof of ownership for deduplicated cloud storage. In: Proceedings of the 10th International Conference on Security of Information and Networks (SIN’17). ACM, New York, NY, USA, 19–26 (2017)

    Google Scholar 

  10. Wang, Jibin, Zhao, Zhigang, Zhaogang, Xu, Zhang, Hu, Li, Liang, Guo, Ying: I-sieve: an inline high performance deduplication system used in cloud storage. Tsinghua Sci. Technol. 20(1), 17–27 (2015)

    CrossRef  Google Scholar 

  11. Eshghi, K., Tang, H.K.: A framework for analysing and improving content-based chunking algorithms, Technical Report HPL-2005-30(RI) (2005)

    Google Scholar 

  12. Kumar, Sandeep, Gupta, Er Piyush: A comparative analysis of SHA and MD5 algorithm. Int. J. Comput. Sci. Inf. Technol. 5, 4492–4495 (2014)

    Google Scholar 

  13. Lillibridge, M., Eshghi, K., Bhagwat, D., et al.: Sparse indexing: Large scale, inline deduplication using sampling and locality. In: Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST’09), vol. 9. USENIX Association, San Jose, CA, February 2009, pp. 111–123

    Google Scholar 

  14. NetApp deduplication and compression. www.netapp.com/us/products/platform-os/dedupe.html, April 2016

  15. Opendedup. http://opendedup.org/

  16. Rivest, R.: The md5 message-digest algorithm, April 1992. http://tools.ietf.org/html/rfc1321

  17. Quinlan, S., Dorward, S.: Venti: a new approach to archival data storage. In: Proceedings of the 1st USENIX Conference on File and Storage Technologies (2002)

    Google Scholar 

  18. Suel, T., Memon, N.: Algorithms for delta compression and remote file synchronization. Lossless Compression Handbook (2002)

    Google Scholar 

  19. Venish, A., Siva Sankar, K.: Study of chunking algorithm in data deduplication. In: Proceedings of the International Conference on Soft Computing Systems. Springer, New Delhi (2016)

    Google Scholar 

  20. Wang, X., Yu, H.: How to break MD5 and other hash functions. In: EUROCRYPT (2005), vol. 3494, pp. 19–35. Lecture Notes in Computer Science. Springer

    Google Scholar 

  21. Zhang, Y., Wu, Y., Yang, G.: Droplet: a distributed solution of data deduplication. In: 2012 ACM/IEEE 13th International Conference on Grid Computing, Beijing, pp. 114–121 (2012)

    Google Scholar 

  22. Yan, Z., Ding, W.X., Zhu, H.Q.: A scheme to manage encrypted data storage with deduplication in cloud. In: Proceedings of ICA3PP2015, November 2015

    Google Scholar 

  23. Kaur, R., Chana, I., Bhattacharya, J.: Data deduplication techniques for efficient cloud storage management: a systematic review. J. Supercomputing 74(5), 2035–2085 (2017)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to S. Hema .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Hema, S., Kangaiammal, A. (2020). Distributed Storage Hash Algorithm (DSHA) for File-Based Deduplication in Cloud Computing. In: Smys, S., Senjyu, T., Lafata, P. (eds) Second International Conference on Computer Networks and Communication Technologies. ICCNCT 2019. Lecture Notes on Data Engineering and Communications Technologies, vol 44. Springer, Cham. https://doi.org/10.1007/978-3-030-37051-0_64

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-37051-0_64

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-37050-3

  • Online ISBN: 978-3-030-37051-0

  • eBook Packages: EngineeringEngineering (R0)