Abstract
Cloud Storage has been designed for storing large file that could occupy several HDDs (Hard Disk Drive). Most Cloud Storage Systems use Block Storage, which is efficient for large file, however, wasteful for small and medium files that are generally less than the size of a typical block (64/128 Mb). Research shows that most of the files created for personal usage is most of the time less than block size. As such for a Network Attached Storage (NAS), it is imperative to adopt another approach than Block Storage. Object Storage is the technique for storing of a group of files as one object and associated metadata stored independently. In this work, an Object Based NAS system of 12 TB is being implemented, and it has been developed using Hadoop 3.0. Given that Redundancy is important to guard against failure, the novelty of OSSREC is that is uses the default 3x Replication for individual files and coupled with Erasure Coding for Objects. A web interface is implemented to interact with Hadoop through the use of WebHDFS API and added to that, the functionality of Erasure Coding is also developed. The system is then evaluated in terms of Storage Capacity, Availability and Write/Read Performances. And finally, the Object Storage solution of OSSREC is benchmarked against the default Block Storage of Hadoop.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Poojary, N.: Understanding Object Storage and Block Storage use cases. Cloud Academy (2015). https://cloudacademy.com/blog/object-storage-block-storage/. Accessed 26 Sep 2018
Bernstein, P.A., Hadzilacos, V., Goodman, N.: Concurrency control and recovery in database systems (1987)
Anrg, M.: HDFS-RAID - Hadoop Wiki. Wiki.apache.org (2018). https://wiki.apache.org/hadoop/HDFS-RAID. Accessed 8 Aug 2018
Chiniah, A., Dhora, J.A.D., Sandooram, C.J.: Erasure-coded network backup system (ECNBS). In: International Conference on Information, Communication and Computing Technology, pp. 35–43. Springer, Singapore, May 2017
Reed, I.S., Solomon, G.: Polynomial codes over certain finite fields. J. Soc. Ind. Appl. Math. 8(2), 300–304 (1960)
Hadoop: Hadoop – Apache Hadoop 3.0.0. Hadoop.apache.org (2018). https://hadoop.apache.org/docs/r3.0.0/index.html. Accessed 6 Jul 2018
Hadoop.apache.org. Apache Hadoop 3.0.0 – HDFS Erasure Coding (2018). https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html. Accessed 17 Aug 2018
CS-08-627 - Jerasure Version 1.2. http://web.eecs.utk.edu/~plank/plank/papers/CS-08-627.html. Accessed 8 Apr 2018
Hafner, J.L.: WEAVER codes: highly fault tolerant erasure codes for storage systems. In: Fast, vol. 5, p. 16, December 2005
Cloudian - WikiVisually. https://wikivisually.com/wiki/Cloudian. Accessed 8 Apr 2018
Rangarao, D., Gucer, V.: IBM Cloud Object Storage. Redbooks.ibm.com (2018). http://www.redbooks.ibm.com/redpapers/pdfs/redp5435.pdf. Accessed 11 Jul 2018
Chandy, J.A.: Parity Redundancy Strategies in A Large Scale Distributed Storage System. Department of Electrical and Computer Engineering, University of Connecticut. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.6452&rep=rep1&type=pdf. Accessed 12 Jul 2018
Xin, Q., et al.: Reliability mechanisms for very large storage systems. In: Proceedings of 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2003), pp. 146–156. IEEE Computer Society (2003)
Huseyin, S., et al.: Erasure Coding In Windows Azure Storage (2012). https://www.cs.princeton.edu/courses/archive/spring13/cos598C/atc12-final181.pdf. Accessed 12 Jul 2018
Huang, C., Simitci, H., Xu, Y., Ogus, A., Calder, B., Gopalan, P., Li, J., Yekhanin, S.: Erasure coding in windows azure storage. In: Usenix Annual Technical Conference, pp. 15–26, June 2012
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Chiniah, A., Ramchurn, N. (2019). Object Storage System Using Replication and Erasure Codes (OSSREC). In: Fleming, P., Lacquet, B., Sanei, S., Deb, K., Jakobsson, A. (eds) Smart and Sustainable Engineering for Next Generation Applications. ELECOM 2018. Lecture Notes in Electrical Engineering, vol 561. Springer, Cham. https://doi.org/10.1007/978-3-030-18240-3_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-18240-3_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-18239-7
Online ISBN: 978-3-030-18240-3
eBook Packages: EngineeringEngineering (R0)