Skip to main content
SpringerLink
Log in

Storage Service Reliability and Availability Predictions of Hadoop Distributed File System

  • Conference paper
  • First Online:
Reliability, Safety and Hazard Assessment for Risk-Based Technologies

Part of the book series: Lecture Notes in Mechanical Engineering ((LNME))

Abstract

Hadoop is a de facto standard for Big Data storage and provides a complete arrangement of components for Big Data elaboration. Hadoop Distributed File System (HDFS), the fundamental module of Hadoop, has been evolved to deliver fault-tolerant data storage services in cloud. This work proposes a precise mathematical model of HDFS and estimates its data storage service availability and reliability. In this connection, a stochastic Petri net (SPN)-based dependability modelling strategy is adopted. In addition, a structural decomposition technique has been advocated to address the state space complexity of the said model. The proposed model is useful to measure crucial quality of service parameters, namely storage service reliability and availability for emerging distributed data storage systems in the context of cloud.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Platform-Independent Petri net Editor—http://pipe2.sourceforge.net.

  2. 2.

    A small part of this work has been accepted in IEMIS-2018, Kolkata, India. https://doi.org/10.1007/978-981-13-1498-8_42.

References

  1. Apache Hadoop, http://hadoop.apache.org/. Last access: Aug. 2018

  2. K. Shvachko, H. Kuang, S. Radia, R. Chansler, The Hadoop distributed file system, in Proceedings of 26th symposium on Mass Storage, Systems and Technologies, IEEE (2010), pp. 1–10

    Google Scholar 

  3. D. Chattaraj, M. Sarma, D. Samanta, Stochastic petri net based modeling for analyzing dependability of big data storage system, in Proceeding of 1st International Conference on Emerging Technologies in Data Mining and Information Security (IEMIS’18), Advances in Intelligent Systems and Computing, vol. 813 (2019). https://doi.org/10.1007/978-981-13-1498-8_42

    Google Scholar 

  4. HDFS Architecture, https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html. Last access: Aug. 2018

  5. K. McKusick, S. Quinlan, GFS: evolution on fast-forward. Commun. ACM 53(3), 42–49 (2010)

    Article  Google Scholar 

  6. K.V. Shvachko, H.D.F.S. Scalability, The limits to growth. Mag. USENIX & SAGE 35(2), 6–16 (2010)

    Google Scholar 

  7. D. Bruneo, F. Longo, D. Hadas, H. Kolodner, Availability assessment of a vision cloud storage cluster, in Proceedings of European Conference on Service-Oriented and Cloud Computing (Springer, 2013), pp. 71–82

    Google Scholar 

  8. R. Ghosh, F. Longo, F. Frattini, S. Russo, K.S. Trivedi, Scalable analytics for IaaS cloud availability. Trans. Cloud Comput. IEEE 2(1), 57–70 (2014)

    Article  Google Scholar 

  9. F. Longo, R. Ghosh, V.K. Naik, K.S. Trivedi, A scalable availability model for infrastructure-as-a-service cloud, in Proceedings of 41st International Conference on Dependable Systems & Networks (DSN), IEEE (2011), pp. 335–346

    Google Scholar 

  10. H. Li, Z. Zhao, L. He, Model and analysis of cloud storage service reliability based on stochastic petri nets. J. Inf. Comput. Sci. 11(7), 2341–2354 (2014)

    Article  Google Scholar 

  11. D. Bruneo, F. Longo, D. Hadas, E.K. Kolodner, Analytical investigation of availability in a vision cloud storage cluster. Scalable Comput.: Pract. Experience 14(4), 279–290 (2014)

    Google Scholar 

  12. J. Shafer, S. Rixner, A.L. Cox, The Hadoop distributed file system: balancing portability and performance, in International Symposium on Performance Analysis of Systems & Software (ISPASS-10). IEEE (2010), pp. 122–133

    Google Scholar 

  13. D. Bruneo, A stochastic model to investigate data center performance and QOS in IaaS cloud computing systems. Trans. Parallel Distrib. Syst. IEEE 25(3), 560–569 (2014)

    Article  Google Scholar 

  14. J. Dantas, R. Matos, J. Araujo, P. Maciel, Models for dependability analysis of cloud computing architectures for eucalyptus platform. Int. Trans. Syst. Sci. Appl. 8, 13–25 (2012)

    Google Scholar 

  15. X. Wu, Y. Liu and I. Gorton, Exploring performance models of hadoop applications on cloud architecture, in Proceedings of the 11th International ACM SIGSOFT Conference on Quality of Software Architectures (QoSA), IEEE, (2015), pp. 93–101

    Google Scholar 

  16. J. Wang, Petri nets for dynamic event-driven system modeling, in Handbook of Dynamic System Modeling, edited by P. Fishwick (CRC Press, 2007)

    Google Scholar 

  17. R. Zeng, Y. Jiang, C. Lin, X. Shen, Dependability analysis of control center networks in smart grid using stochastic petri nets. Trans. Parallel Distrib. Syst. IEEE 23(9), 1721–1730 (2012)

    Article  Google Scholar 

  18. S.A. Chamazcoti, S.G. Miremadi, Hybrid RAID: a solution for enhancing the reliability of SSD-based RAIDs. Trans. Multi-Scale Comput. Syst. IEEE 3(3), 181–192 (2015)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Subir Chowdhury School of Quality and Reliability, Indian Institute of Technology Kharagpur, Kharagpur, India

    Durbadal Chattaraj & Monalisa Sarma

  2. Tata Motors, Pune, India

    Sumit Bhagat

Authors
  1. Durbadal Chattaraj
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sumit Bhagat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Monalisa Sarma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Durbadal Chattaraj .

Editor information

Editors and Affiliations

  1. Reactor Group, Bhabha Atomic Research Centre, Mumbai, Maharashtra, India

    Prabhakar V. Varde

  2. Department of Mechanical Engineering, Indian Institute of Technology Madras, Chennai, Tamil Nadu, India

    Raghu V. Prakash

  3. Probabilistic Safety Section, Bhabha Atomic Research Centre, Mumbai, Maharashtra, India

    Gopika Vinod

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chattaraj, D., Bhagat, S., Sarma, M. (2020). Storage Service Reliability and Availability Predictions of Hadoop Distributed File System. In: Varde, P., Prakash, R., Vinod, G. (eds) Reliability, Safety and Hazard Assessment for Risk-Based Technologies. Lecture Notes in Mechanical Engineering. Springer, Singapore. https://doi.org/10.1007/978-981-13-9008-1_52

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-9008-1_52

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-9007-4

  • Online ISBN: 978-981-13-9008-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation