Abstract
Although various performance characteristics of distributed file system have been documented, the potential performance efficiency of distributed file system on virtualized cloud computing infrastructure is not clear. This chapter focuses on the performance of Hadoop Distributed File System (HDFS) on virtualized Hadoop. We construct a virtualized Hadoop platform and perform a series of experiments to investigate the performance of HDFS on the virtualized Hadoop cluster. Experimental results verify the efficiency of distributed file system on virtualized Hadoop to process the mass-intensive application.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cheng, K., & Wang, N. (2012). The feasibility research of cloud storage based on global file system. In Proceeding of 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery (pp. 2507–2511). Piscataway, NJ: IEEE.
Konstantin, S., Hairong, K., Sanjay, R., et al. (2010). The Hadoop distributed file system. In Proceedings of the 2010 I.E. 26th Symposium on Mass Storage Systems and Technologies (pp. 1–10). Washington, DC: IEEE Computer Society.
Sun Microsystems Inc. (2010) Using Lustre with Apache Hadoop. White Paper. pp. 1–25.
Xyratex Inc. (2011). Map/reduce on Lustre. White Paper. pp. 1–16.
Wang, F., Yue, Y. L., Feng, D., et al. (2007). High availability storage system based on two-level metadata management. In Proceedings of the 2007 Japan–China Joint Workshop on Frontier of Computer Science and Technology (pp. 41–48). Washington, DC: IEEE Computer Society.
Yu, W., Vetter, J. S., Canon, R. S., et al. (2007). Exploiting lustre file joining for effective collective IO. In Proceeding of the Seventh IEEE International Symposium on Cluster Computing and the Grid (pp. 267–274). Washington, DC: IEEE Computer Society.
Yu, W., Vetter, J. S., & Oral, H. S. (2008). Performance characterization and optimization of parallel I/O on the Cray XT. In Proceeding of the 2008 I.E. International Symposium on Parallel and Distributed Processing (pp. 1–11). Piscataway, NJ: IEEE.
Li, H. Y., Liu, Y., & Cao, Q. (2008). Approximate parameters analysis of a closed fork-join queue model in an object-based storage system. In Proceeding of the Eighth International Symposium on Optical Storage and 2008 International Workshop on Information Data Storage (pp. 1–8). Bellingham, WA: SPIE.
Yu, W., Oral, H. S., Canon, R. S., et al. (2008). Empirical analysis of a large-scale hierarchical storage system. In Euro-Par 2008, LNCS, 5168 (pp. 130–140). Berlin: Springer.
Piernas, J., Nieplocha, J., & Felix, E. J. (2007). Evaluation of active storage strategies for the lustre parallel file system. In Proceeding of the 2007 ACM/IEEE Conference on Supercomputing (pp. 1–8). New York, NY: ACM.
Acknowledgements
This work is supported by the Natural Science Foundation of Guangdong Province, China (Grant No. S2012040007746),the Scientific Research Foundation for Doctors of DGUT(ZJ130604), the National Natural Science Foundation of China (Grant No. 61170216, 10805019, 61272200).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Zhao, T., Zhang, Z., Yuan, H. (2014). Analysis of Distributed File Systems on Virtualized Cloud Computing Environment. In: Wong, W.E., Zhu, T. (eds) Computer Engineering and Networking. Lecture Notes in Electrical Engineering, vol 277. Springer, Cham. https://doi.org/10.1007/978-3-319-01766-2_94
Download citation
DOI: https://doi.org/10.1007/978-3-319-01766-2_94
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-01765-5
Online ISBN: 978-3-319-01766-2
eBook Packages: EngineeringEngineering (R0)