Abstract
Big data systems are designed to manage and analyze huge amounts of data that can reach multiple yottabytes. Those systems often rely on Distributed File System (DFS) environments to store data and optimize their processing and exploitation such as HDFS (Hadoop File System) or GFS (Google File System). The particularity of a DFS cluster is to allow Big Data systems to store data on a large number of machines, while guaranteeing their accessibility, their security and the optimization of access.
DFS often uses data replication and distribution on several machines to ensure their durability, optimize paralleled queries, and enable data recovery in case of machine failure. The data placement on machines of a cluster generally respects a strategy that promotes data security (having replicas on different machines, racks, sites …). However, it often ignores the optimization of response times, which depends on the capacities of storage machines, the network and the intensity of use of this data.
In this study, we propose to predict response time of a DFS Cluster, by taking into account a set of criteria such as locations, file sizes …, using the techniques of “Deep Learning”.
Once the response time is predicted, this will allow us to re-configure the system in order to modify the proposed storage locations and make the strategy of files placement more effective.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
kSI2000 is the unit for 1000 of the CINT2000 measurement unit, an Intel P4 Xeon at 2.8 GHz is approximately 1 kSI2000, see www.spec.org).
References
Mostafa, N., Al Ridhawi, I., Hamza, A.: An intelligent dynamic replica selection model within grid systems. In: IEEE 8th GCC Conference & Exhibition (2015)
Nithya, M., Maheshwari, N.U.: Load rebalancing for Hadoop Distributed File System using distributed hash table. In: International Conference on Intelligent Sustainable Systems (2017)
Beigrezaei, M., Toroghi Haghighat, A., Rashidy Kanan, H.: A new fuzzy based dynamic data replication algorithm in data grids. In: 13th Iranian Conference on Fuzzy Systems (2013)
Abawajy, J.H., Deris, M.M.: Data Replication Approach with Consistency Guarantee for Data Grid. IEEE Trans. Comput. 63(12), 2975–2987 (2014)
Hua, X., Wu, H., Ren, S.: Enhancing throughput of hadoop distributed file system for interaction-intensive tasks. In: 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (2014)
Bhatt, G., Bhavsar, M.: Performance analysis of local, network and distributed file systems running inside user’s virtual machines in cloud environment. Adv. Model. Anal. B 61(1), 48–55 (2018)
Kamal, K., Anyanwu, K.: Scheduling hadoop jobs to meet deadlines. In: IEEE Second International Conference on Cloud Computing Technology and Science (2010)
Bell, W.H., Cameron, D.G., Millar, A.P., Capozza, L., Stockinger, K., Zini, F.: Optorsim: a grid simulator for studying dynamic data replication strategies. Int. J. High Perform. Comput. Appl. 17(4), 403–416 (2003)
What is machine learning. https://cloud.google.com
da Silva, I.N., Hernane Spatti, D., Andrade Flauzino, R., Liboni, L.H.B., dos Reis Alves, S.F.: Artificial Neural Networks. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-43162-8
Yazan, E., Talu, M.F.: Comparison of the stochastic gradient descent based optimization techniques. In: International Artificial Intelligence and Data Processing Symposium (IDAP) (2017)
Sai, Y., Jinxia, R., Zhongxia, L.: Learning of neural networks based on weighted mean squares error function. In: Second International Symposium on Computational Intelligence and Design (2009)
Mendo, L.: Estimation of a probability with guaranteed normalized mean absolute error. IEEE Commun. Lett. 13(11), 817–819 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Elomari, A., Hassouni, L., Maizate, A. (2021). DFS Response Time Prediction Using the Techniques of “Deep Learning”. In: Masrour, T., El Hassani, I., Cherrafi, A. (eds) Artificial Intelligence and Industrial Applications. A2IA 2020. Lecture Notes in Networks and Systems, vol 144. Springer, Cham. https://doi.org/10.1007/978-3-030-53970-2_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-53970-2_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-53969-6
Online ISBN: 978-3-030-53970-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)