Abstract
I.T. industries and private organizations generate a massive volume of data every day. Storing and processing Big data is challenging due to scalability and performance issues. Nowadays, a distributed architecture is used to process Big data. In a distributed architecture, several nodes/systems communicate to store and process data in a distributed architecture. Search engines use distributed architecture to store and retrieve documents for the user query. Elasticsearch is an open-source search engine, which uses distributed architecture. The main goal of this paper is to configure elastic search clusters, implement the shard selection algorithms, and perform the comparative study analysis of the existing shard selection techniques with the proposed shard selection technique. The sharding technique is applied to partition and retrieve relevant data from the nodes. Shards are created on each data node of the cluster. Shard is the small unit of storage in the memory of the data node. Data is horizontally partitioned according to topic-based and stored on different shards. This paper proposes a Modified ReDDE shard selection algorithm that enhances the throughput by searching only the relevant shards in the distributed processing architecture instead of all the shards. The results interpret that the Modified ReDDE algorithm improves the performance parameters compared to existing shard selection techniques by 26%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
P. Berglund, Shard selection in distributed collaborative search engines, a design, implementation and evaluation of shard selection in ElasticSearch, University of Gothenburg (2013)
A. Kulkarni, J. Callan, Selective Search: Efficient and Effective Search of Large Textual Collections (San Francisco State University, 2015)
A. L'Heureux, Machine Learning with Big Data: Challenges and Approaches (The University of Western Ontario, 2017)
M.D. Praveen, Vijay, S.G. Totad, Performance Analysis of Distributed Processing System using Shard Selection Technique in Elasticsearch (KLE Technological University, 2019)
E. Rodrigues, R. Morlay, Run Time Prediction for Big Data Iterative ML Algorithms: A KMeans Case Study (Faculty of Engineering, University of Porto, Porto, 2017)
J.P. Callan, Z. Lu, W.B. Croft, Searching distributed collections with inference networks, in Proceedings of the 18th Annual İnternational ACM SIGIR Conference on Research and Development in İnformation Retrieval (ACM, 1995)
P. Dhulavvagol, V. Bhajantri, S. Totad, Performance analysis of distributed processing system using shard selection techniques on elasticsearch. Procedia Comput. Sci. 167, 1626–1635 (2020). https://doi.org/10.1016/j.procs.2020.03.373
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Dhulavvagol, P.M., Totad, S.G., Bhandage, N., Bilagi, P. (2022). Efficient Data Partitioning and Retrieval Using Modified ReDDE Technique. In: Jacob, I.J., Kolandapalayam Shanmugam, S., Bestak, R. (eds) Expert Clouds and Applications. Lecture Notes in Networks and Systems, vol 444. Springer, Singapore. https://doi.org/10.1007/978-981-19-2500-9_13
Download citation
DOI: https://doi.org/10.1007/978-981-19-2500-9_13
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-2499-6
Online ISBN: 978-981-19-2500-9
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)