Abstract
With the tremendous increase in the amount of data being generated from variety of sources there is a need of efficient data storage and processing techniques. Some of the sources generating this large amount of data are Weather Sensors, Scientific experiments, etc. This huge voluminous data is termed as BigData. Due to ever-increasing amount of data there is a demand for faster data ingestion and processing. Apache Spark, a dominant processing tool is a publicly available platform for processing outsized data and is mostly intended for iterative machine learning jobs. In this study, an integrated approach i.e., Spark MLlib Clustering on batch weather data stored in Cassandra database is proposed. This helps to analyze our data into number of Clusters which is required and useful for further examination of data. The main idea of this study is to evaluate Batch Processing performance of an integrated approach with two popular clustering algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
X. Meng, MLlib: machine learning in Apache Spark. J. Mach. Learn. Res. 17 (2016)
A. Ghosh, A. Kumar Pasayat, Identifying spam SMS using Apache Spark Mllib. J. Emerg. Technol. Innov. Res. 5(5) (2018). ISSN: 2349-5162
T. Nelson Gnanaraj, K. Ramesh Kumar, N. Monica, Survey on mining clusters using new k-mean algorithm from structured and unstructured data. Int. J. Adv. Comput. Sci. Technol. 3(2) (2014). ISSN: 2320-2602
S. Harifi, E. Byagowi, M. Khalilian, Comparative Study of Apache Spark MLlib Clustering Algorithms Conference Paper (2017)
K. Abirami, P. Mayilvahanan, Performance analysis of K-means and bisecting K-means algorithms in Weblog data. Int. J. Emerg. Technol. Eng. Res. (IJETER) 4(8) (2016)
M. Assefi, E. Behravesh, G. Liu, A.P. Tafti, Big data machine learning using Apache Spark MLlib, in Conference IEEE Big Data 2017, Boston, USA (2017)
A. Chaudhari, P. Mulay, SCSI: Real-Time Data Analysis with Cassandra and Spark Research Gate (2019)
D. Jayanthi, G. Sumathi, Weather data analysis using spark—an in-memory computing framework, in International Conference on Innovations in Power and Advanced Computing Technologies (2017)
https://dzone.com/articles/cluster-analysis-using-apache-spark-exploring-colo
https://www.tutorialspoint.com/cassandra/cassandra_introduction.htm
K. Anusha, K. UshaRani, Big data techniques for efficient storage and processing of weather data. Int. J. Res. Appl. Sci. Eng. Technol. (IJRASET) 5(VII) (2017). ISSN: 2321-9653
K. Anusha, K. Usha, Rani performance evaluation of Spark SQL for batch processing, in Advances in Intelligent Systems and Computing. Accepted for publication in Springer series
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Anusha, K., UshaRani, K. (2020). Performance Analysis of Apache Spark MLlib Clustering on Batch Data Stored in Cassandra. In: Jyothi, S., Mamatha, D., Satapathy, S., Raju, K., Favorskaya, M. (eds) Advances in Computational and Bio-Engineering. CBE 2019. Learning and Analytics in Intelligent Systems, vol 15. Springer, Cham. https://doi.org/10.1007/978-3-030-46939-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-46939-9_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-46938-2
Online ISBN: 978-3-030-46939-9
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)