Abstract
The clustering of datasets is a widely used technique in unsupervised machine learning. The cluster quality evaluation is a tricky problem because external validation is usually not possible for clustering. This happens due to the unavailability of external proof. Although there are many methods developed and experimented with to validate the results obtained from clustering, it is always preferred to use more than a few cluster validity measures. The big data usually contain a high percentage of noise in it, making it necessary to use some additional techniques to incorporate while clustering big data. In this work, we have clustered document big data using fuzzy logic-based enhancement of traditional K-Means and K-Medoids. This work has suggested the Ensembling of seven different cluster quality validity measures to determine the best quality of fuzzy clusters. The Reuters standard document dataset is clustered using different cluster numbers, and the proposed ensemble methodology is proven to determine optimal numbers of clusters.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Sardar TH, Ansari Z (2022a) Distributed big data clustering using MapReduce-based fuzzy C-medoids. J Inst Eng India Ser B 103:73–82. https://doi.org/10.1007/s40031-021-00647-w
Sardar TH, Ansari Z (2022b) MapReduce-based fuzzy C-means algorithm for distributed document clustering. J Inst Eng India Ser B 103:131–142. https://doi.org/10.1007/s40031-021-00651-0
Sardar TH, Ansari Z (2020) An analysis of distributed document clustering using MapReduce based K-means algorithm. J Inst Eng India Ser B 101:641–650. https://doi.org/10.1007/s40031-020-00485-2
Ansari Z, Afzal A, Sardar TH (2019) Data categorization using Hadoop MapReduce-based parallel K-means clustering. J Inst Eng India Ser B 100:95–103. https://doi.org/10.1007/s40031-019-00388-x
Sardar TH, Ansari Z (2018a) An analysis of MapReduce efficiency in document clustering using parallel K-means algorithm. Future Comput Inform J 3(2):200–209
Sardar TH, Ansari Z (2018b) Partition based clustering of large datasets using MapReduce framework: an analysis of recent themes and directions. Future Comput Inform J 3(2):247–261
Marutho D, Handaka SH, Wijaya E (2018) The determination of cluster number at k-mean using elbow method and purity evaluation on headline news. In: 2018 international seminar on application for technology of information and communication. IEEE, pp 533–538
Nazari A, Dehghan A, Nejatian S et al (2019) A comprehensive study of clustering ensemble weighting based on cluster quality and diversity. Pattern Anal Appl 22:133–145. https://doi.org/10.1007/s10044-017-0676-x
Raskutti B, Leckie C (1999) An evaluation of criteria for measuring the quality of clusters. IJCAI 99:905–910
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Sardar, T.H., Sarkar, R., Ahmed, S., Bandyopadhyay, A. (2023). A Novel Ensemble Methodology to Validate Fuzzy Clusters of Big Data. In: Kaiser, M.S., Waheed, S., Bandyopadhyay, A., Mahmud, M., Ray, K. (eds) Proceedings of the Fourth International Conference on Trends in Computational and Cognitive Engineering. Lecture Notes in Networks and Systems, vol 618. Springer, Singapore. https://doi.org/10.1007/978-981-19-9483-8_23
Download citation
DOI: https://doi.org/10.1007/978-981-19-9483-8_23
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-9482-1
Online ISBN: 978-981-19-9483-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)