Advertisement

Big Data Performance Analysis on a Hadoop Distributed File System Based on Modified Partitional Clustering Algorithm

  • V. Santhana MarichamyEmail author
  • V. Natarajan
Conference paper
Part of the Lecture Notes on Data Engineering and Communications Technologies book series (LNDECT, volume 39)

Abstract

This paper proposes a Big Data Performance Analysis based on a modified Partitional Clustering Algorithm (PCA) on a Hadoop Distributed File System (HDFS) which is commonly used in various business applications. This paper has utilized an improved K-means clustering algorithm, which selects the initial clustering centers based on the density parameters. After calculating the density parameter, the data with largest density parameter is selected as the first initial clustering center point, all the left data in the field is deleted from the dataset. By repeating the above phases, K initial clustering centers are found. A new method to improve the precision and packing effect of the K-means computation is needed as there is a poor assurance of finding an initial centers. The proposed approach does not select the initial clustering algorithm randomly, so the stable K value can be obtained by calculating Variance based Cluster Validity Index (VCVI). The performance of the proposed method is evaluated with the parameters Precision, Clustering time and Recall. The experimental result shows that the proposed approach reduces the complexity along with various parameters are compared with existing methods.

Keywords

HDFS PCA VCVI K-means cluster Density parameter Clustering time Recall 

References

  1. 1.
    Craven, M., Page, C.: Big data in healthcare: opportunities and challenges. Big Data 3(4), 209–210 (2015)CrossRefGoogle Scholar
  2. 2.
    Wolfert, S., Ge, L., Verdouw, C., Bogaardt, M.: Big data in smart farming – a review. Agric. Syst. 153, 69–80 (2017)CrossRefGoogle Scholar
  3. 3.
    Liang, F., Yu, W., An, D., Yang, Q., Fu, X., Zhao, W.: A survey on big data market: pricing, trading and protection. IEEE Access 6, 15132–15154 (2018)CrossRefGoogle Scholar
  4. 4.
    Mellor, J., Stone, M., Keane, J.: Application of data mining to “big data” acquired in audiology: principles and potential. Trends Hear. 22, 233121651877681 (2018)CrossRefGoogle Scholar
  5. 5.
    Anwar, M., Oakes, M.: Data mining of audiology patient records: factors influencing the choice of hearing aid type. In: BMC Medical Informatics and Decision Making, vol. 12, no. 1 (2012)Google Scholar
  6. 6.
    Eluri, V., Ramesh, M., Al-Jabri, A., Jane, M.: A comparative study of various clustering techniques on big data sets using Apache Mahout. In: 2016 3rd MEC International Conference on Big Data and Smart City (ICBDSC) (2016)Google Scholar
  7. 7.
    Zhu, E., Ma, R.: An effective Partitional clustering algorithm based on new clustering validity index. Appl. Soft Comput. 71, 608–621 (2018)CrossRefGoogle Scholar
  8. 8.
  9. 9.
    Borthakur, D.: The hadoop distributed file system: architecture and design. Hadoop Project Website, pp. 1–14 (2007)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Computer ApplicationsSRM Valliammai Engineering CollegeChennaiIndia
  2. 2.Department of Instrumentation EngineeringAnna UniversityChennaiIndia

Personalised recommendations