Cluster Validation for High-Dimensional Datasets

  • Minho Kim
  • Hyunjin Yoo
  • R. S. Ramakrishna
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3192)


Cluster validation is the process of evaluating performance of cluster-ing algorithms under varying input conditions. This paper presents a new solu-tion to the problem of cluster validation in high-dimensional applications. We examine the applicability of conventional cluster validity indices in evaluating the results of high-dimensional clustering and propose new indices that can be applied to high-dimensional datasets. We also propose an algorithm for auto-matically determining cluster dimension. By utilizing the proposed indices and the algorithm, we can discard the input parameters that PROCLUS needs. Ex-perimental studies show that the proposed cluster validity indices yield better cluster validation performance than is possible with conventional indices.


high-dimensional clustering cluster validity index unsupervised learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agrawal, R., Gehrke, J., Gunopulos, D., Raghavan, P.: Automatic Subspace Clustering of High Dimensional Data for Data Mining Applications. In: Proc. ACM-SIGMOD Int’l Conf. Management of Data, pp. 94–105 (1998)Google Scholar
  2. 2.
    Aggarwal, C.C., Procopiuc, C., Wolf, J.L., Yu, P.S., Park, J.S.: Fast Algorithms for Projected Clustering. In: Proc. ACM SIGMOD Int’l Conf. Management of Data, pp. 61–72 (1999)Google Scholar
  3. 3.
    Aggarwal, C.C., Yu, P.S.: Redefining Clustering for High-Dimensional Applications. IEEE Trans. Knowledge and Data Engineering 14(2), 210–225 (2002)CrossRefGoogle Scholar
  4. 4.
    Berry, M.J.A., Linoff, G.: Data Mining Techniques: For Marketing, Sales, and Customer Support. John Wiley & Sons, Chichester (1997)Google Scholar
  5. 5.
    Beyer, K.S., Goldstein, J., Ramakrishnan, R., Shaft, U.: When Is ‘Nearest Neighbor’ Meaningful? In: Beeri, C., Bruneman, P. (eds.) ICDT 1999. LNCS, vol. 1540, pp. 217–235. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  6. 6.
    Davies, D.L., Bouldin, D.W.: A Cluster Separation Measure. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 1(2), 224–227 (1979)CrossRefGoogle Scholar
  7. 7.
    Dunn, J.C.: A Fuzzy Relative of the ISODATA Process and its Use in Detecting Compact Well-Separated Clusters. J. Cybernetics 3, 32–57 (1973)zbMATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Ester, M., Kriegel, H.-P., Sander, J., Xu, X.: A Density-based Algorithm for Discovering Clusters in Large Spatial Databases. In: Proc. Int’l Conf. Knowledge Discovery and Data Mining (KDD), pp. 226–231 (1996)Google Scholar
  9. 9.
    Halkidi, M., Vazirgiannis, M.: Quality Scheme Assessment in the Clustering Process. In: Zighed, D.A., Komorowski, J., Żytkow, J.M. (eds.) PKDD 2000. LNCS (LNAI), vol. 1910, pp. 265–276. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  10. 10.
    Halkidi, M., Vazirgiannis, M.: Clustering Validity Assessment: Finding the Optimal Partitioning of a Dataset. In: Proc. Int’l Conf. Data Mining (ICDM), pp. 187–194 (2001)Google Scholar
  11. 11.
    Han, J., Kamber, M.: Data Mining: Concepts and Techniques. Morgan Kaufmann, San Francisco (2001)Google Scholar
  12. 12.
    Hinneburg, A., Keim, D.A.: An Optimal Grid-Clustering: Towards Breaking the Curse of Dimensionality in High-Dimensional Clustering. In: Proc. Int’l Conf. Very Large DataBases (VLDB), pp. 506–517 (1999)Google Scholar
  13. 13.
    Jain, A.K., Murty, M.N., Flynn, P.J.: Data Clustering: A Review. ACM Computing Surveys 31(3), 264–323 (1999)CrossRefGoogle Scholar
  14. 14.
    Kim, D.-J., Park, Y.-W., Park, D.-J.: A Novel Validity Index for Determination of the Optimal Number of Clusters. IEICE Trans. Inf. & Syst. E84-D(2), 281–285 (2001)Google Scholar
  15. 15.
    Maulik, U., Bandyopadhyay, S.: Performance Evaluation of Some Clustering Algorithms and Validity Indices. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 24(12), 1650–1654 (2002)CrossRefGoogle Scholar
  16. 16.
    Schwarz, G.: Estimating the Dimension of a Model. Annals of Statistics 6(2), 461–464 (1978)zbMATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Xie, X.L., Beni, G.A.: A Validity Measure for Fuzzy Clustering. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 3(8), 841–846 (1991)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Minho Kim
    • 1
  • Hyunjin Yoo
    • 1
  • R. S. Ramakrishna
    • 1
  1. 1.Department of Information and Communications GISTGwangjuRepublic of Korea

Personalised recommendations