Simultaneous model-based clustering and visualization in the Fisher discriminative subspace

Statistics and Computing volume 22, pages301–324(2012)Cite this article

Abstract

Clustering in high-dimensional spaces is nowadays a recurrent problem in many scientific domains but remains a difficult task from both the clustering accuracy and the result understanding points of view. This paper presents a discriminative latent mixture (DLM) model which fits the data in a latent orthonormal discriminative subspace with an intrinsic dimension lower than the dimension of the original space. By constraining model parameters within and between groups, a family of 12 parsimonious DLM models is exhibited which allows to fit onto various situations. An estimation algorithm, called the Fisher-EM algorithm, is also proposed for estimating both the mixture parameters and the discriminative subspace. Experiments on simulated and real datasets highlight the good performance of the proposed approach as compared to existing clustering methods while providing a useful representation of the clustered data. The method is as well applied to the clustering of mass spectrometry data.

This is a preview of subscription content, log in to check access.

References

  1. Agrawal, R., Gehrke, J., Gunopulos, D., Raghavan, P.: Automatic subspace clustering of high-dimensional data for data mining application. In: ACM SIGMOD International Conference on Management of Data, pp. 94–105 (1998)

    Google Scholar 

  2. Akaike, H.: A new look at the statistical model identification. IEEE Trans. Autom. Control 19(6), 716–723 (1974)

    MathSciNet  MATH  Article  Google Scholar 

  3. Alexandrov, T., Decker, J., Mertens, B., Deelder, A., Tollenaar, R., Maass, P., Thiele, H.: Biomarker discovery in MALDI-TOF serum protein profiles using discrete wavelet transformation. Bioinformatics 25(5), 643–649 (2009)

    Article  Google Scholar 

  4. Anderson, E.: The irises of the Gaspé Peninsula. Bull. Am. Iris Soc. 59, 2–5 (1935)

    Google Scholar 

  5. Baek, J., McLachlan, G., Flack, L.: Mixtures of factor analyzers with common factor loadings: applications to the clustering and visualisation of high-dimensional data. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1298–1309 (2010)

    Article  Google Scholar 

  6. Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)

    Google Scholar 

  7. Biernacki, C., Celeux, G., Govaert, G.: Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Trans. Pattern Anal. Mach. Intell. 22(7), 719–725 (2000)

    Article  Google Scholar 

  8. Biernacki, C., Celeux, G., Govaert, G.: Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models. Comput. Stat. Data Anal. 41, 561–575 (2003)

    MathSciNet  Article  Google Scholar 

  9. Bishop, C., Svensen, M.: The generative topographic mapping. Neural Comput. 10(1), 215–234 (1998)

    Article  Google Scholar 

  10. Boutemedjet, S., Bouguila, N., Ziou, D.: A hybrid feature extraction selection approach for high-dimensional non-Gaussian data clustering. IEEE Trans. PAMI 31(8), 1429–1443 (2009)

    Article  Google Scholar 

  11. Bouveyron, C., Girard, S., Schmid, C.: High-dimensional data clustering. Comput. Stat. Data Anal. 52(1), 502–519 (2007)

    MathSciNet  MATH  Article  Google Scholar 

  12. Campbell, N.: Canonical variate analysis: a general model formulation. Aust. J. Stat. 28, 86–96 (1984)

    Google Scholar 

  13. Celeux, G., Diebolt, J.: The SEM algorithm: a probabilistic teacher algorithm from the EM algorithm for the mixture problem. Comput. Stat. Q. 2(1), 73–92 (1985)

    Google Scholar 

  14. Celeux, G., Govaert, G.: A classification E.M. algorithm for clustering and two stochastic versions. Comput. Stat. Data Anal. 14, 315–332 (1992)

    MathSciNet  MATH  Article  Google Scholar 

  15. Clausi, D.A.: K-means Iterative Fisher (KIF) unsupervised clustering algorithm applied to image texture segmentation. Pattern Recognit. 35, 1959–1972 (2002)

    MATH  Article  Google Scholar 

  16. Ding, C., Li, T.: Adaptative dimension reduction using discriminant analysis and k-means clustering. In: ICML (2007)

  17. Duda, R., Hart, P., Stork, D.: Pattern Classification. Wiley, New York (2000)

    Google Scholar 

  18. Fisher, R.: The use of multiple measurements in taxonomic problems. Ann. Eugen. 7, 179–188 (1936)

    Article  Google Scholar 

  19. Foley, D., Sammon, J.: An optimal set of discriminant vectors. IEEE Trans. Comput. 24, 281–289 (1975)

    MATH  Article  Google Scholar 

  20. Fraley, C., Raftery, A.: MCLUST: software for model-based cluster analysis. J. Classif. 16, 297–306 (1999)

    MATH  Article  Google Scholar 

  21. Fraley, C., Raftery, A.: Model-based clustering, discriminant analysis, and density estimation. J. Am. Stat. Assoc. 97(458) (2002)

  22. Friedman, J.: Regularized discriminant analysis. J. Am. Stat. Assoc. 84, 165–175 (1989)

    Article  Google Scholar 

  23. Fukunaga, K.: Introduction to Statistical Pattern Recognition. Academic Press, San Diego (1990)

    Google Scholar 

  24. Golub, G., Van Loan, C.: Matrix Computations, 2nd edn. Hopkins University Press, Baltimore (1991)

    Google Scholar 

  25. Guo, Y.F., Li, S.J., Yang, J.Y., Shu, T.T., Wu, L.D.: A generalized Foley-Sammon transform based on generalized Fisher discriminant criterion and its application to face recognition. Pattern Recognit. Lett. 24, 147–158 (2003)

    MATH  Article  Google Scholar 

  26. Hamamoto, Y., Matsuura, Y., Kanaoka, T., Tomita, S.: A note on the orthonormal discriminant vector method for feature extraction. Pattern Recognit. 24(7), 681–684 (1991)

    Article  Google Scholar 

  27. Hastie, T., Buja, A., Tibshirani, R.: Penalized discriminant analysis. Ann. Stat. 23, 73–102 (1995)

    MathSciNet  MATH  Article  Google Scholar 

  28. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning, 2nd edn. Springer, New York (2009)

    Google Scholar 

  29. Howland, P., Park, H.: Generalizing discriminant analysis using the generalized singular value decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 26(8), 995–1006 (2004)

    Article  Google Scholar 

  30. Jain, A., Marty, M., Flynn, P.: Data clustering: a review. ACM Comput. Surv. 31(3), 264–323 (1999)

    Article  Google Scholar 

  31. Jin, Z., Yang, J., Hu, Z., Lou, Z.: Face recognition based on the uncorrelated optimal discriminant vectors. Pattern Recognit. 10(34), 2041–2047 (2001)

    Article  Google Scholar 

  32. Jolliffe, I.: Principal Component Analysis. Springer, New York (1986)

    Google Scholar 

  33. Kimeldorf, G., Wahba, G.: Some results on Tchebycheffian spline functions. J. Math. Anal. Appl. 33(1), 82–95 (1971)

    MathSciNet  MATH  Article  Google Scholar 

  34. Krzanowski, W.: Principles of Multivariate Analysis. Oxford University Press, Oxford (2003)

    Google Scholar 

  35. la Torre Frade, F.D., Kanade, T.: Discriminative cluster analysis. In: ICML, pp. 241–248 (2006)

  36. Law, M., Figueiredo, M., Jain, A.: Simultaneous feature selection and clustering using mixture models. IEEE Trans. PAMI 26(9), 1154–1166 (2004)

    Article  Google Scholar 

  37. Liu, K., Cheng, Y.Q., Yang, J.Y.: A generalized optimal set of discriminant vectors. Pattern Recognit. 25(7), 731–739 (1992)

    Article  Google Scholar 

  38. Maugis, C., Celeux, G., Martin-Magniette, M.L.: Variable selection for clustering with Gaussian mixture models. Biometrics 65(3), 701–709 (2009)

    MathSciNet  MATH  Article  Google Scholar 

  39. McLachlan, G., Krishnan, T.: The EM algorithm and extensions. Wiley, New York (1997)

    Google Scholar 

  40. McLachlan, G., Peel, D.: Finite Mixture Models. Wiley, New York (2000)

    Google Scholar 

  41. McLachlan, G., Peel, D., Bean, R.: Modelling high-dimensional data by mixtures of factor analyzers. Comput Stat. Data Anal. 41, 379 (2003)

    MathSciNet  Article  Google Scholar 

  42. McNicholas, P., Murphy, B.: Parsimonious Gaussian mixture models. Stat. Comput. 18(3), 285–296 (2008)

    MathSciNet  Article  Google Scholar 

  43. Montanari, A., Viroli, C.: Heteroscedastic factor mixture analysis. Stat. Model. 10(4), 441–460 (2010)

    MathSciNet  Article  Google Scholar 

  44. Parsons, L., Haque, E., Liu, H.: Subspace clustering for high dimensional data: a review. SIGKDD Explor. Newsl. 6(1), 69–76 (1998)

    Google Scholar 

  45. Raftery, A., Dean, N.: Variable selection for model-based clustering. J. Am. Stat. Assoc. 101(473), 168–178 (2006)

    MathSciNet  MATH  Article  Google Scholar 

  46. Rubin, D., Thayer, D.: EM algorithms for ML factor analysis. Psychometrika 47(1), 69–76 (1982)

    MathSciNet  MATH  Article  Google Scholar 

  47. Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6, 461–464 (1978)

    MATH  Article  Google Scholar 

  48. Scott, D., Thompson, J.: Probability density estimation in higher dimensions. In: Fifteenth Symposium in the Interface, pp. 173–179. (1983)

    Google Scholar 

  49. Tipping, E., Bishop, C.: Mixtures of probabilistic principal component analysers. Neural Comput. 11(2), 443–482 (1999)

    Article  Google Scholar 

  50. Trendafilov, N., Jolliffe, I.T.: DALASS: variable selection in discriminant analysis via the LASSO. Comput. Stat. Data Anal. 51, 3718–3736 (2007)

    MathSciNet  MATH  Article  Google Scholar 

  51. Verleysen, M., François, D.: The curse of dimensionality in data mining and time series prediction. In: IWANN (2005)

  52. Ye, J.: Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems. J. Mach. Learn. Res. 6, 483–502 (2005)

    MathSciNet  MATH  Google Scholar 

  53. Ye, J., Zhao, Z., Wu, M.: Discriminative k-means for clustering. Adv. Neural Inf. Process. Syst. 20, 1649–1656 (2007)

    Google Scholar 

Download references

Author information

Affiliations

  1. Laboratoire SAMM, EA 4543, Université Paris 1 Panthéon-Sorbonne, 90 rue de Tolbiac, 75013, Paris, France

    Charles Bouveyron

  2. IBISC, TADIB, FRE CNRS 3190, Université d’Evry Val d’Essonne, 40 rue de Pelvoux, 91020, Evry Courcouronnes, France

    Camille Brunet

Authors
  1. Charles Bouveyron
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Camille Brunet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Camille Brunet.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Bouveyron, C., Brunet, C. Simultaneous model-based clustering and visualization in the Fisher discriminative subspace. Stat Comput 22, 301–324 (2012). https://doi.org/10.1007/s11222-011-9249-9

Download citation

Keywords

  • High-dimensional clustering
  • Model-based clustering
  • Discriminative subspace
  • Fisher criterion
  • Visualization
  • Parsimonious models