Multivariate Response and Parsimony for Gaussian Cluster-Weighted Models

Abstract

A family of parsimonious Gaussian cluster-weighted models is presented. This family concerns a multivariate extension to cluster-weighted modelling that can account for correlations between multivariate responses. Parsimony is attained by constraining parts of an eigen-decomposition imposed on the component covariance matrices. A sufficient condition for identifiability is provided and an expectation-maximization algorithm is presented for parameter estimation. Model performance is investigated on both synthetic and classical real data sets and compared with some popular approaches. Finally, accounting for linear dependencies in the presence of a linear regression structure is shown to offer better performance, vis-à-vis clustering, over existing methodologies.

This is a preview of subscription content, access via your institution.

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

References

  1. AITKEN, A.C. (1926), “A Series Formula for the Roots of Algebraic and Transcendental Equations, in Proceedings of the Royal Society of Edinburgh, 45, 14–22.

  2. ANDERLUCCI, L., and VIROLI, C. (2015), “Covariance Pattern Mixture Models for Multivariate Longitudinal Data”, The Annals of Applied Statistics, 9(2), 777–800.

    MathSciNet  Article  MATH  Google Scholar 

  3. ANDERSON, E. (1935), “The Irises of the Gaspé Peninsula”, Bulletin of the American Iris Society, 59, 2–5.

    Google Scholar 

  4. ANDERSON, J.A. (1972), “Separate Sample Logistic Discrimination”, Biometrikam, 59(1), 19–35.

    MathSciNet  Article  MATH  Google Scholar 

  5. ANDREWS, J.L., and MCNICHOLAS, P.D. (2012), “Model-Based Clustering, Classification, and Discriminant Analysis via Mixtures of Multivariate t-Distributions”, Statistics and Computing, 22(5), 1021–1029.

    MathSciNet  Article  MATH  Google Scholar 

  6. ANDREWS, J.L., and MCNICHOLAS, P.D. (2014), “Variable Selection for Clustering and Classification”, Journal of Classification, 31(2), 136–153.

    MathSciNet  Article  MATH  Google Scholar 

  7. ANDREWS, J.L., MCNICHOLAS, P.D., and SUBEDI, S. (2011), “Model-Based Classification via Mixtures of Multivariate t-Distributions”, Computational Statistics and Data Analysis, 55(1), 520–529.

    MathSciNet  Article  MATH  Google Scholar 

  8. AZZALINI, A. (2013), sn: The Skew-Normal and Skew-t Distributions, R Package Version 0.4-18.

  9. BAUDRY, J.-P., and CELEUX, G. (2015), “EM for Mixtures”, Statistics and Computing, 25(4), 713–726.

    MathSciNet  Article  MATH  Google Scholar 

  10. BIERNACKI, C., and LOURME, A. (2014), “Stable and Visualizable Gaussian Parsimonious Clustering Models”, Statistics and Computing, 24(6), 953–969.

    MathSciNet  Article  MATH  Google Scholar 

  11. BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2000), “Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7), 719–725.

    Article  Google Scholar 

  12. BÖHNING, D. (1995), “A Review of Reliable Algorithms for the Semi-Parametric Maximum Likelihood Estimator of a Mixture Distribution”, Journal of Statistical Planning and Inference, 47, 5–28.

    MathSciNet  Article  MATH  Google Scholar 

  13. BÖHNING, D. (2000), Computer-Assisted Analysis of Mixtures and Applications: Metaanalysis, Disease Mapping and Others, London: Chapman & Hall/CRC.

  14. BÖHNING, D. (2003), “The EM Algorithm with Gradient Function Update for Discrete Mixtures with Known (Fixed) Number of Components”, Statistics and Computing, 13(3), 257–265.

    MathSciNet  Article  Google Scholar 

  15. BÖHNING, D., DIETZ, E., SCHAUB, R., SCHLATTMANN, P., and LINDSAY, B.G. (1994), “The Distribution of the Likelihood Ratio for Mixtures of Densities from the One-Parameter Exponential Family”, Annals of the Institute of Statistical Mathematics, 46(2), 373–388.

    Article  MATH  Google Scholar 

  16. BOUVEYRON, C., and BRUNET-SAUMARD, C. (2014), “Model-Based Clustering of High-Dimensional Data: A Review”, Computational Statistics and Data Analysis, 71, 52–78.

    MathSciNet  Article  MATH  Google Scholar 

  17. BREIMAN, L., and FRIEDMAN, J.H. (1997), “Predicting Multivariate Responses in Multiple Linear Regression”, Journal of the Royal Statistical Society: Series B, 59(1), 3–54.

    MathSciNet  Article  MATH  Google Scholar 

  18. BROWNE, R.P., and MCNICHOLAS, P.D. (2014a), “Estimating Common Principal Components in High Dimensions”, Advances in Data Analysis and Classification, 8(2), 217–226.

  19. BROWNE, R.P., and MCNICHOLAS, P.D. (2014b), “Orthogonal Stiefel Manifold Optimization for Eigen-Decomposed Covariance Parameter Estimation in Mixture Models”, Statistics and Computing, 24(2), 203–210.

    MathSciNet  Article  MATH  Google Scholar 

  20. BROWNE, R.P., and MCNICHOLAS, P.D. (2015), mixture: Mixture Models for Clustering and Classification, R Package Version 1.1.

  21. BROWNE, R.P., SUBEDI, S., and MCNICHOLAS, P.D. (2013), “Constrained Optimization for a Subset of the Gaussian Parsimonious Clustering Models”, arXiv preprint arXiv:1306.5824.

  22. CAMPBELL, N.A., and MAHON, R.J. (1974), “A Multivariate Study of Variation in Two Species of Rock Crab of the Genus Leptograpsus”, Australian Journal of Zoology, 22(3), 417–425.

    Article  Google Scholar 

  23. CELEUX, G., and DIEBOLT, J. (1988), “A Random Imputation Principle: The Stochastic EM Algorithm”, in Rapports de Recherche 901, INRIA.

  24. CELEUX, G., and GOVAERT, G. (1995), “Gaussian Parsimonious Clustering Models”, Pattern Recognition, 28(5), 781–793.

    Article  Google Scholar 

  25. COOK, R.D., and WEISBERG, S. (1994), An Introduction to Regression Graphics, John Wiley & Sons.

  26. DANG, U.J., and MCNICHOLAS, P.D. (2015), “Families of Parsimonious Finite Mixtures of Regression Models”, in Advances in Statistical Models for Data Analysis, Studies in Classification, Data Analysis and Knowledge Organization, eds. I. Morlini, T. Minerva, and M. Vichi, Switzerland: Springer International Publishing, pp. 73–84.

  27. DANG, U.J., BROWNE, R.P., and MCNICHOLAS, P.D. (2015), “Mixtures of Multivariate Power Exponential Distributions”, Biometrics, 71(4), 1081–1089.

    MathSciNet  Article  MATH  Google Scholar 

  28. DASGUPTA, A., and RAFTERY, A.E. (1998), “Detecting Features in Spatial Point Processes with Clutter via Model-Based Clustering”, Journal of the American Statistical Association, 93(441), 294–302.

    Article  MATH  Google Scholar 

  29. DEMPSTER, A., LAIRD, N., and RUBIN, D. (1977), “Maximum Likelihood from Incomplete Data via the EM Algorithm”, Journal of the Royal Statistical Society: Series B, 39(1), 1–38.

    MathSciNet  MATH  Google Scholar 

  30. DESARBO, W.S., and CRON, W.L. (1988), “A Maximum Likelihood Methodology for Clusterwise Linear Regression”, Journal of Classification, 5(2), 249–282.

    MathSciNet  Article  MATH  Google Scholar 

  31. FISHER, R.A. (1936), “The Use of Multiple Measurements in Taxonomic Problems”, Annals of Eugenics, 7(2), 179–188.

    Article  Google Scholar 

  32. FLURY, B.N., and GAUTSCHI, W. (1986), “An Algorithm for Simultaneous Orthogonal Transformation of Several Positive Definite Matrices to Nearly Diagonal Form”, SIAM Journal on Scientific and Statistical Computing, 7(1), 169–184.

    MathSciNet  Article  MATH  Google Scholar 

  33. FRALEY, C., and RAFTERY, A.E. (2002), “Model-Based Clustering, Discriminant Analysis, and Density Estimation”, Journal of the American Statistical Association, 97(458), 611–631.

    MathSciNet  Article  MATH  Google Scholar 

  34. FRALEY, C., RAFTERY, A.E., MURPHY, T.B., and SCRUCCA, L. (2012), “Mclust Version 4 for R: Normal Mixture Modeling for Model-Based Clustering, Classification, and Density Estimation”, Technical Report 597, Department of Statistics, University of Washington, Seattle, WA.

  35. FRANCZAK, B.C., TORTORA, C., BROWNE, R.P., and MCNICHOLAS, P.D. (2015), “Unsupervised Learning viaMixtures of Skewed Distributions with Hypercube Contours” Pattern Recognition Letters, 58(1), 69–76.

    Article  Google Scholar 

  36. GALIMBERTI, G., and SOFFRITTI,G. (2013), “A Multivariate Linear Regression Analysis Using Finite Mixtures of t Distributions”, Computational Statistics and Data Analysis, 71, 138–150.

    MathSciNet  Article  MATH  Google Scholar 

  37. GALIMBERTI, G., SCARDOVI, E., and SOFFRITTI, G. (2015), “Using Mixtures in Seemingly Unrelated Linear Regression Models with Non-Normal Errors”, Statistics and Computing, 1–14.

  38. GERSHENFELD,N. (1997), “Nonlinear Inference and Cluster-Weighted Modeling”, Annals of the New York Academy of Sciences, 808(1), 18–24.

    Article  Google Scholar 

  39. GRÜN, B., and LEISCH, F. (2008), “Flexmix Version 2: Finite Mixtures with Concomitant Variables and Varying and Constant Parameters”, R package version 2.3.13, Journal of Statistical Software, 28(4), 1–35.

    Article  Google Scholar 

  40. HARTIGAN, J.A., and WONG, M.A. (1979), “A k-Means Clustering Algorithm”, Journal of the Royal Statistical Society: Series C, 28(1), 100–108.

    MATH  Google Scholar 

  41. HASTIE, T., and TIBSHIRANI, R. (1996), “Discriminant Analysis by Gaussian Mixtures”, Journal of the Royal Statistical Society: Series B, 58(1), 155–176.

    MathSciNet  MATH  Google Scholar 

  42. HENNIG, C. (2000), “Identifiablity of Models for Clusterwise Linear Regression”, Journal of Classification, 17(2), 273–296.

    MathSciNet  Article  MATH  Google Scholar 

  43. HENNIG, C., and LIAO, T.F. (2013), “How to Find an Appropriate Clustering for Mixed-Type Variables with Application to Socio-Economic Stratification (With Discussion)”, Journal of the Royal Statistical Society: Series C, 62(3), 309–369.

    MathSciNet  Article  Google Scholar 

  44. HUBERT, L., and ARABIE, P. (1985), “Comparing Partitions”, Journal of Classification, 2(1), 193–218.

    Article  MATH  Google Scholar 

  45. INGRASSIA, S. (2004), “A Likelihood-Based Constrained Algorithm for Multivariate Normal Mixture Models”, Statistical Methods and Applications, 13(2), 151–166.

    MathSciNet  Article  Google Scholar 

  46. INGRASSIA, S., and PUNZO, A. (2016), “Decision Boundaries for Mixtures of Regressions”, Journal of the Korean Statistical Society, DOI: 10.1016/j.jkss.2015.11.005.

  47. INGRASSIA, S., and ROCCI, R. (2007), “Constrained Monotone EM Algorithms for Finite Mixture of Multivariate Gaussians”, Computational Statistics and Data Analysis, 51(11), 5339–5351.

    MathSciNet  Article  MATH  Google Scholar 

  48. INGRASSIA, S., MINOTTI, S.C., and VITTADINI, G. (2012), “Local Statistical Modeling via a Cluster-Weighted Approach with Elliptical Distributions”, Journal of Classification, 29(3), 363–401.

    MathSciNet  Article  MATH  Google Scholar 

  49. INGRASSIA, S., MINOTTI, S.C., and PUNZO, A. (2014), “Model-Based Clustering via Linear Cluster-Weighted Models”, Computational Statistics and Data Analysis, 71, 159–182.

    MathSciNet  Article  Google Scholar 

  50. INGRASSIA, S., PUNZO, A., VITTADINI, G., and MINOTTI, S.C. (2015), “The Generalized Linear Mixed Cluster-Weighted Model”, Journal of Classification, 32(1), 85–113.

    MathSciNet  Article  MATH  Google Scholar 

  51. KERIBIN, C. (1998), “Estimation Consistante de l’Ordre de Modèles deMélange”, Comptes Rendus de l’Académie des Sciences-Series I-Mathematics, 326(2), 243–248.

  52. KERIBIN, C. (2000), “Consistent Estimation of the Order of Mixture Models”, Sankhyā: The Indian Journal of Statistics, Series A, 62(1), 49–66.

  53. LEBRET, R., IOVLEFF, S., LANGROGNET, F., BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2015), “Rmixmod: The R Package of the Model-Based Unsupervised, Supervised, and Semi-Supervised Classification Mixmod Library”, Journal of Statistical Software, 67(6), 1–29.

    Article  Google Scholar 

  54. LEE, S.X., and MCLACHLAN, G.J. (2013), “On Mixtures of Skew Normal and Skew t-Distributions”, Advances in Data Analysis and Classification, 7(3), 241–266.

    MathSciNet  Article  MATH  Google Scholar 

  55. LEISCH, F. (2004), “FlexMix: A General Framework for Finite Mixture Models and Latent Class Regression in R”, Journal of Statistical Software, 11(8), 1–18.

    Article  Google Scholar 

  56. LIN, T.-I., MCNICHOLAS, P.D., and HSIU, J.H. (2014), “Capturing Patterns via Parsimonious t Mixture Models”, Statistics and Probability Letters, 88, 80–87.

    MathSciNet  Article  MATH  Google Scholar 

  57. LINDSAY, B.G. (1995), “Mixture Models: Theory, Geometry and Applications”, in NSFCBMS Regional Conference Series in Probability and Statistics, pp. 1–163.

  58. MACQUEEN, J. (1967), “Some Methods for Classification and Analysis of Multivariate Observations”, in Proceedings of the Fifth Berkeley Symposium onMathematical Statistics and Probability, Vol. 1, Oakland CA: University of California Press, pp. 281–297.

  59. MCLACHLAN, G.J., and BASFORD, K.E. (1988), Mixture Models: Inference and Applications to Clustering, New York: Marcel Dekker.

  60. MCLACHLAN, G.J., and PEEL, D. (2000), Finite Mixture Models, New York: John Wiley & Sons Inc.

    Google Scholar 

  61. MCNICHOLAS, P.D. (2010), “Model-Based Classification Using Latent Gaussian Mixture Models”, Journal of Statistical Planning and Inference, 140(5), 1175–1181.

    MathSciNet  Article  MATH  Google Scholar 

  62. MCNICHOLAS, P.D. (2016a), Mixture Model-Based Classification, Boca Raton FL: Chapman & Hall/CRC Press.

  63. MCNICHOLAS, P.D. (2016b),“Model-Based Clustering”, Journal of Classification 33, in press.

  64. MCNICHOLAS, P.D., MURPHY, T.B., MCDAID, A.F., and FROST, D. (2010), “Serial and Parallel Implementations of Model-Based Clustering via Parsimonious Gaussian Mixture Models”, Computational Statistics and Data Analysis, 54(3), 711–723.

    MathSciNet  Article  MATH  Google Scholar 

  65. MURRAY, P.M., BROWNE, R.P., and MCNICHOLAS, P.D. (2014), “ Mixtures of Skew-t Factor Analyzers”, Computational Statistics and Data Analysis, 77, 326–335.

  66. O’HAGAN, A., MURPHY, T.B., GORMLEY, I.C., MCNICHOLAS, P.D., and KARLIS, D. (2016), “Clustering with the Multivariate Normal Inverse Gaussian Distribution”, Computational Statistics and Data Analysis, 93, 18–30.

    MathSciNet  Article  Google Scholar 

  67. PUNZO, A. (2014), “Flexible Mixture Modeling with the Polynomial Gaussian Cluster-Weighted Model”, Statistical Modelling, 14(3), 257–291.

    MathSciNet  Article  Google Scholar 

  68. PUNZO, A., and INGRASSIA, S. (2013), “On the Use of the Generalized Linear Exponential Cluster-Weighted Model to Assess Local Linear Independence in Bivariate Data”, QdS -Journal of Methodological and Applied Statistics, 15, 131–144.

    Google Scholar 

  69. PUNZO, A., and INGRASSIA, S. (2015), “Clustering Bivariate Mixed-Type Data via the Cluster-Weighted Model”, Computational Statistics, 31(3), 989–1013.

    MathSciNet  Article  MATH  Google Scholar 

  70. PUNZO, A., and MCNICHOLAS, P.D. (2016), “Parsimonious Mixtures of Multivariate Contaminated Normal Distributions”, Biometrical Journal, in press.

  71. PUNZO, A., BROWNE, R.P., and MCNICHOLAS, P.D. (2016), “Hypothesis Testing for MixtureModelling Selection”, Journal of Statistical Computation and Simulation, 86(14), 2797–2818.

    MathSciNet  Article  Google Scholar 

  72. R CORE TEAM(2015), R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.

  73. RAND,W.M. (1971). “Objective Criteria for the Evaluation of ClusteringMethods”, Journal of the American Statistical Association, 66(336), 846–850.

    Article  Google Scholar 

  74. SCHWARZ, G. (1978), “Estimating the Dimension of a Model”, Annals of Statistics, 6(2), 461–464.

    MathSciNet  Article  MATH  Google Scholar 

  75. SOFFRITTI, G., and GALIMBERTI, G. (2011), “Multivariate Linear Regression with Non-Normal Errors: A Solution Based on Mixture Models”, Statistics and Computing, 21(4), 523–536.

    MathSciNet  Article  MATH  Google Scholar 

  76. STEINLEY, D. (2004), “Properties of the Hubert-Arabie Adjusted Rand Index”, Psychological Methods, 9(3), 386–396.

    Article  Google Scholar 

  77. SUBEDI, S., PUNZO, A., INGRASSIA, S., and MCNICHOLAS, P.D. (2013), “Clustering and Classification via Cluster-Weighted Factor Analyzers”, Advances in Data Analysis and Classification, 7(1), 5–40.

    MathSciNet  Article  MATH  Google Scholar 

  78. SUBEDI, S., PUNZO, A., INGRASSIA, S., and MCNICHOLAS, P.D. (2015), “Cluster-Weighted t-Factor Analyzers for Robust Model-Based Clustering and Dimension Reduction”, Statistical Methods and Applications, 24(4), 623–649.

    MathSciNet  Article  MATH  Google Scholar 

  79. TEICHER, H. (1963), “Identifiability of Finite Mixtures”, The Annals of Mathematical Statistics, 34(4), 1265–1269.

    MathSciNet  Article  MATH  Google Scholar 

  80. TITTERINGTON, D.M., SMITH, A.F.M., and MAKOV, U.E. (1985), Statistical Analysis of Finite Mixture Distributions, New York: Wiley.

    Google Scholar 

  81. VENABLES, W.N., and RIPLEY, B.D. (2002), Modern Applied Statistics with S (4th ed.), New York: Springer.

  82. VRBIK, I., and MCNICHOLAS, P.D. (2014), “Parsimonious Skew Mixture Models for Model-Based Clustering and Classification”, Computational Statistics and Data Analysis, 71, 196–210.

    MathSciNet  Article  Google Scholar 

  83. WEDEL, M. (2002), “Concomitant Variables in Finite Mixture Models”, Statistica Neerlandica, 56(3), 362–375.

    MathSciNet  Article  MATH  Google Scholar 

  84. WEI, Y., and MCNICHOLAS, P.D. (2015), “Mixture Model Averaging for Clustering”, Advances in Data Analysis and Classification, 9(2), 197–217.

    MathSciNet  Article  Google Scholar 

  85. YAKOWITZ, S.J., and SPRAGINS, J.D. (1968), “On the Identifiability of Finite Mixtures”, The Annals of Mathematical Statistics, 39(1), 209–214.

    MathSciNet  Article  MATH  Google Scholar 

  86. ZHOU, H., and LANGE, K.L. (2010), “On the Bumpy Road to the Dominant Mode”, Scandinavian Journal of Statistics, 37(4), 612–631.

    MathSciNet  Article  MATH  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Utkarsh J. Dang.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dang, U.J., Punzo, A., McNicholas, P.D. et al. Multivariate Response and Parsimony for Gaussian Cluster-Weighted Models. J Classif 34, 4–34 (2017). https://doi.org/10.1007/s00357-017-9221-2

Download citation

Keywords

  • Cluster-weighted model
  • EM algorithm
  • Multivariate response
  • Modelbased clustering
  • Mixture models
  • Parsimonious models
  • Eigen-decomposition
  • Regression