Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Multivariate Response and Parsimony for Gaussian Cluster-Weighted Models

Abstract

A family of parsimonious Gaussian cluster-weighted models is presented. This family concerns a multivariate extension to cluster-weighted modelling that can account for correlations between multivariate responses. Parsimony is attained by constraining parts of an eigen-decomposition imposed on the component covariance matrices. A sufficient condition for identifiability is provided and an expectation-maximization algorithm is presented for parameter estimation. Model performance is investigated on both synthetic and classical real data sets and compared with some popular approaches. Finally, accounting for linear dependencies in the presence of a linear regression structure is shown to offer better performance, vis-à-vis clustering, over existing methodologies.

This is a preview of subscription content, log in to check access.

References

  1. AITKEN, A.C. (1926), “A Series Formula for the Roots of Algebraic and Transcendental Equations, in Proceedings of the Royal Society of Edinburgh, 45, 14–22.

  2. ANDERLUCCI, L., and VIROLI, C. (2015), “Covariance Pattern Mixture Models for Multivariate Longitudinal Data”, The Annals of Applied Statistics, 9(2), 777–800.

  3. ANDERSON, E. (1935), “The Irises of the Gaspé Peninsula”, Bulletin of the American Iris Society, 59, 2–5.

  4. ANDERSON, J.A. (1972), “Separate Sample Logistic Discrimination”, Biometrikam, 59(1), 19–35.

  5. ANDREWS, J.L., and MCNICHOLAS, P.D. (2012), “Model-Based Clustering, Classification, and Discriminant Analysis via Mixtures of Multivariate t-Distributions”, Statistics and Computing, 22(5), 1021–1029.

  6. ANDREWS, J.L., and MCNICHOLAS, P.D. (2014), “Variable Selection for Clustering and Classification”, Journal of Classification, 31(2), 136–153.

  7. ANDREWS, J.L., MCNICHOLAS, P.D., and SUBEDI, S. (2011), “Model-Based Classification via Mixtures of Multivariate t-Distributions”, Computational Statistics and Data Analysis, 55(1), 520–529.

  8. AZZALINI, A. (2013), sn: The Skew-Normal and Skew-t Distributions, R Package Version 0.4-18.

  9. BAUDRY, J.-P., and CELEUX, G. (2015), “EM for Mixtures”, Statistics and Computing, 25(4), 713–726.

  10. BIERNACKI, C., and LOURME, A. (2014), “Stable and Visualizable Gaussian Parsimonious Clustering Models”, Statistics and Computing, 24(6), 953–969.

  11. BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2000), “Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7), 719–725.

  12. BÖHNING, D. (1995), “A Review of Reliable Algorithms for the Semi-Parametric Maximum Likelihood Estimator of a Mixture Distribution”, Journal of Statistical Planning and Inference, 47, 5–28.

  13. BÖHNING, D. (2000), Computer-Assisted Analysis of Mixtures and Applications: Metaanalysis, Disease Mapping and Others, London: Chapman & Hall/CRC.

  14. BÖHNING, D. (2003), “The EM Algorithm with Gradient Function Update for Discrete Mixtures with Known (Fixed) Number of Components”, Statistics and Computing, 13(3), 257–265.

  15. BÖHNING, D., DIETZ, E., SCHAUB, R., SCHLATTMANN, P., and LINDSAY, B.G. (1994), “The Distribution of the Likelihood Ratio for Mixtures of Densities from the One-Parameter Exponential Family”, Annals of the Institute of Statistical Mathematics, 46(2), 373–388.

  16. BOUVEYRON, C., and BRUNET-SAUMARD, C. (2014), “Model-Based Clustering of High-Dimensional Data: A Review”, Computational Statistics and Data Analysis, 71, 52–78.

  17. BREIMAN, L., and FRIEDMAN, J.H. (1997), “Predicting Multivariate Responses in Multiple Linear Regression”, Journal of the Royal Statistical Society: Series B, 59(1), 3–54.

  18. BROWNE, R.P., and MCNICHOLAS, P.D. (2014a), “Estimating Common Principal Components in High Dimensions”, Advances in Data Analysis and Classification, 8(2), 217–226.

  19. BROWNE, R.P., and MCNICHOLAS, P.D. (2014b), “Orthogonal Stiefel Manifold Optimization for Eigen-Decomposed Covariance Parameter Estimation in Mixture Models”, Statistics and Computing, 24(2), 203–210.

  20. BROWNE, R.P., and MCNICHOLAS, P.D. (2015), mixture: Mixture Models for Clustering and Classification, R Package Version 1.1.

  21. BROWNE, R.P., SUBEDI, S., and MCNICHOLAS, P.D. (2013), “Constrained Optimization for a Subset of the Gaussian Parsimonious Clustering Models”, arXiv preprint arXiv:1306.5824.

  22. CAMPBELL, N.A., and MAHON, R.J. (1974), “A Multivariate Study of Variation in Two Species of Rock Crab of the Genus Leptograpsus”, Australian Journal of Zoology, 22(3), 417–425.

  23. CELEUX, G., and DIEBOLT, J. (1988), “A Random Imputation Principle: The Stochastic EM Algorithm”, in Rapports de Recherche 901, INRIA.

  24. CELEUX, G., and GOVAERT, G. (1995), “Gaussian Parsimonious Clustering Models”, Pattern Recognition, 28(5), 781–793.

  25. COOK, R.D., and WEISBERG, S. (1994), An Introduction to Regression Graphics, John Wiley & Sons.

  26. DANG, U.J., and MCNICHOLAS, P.D. (2015), “Families of Parsimonious Finite Mixtures of Regression Models”, in Advances in Statistical Models for Data Analysis, Studies in Classification, Data Analysis and Knowledge Organization, eds. I. Morlini, T. Minerva, and M. Vichi, Switzerland: Springer International Publishing, pp. 73–84.

  27. DANG, U.J., BROWNE, R.P., and MCNICHOLAS, P.D. (2015), “Mixtures of Multivariate Power Exponential Distributions”, Biometrics, 71(4), 1081–1089.

  28. DASGUPTA, A., and RAFTERY, A.E. (1998), “Detecting Features in Spatial Point Processes with Clutter via Model-Based Clustering”, Journal of the American Statistical Association, 93(441), 294–302.

  29. DEMPSTER, A., LAIRD, N., and RUBIN, D. (1977), “Maximum Likelihood from Incomplete Data via the EM Algorithm”, Journal of the Royal Statistical Society: Series B, 39(1), 1–38.

  30. DESARBO, W.S., and CRON, W.L. (1988), “A Maximum Likelihood Methodology for Clusterwise Linear Regression”, Journal of Classification, 5(2), 249–282.

  31. FISHER, R.A. (1936), “The Use of Multiple Measurements in Taxonomic Problems”, Annals of Eugenics, 7(2), 179–188.

  32. FLURY, B.N., and GAUTSCHI, W. (1986), “An Algorithm for Simultaneous Orthogonal Transformation of Several Positive Definite Matrices to Nearly Diagonal Form”, SIAM Journal on Scientific and Statistical Computing, 7(1), 169–184.

  33. FRALEY, C., and RAFTERY, A.E. (2002), “Model-Based Clustering, Discriminant Analysis, and Density Estimation”, Journal of the American Statistical Association, 97(458), 611–631.

  34. FRALEY, C., RAFTERY, A.E., MURPHY, T.B., and SCRUCCA, L. (2012), “Mclust Version 4 for R: Normal Mixture Modeling for Model-Based Clustering, Classification, and Density Estimation”, Technical Report 597, Department of Statistics, University of Washington, Seattle, WA.

  35. FRANCZAK, B.C., TORTORA, C., BROWNE, R.P., and MCNICHOLAS, P.D. (2015), “Unsupervised Learning viaMixtures of Skewed Distributions with Hypercube Contours” Pattern Recognition Letters, 58(1), 69–76.

  36. GALIMBERTI, G., and SOFFRITTI,G. (2013), “A Multivariate Linear Regression Analysis Using Finite Mixtures of t Distributions”, Computational Statistics and Data Analysis, 71, 138–150.

  37. GALIMBERTI, G., SCARDOVI, E., and SOFFRITTI, G. (2015), “Using Mixtures in Seemingly Unrelated Linear Regression Models with Non-Normal Errors”, Statistics and Computing, 1–14.

  38. GERSHENFELD,N. (1997), “Nonlinear Inference and Cluster-Weighted Modeling”, Annals of the New York Academy of Sciences, 808(1), 18–24.

  39. GRÜN, B., and LEISCH, F. (2008), “Flexmix Version 2: Finite Mixtures with Concomitant Variables and Varying and Constant Parameters”, R package version 2.3.13, Journal of Statistical Software, 28(4), 1–35.

  40. HARTIGAN, J.A., and WONG, M.A. (1979), “A k-Means Clustering Algorithm”, Journal of the Royal Statistical Society: Series C, 28(1), 100–108.

  41. HASTIE, T., and TIBSHIRANI, R. (1996), “Discriminant Analysis by Gaussian Mixtures”, Journal of the Royal Statistical Society: Series B, 58(1), 155–176.

  42. HENNIG, C. (2000), “Identifiablity of Models for Clusterwise Linear Regression”, Journal of Classification, 17(2), 273–296.

  43. HENNIG, C., and LIAO, T.F. (2013), “How to Find an Appropriate Clustering for Mixed-Type Variables with Application to Socio-Economic Stratification (With Discussion)”, Journal of the Royal Statistical Society: Series C, 62(3), 309–369.

  44. HUBERT, L., and ARABIE, P. (1985), “Comparing Partitions”, Journal of Classification, 2(1), 193–218.

  45. INGRASSIA, S. (2004), “A Likelihood-Based Constrained Algorithm for Multivariate Normal Mixture Models”, Statistical Methods and Applications, 13(2), 151–166.

  46. INGRASSIA, S., and PUNZO, A. (2016), “Decision Boundaries for Mixtures of Regressions”, Journal of the Korean Statistical Society, DOI: 10.1016/j.jkss.2015.11.005.

  47. INGRASSIA, S., and ROCCI, R. (2007), “Constrained Monotone EM Algorithms for Finite Mixture of Multivariate Gaussians”, Computational Statistics and Data Analysis, 51(11), 5339–5351.

  48. INGRASSIA, S., MINOTTI, S.C., and VITTADINI, G. (2012), “Local Statistical Modeling via a Cluster-Weighted Approach with Elliptical Distributions”, Journal of Classification, 29(3), 363–401.

  49. INGRASSIA, S., MINOTTI, S.C., and PUNZO, A. (2014), “Model-Based Clustering via Linear Cluster-Weighted Models”, Computational Statistics and Data Analysis, 71, 159–182.

  50. INGRASSIA, S., PUNZO, A., VITTADINI, G., and MINOTTI, S.C. (2015), “The Generalized Linear Mixed Cluster-Weighted Model”, Journal of Classification, 32(1), 85–113.

  51. KERIBIN, C. (1998), “Estimation Consistante de l’Ordre de Modèles deMélange”, Comptes Rendus de l’Académie des Sciences-Series I-Mathematics, 326(2), 243–248.

  52. KERIBIN, C. (2000), “Consistent Estimation of the Order of Mixture Models”, Sankhyā: The Indian Journal of Statistics, Series A, 62(1), 49–66.

  53. LEBRET, R., IOVLEFF, S., LANGROGNET, F., BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2015), “Rmixmod: The R Package of the Model-Based Unsupervised, Supervised, and Semi-Supervised Classification Mixmod Library”, Journal of Statistical Software, 67(6), 1–29.

  54. LEE, S.X., and MCLACHLAN, G.J. (2013), “On Mixtures of Skew Normal and Skew t-Distributions”, Advances in Data Analysis and Classification, 7(3), 241–266.

  55. LEISCH, F. (2004), “FlexMix: A General Framework for Finite Mixture Models and Latent Class Regression in R”, Journal of Statistical Software, 11(8), 1–18.

  56. LIN, T.-I., MCNICHOLAS, P.D., and HSIU, J.H. (2014), “Capturing Patterns via Parsimonious t Mixture Models”, Statistics and Probability Letters, 88, 80–87.

  57. LINDSAY, B.G. (1995), “Mixture Models: Theory, Geometry and Applications”, in NSFCBMS Regional Conference Series in Probability and Statistics, pp. 1–163.

  58. MACQUEEN, J. (1967), “Some Methods for Classification and Analysis of Multivariate Observations”, in Proceedings of the Fifth Berkeley Symposium onMathematical Statistics and Probability, Vol. 1, Oakland CA: University of California Press, pp. 281–297.

  59. MCLACHLAN, G.J., and BASFORD, K.E. (1988), Mixture Models: Inference and Applications to Clustering, New York: Marcel Dekker.

  60. MCLACHLAN, G.J., and PEEL, D. (2000), Finite Mixture Models, New York: John Wiley & Sons Inc.

  61. MCNICHOLAS, P.D. (2010), “Model-Based Classification Using Latent Gaussian Mixture Models”, Journal of Statistical Planning and Inference, 140(5), 1175–1181.

  62. MCNICHOLAS, P.D. (2016a), Mixture Model-Based Classification, Boca Raton FL: Chapman & Hall/CRC Press.

  63. MCNICHOLAS, P.D. (2016b),“Model-Based Clustering”, Journal of Classification 33, in press.

  64. MCNICHOLAS, P.D., MURPHY, T.B., MCDAID, A.F., and FROST, D. (2010), “Serial and Parallel Implementations of Model-Based Clustering via Parsimonious Gaussian Mixture Models”, Computational Statistics and Data Analysis, 54(3), 711–723.

  65. MURRAY, P.M., BROWNE, R.P., and MCNICHOLAS, P.D. (2014), “ Mixtures of Skew-t Factor Analyzers”, Computational Statistics and Data Analysis, 77, 326–335.

  66. O’HAGAN, A., MURPHY, T.B., GORMLEY, I.C., MCNICHOLAS, P.D., and KARLIS, D. (2016), “Clustering with the Multivariate Normal Inverse Gaussian Distribution”, Computational Statistics and Data Analysis, 93, 18–30.

  67. PUNZO, A. (2014), “Flexible Mixture Modeling with the Polynomial Gaussian Cluster-Weighted Model”, Statistical Modelling, 14(3), 257–291.

  68. PUNZO, A., and INGRASSIA, S. (2013), “On the Use of the Generalized Linear Exponential Cluster-Weighted Model to Assess Local Linear Independence in Bivariate Data”, QdS -Journal of Methodological and Applied Statistics, 15, 131–144.

  69. PUNZO, A., and INGRASSIA, S. (2015), “Clustering Bivariate Mixed-Type Data via the Cluster-Weighted Model”, Computational Statistics, 31(3), 989–1013.

  70. PUNZO, A., and MCNICHOLAS, P.D. (2016), “Parsimonious Mixtures of Multivariate Contaminated Normal Distributions”, Biometrical Journal, in press.

  71. PUNZO, A., BROWNE, R.P., and MCNICHOLAS, P.D. (2016), “Hypothesis Testing for MixtureModelling Selection”, Journal of Statistical Computation and Simulation, 86(14), 2797–2818.

  72. R CORE TEAM(2015), R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.

  73. RAND,W.M. (1971). “Objective Criteria for the Evaluation of ClusteringMethods”, Journal of the American Statistical Association, 66(336), 846–850.

  74. SCHWARZ, G. (1978), “Estimating the Dimension of a Model”, Annals of Statistics, 6(2), 461–464.

  75. SOFFRITTI, G., and GALIMBERTI, G. (2011), “Multivariate Linear Regression with Non-Normal Errors: A Solution Based on Mixture Models”, Statistics and Computing, 21(4), 523–536.

  76. STEINLEY, D. (2004), “Properties of the Hubert-Arabie Adjusted Rand Index”, Psychological Methods, 9(3), 386–396.

  77. SUBEDI, S., PUNZO, A., INGRASSIA, S., and MCNICHOLAS, P.D. (2013), “Clustering and Classification via Cluster-Weighted Factor Analyzers”, Advances in Data Analysis and Classification, 7(1), 5–40.

  78. SUBEDI, S., PUNZO, A., INGRASSIA, S., and MCNICHOLAS, P.D. (2015), “Cluster-Weighted t-Factor Analyzers for Robust Model-Based Clustering and Dimension Reduction”, Statistical Methods and Applications, 24(4), 623–649.

  79. TEICHER, H. (1963), “Identifiability of Finite Mixtures”, The Annals of Mathematical Statistics, 34(4), 1265–1269.

  80. TITTERINGTON, D.M., SMITH, A.F.M., and MAKOV, U.E. (1985), Statistical Analysis of Finite Mixture Distributions, New York: Wiley.

  81. VENABLES, W.N., and RIPLEY, B.D. (2002), Modern Applied Statistics with S (4th ed.), New York: Springer.

  82. VRBIK, I., and MCNICHOLAS, P.D. (2014), “Parsimonious Skew Mixture Models for Model-Based Clustering and Classification”, Computational Statistics and Data Analysis, 71, 196–210.

  83. WEDEL, M. (2002), “Concomitant Variables in Finite Mixture Models”, Statistica Neerlandica, 56(3), 362–375.

  84. WEI, Y., and MCNICHOLAS, P.D. (2015), “Mixture Model Averaging for Clustering”, Advances in Data Analysis and Classification, 9(2), 197–217.

  85. YAKOWITZ, S.J., and SPRAGINS, J.D. (1968), “On the Identifiability of Finite Mixtures”, The Annals of Mathematical Statistics, 39(1), 209–214.

  86. ZHOU, H., and LANGE, K.L. (2010), “On the Bumpy Road to the Dominant Mode”, Scandinavian Journal of Statistics, 37(4), 612–631.

Download references

Author information

Correspondence to Utkarsh J. Dang.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dang, U.J., Punzo, A., McNicholas, P.D. et al. Multivariate Response and Parsimony for Gaussian Cluster-Weighted Models. J Classif 34, 4–34 (2017). https://doi.org/10.1007/s00357-017-9221-2

Download citation

Keywords

  • Cluster-weighted model
  • EM algorithm
  • Multivariate response
  • Modelbased clustering
  • Mixture models
  • Parsimonious models
  • Eigen-decomposition
  • Regression