Journal of Classification

, Volume 34, Issue 1, pp 4–34 | Cite as

Multivariate Response and Parsimony for Gaussian Cluster-Weighted Models

  • Utkarsh J. DangEmail author
  • Antonio Punzo
  • Paul D. McNicholas
  • Salvatore Ingrassia
  • Ryan P. Browne


A family of parsimonious Gaussian cluster-weighted models is presented. This family concerns a multivariate extension to cluster-weighted modelling that can account for correlations between multivariate responses. Parsimony is attained by constraining parts of an eigen-decomposition imposed on the component covariance matrices. A sufficient condition for identifiability is provided and an expectation-maximization algorithm is presented for parameter estimation. Model performance is investigated on both synthetic and classical real data sets and compared with some popular approaches. Finally, accounting for linear dependencies in the presence of a linear regression structure is shown to offer better performance, vis-à-vis clustering, over existing methodologies.


Cluster-weighted model EM algorithm Multivariate response Modelbased clustering Mixture models Parsimonious models Eigen-decomposition Regression 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. AITKEN, A.C. (1926), “A Series Formula for the Roots of Algebraic and Transcendental Equations, in Proceedings of the Royal Society of Edinburgh, 45, 14–22.Google Scholar
  2. ANDERLUCCI, L., and VIROLI, C. (2015), “Covariance Pattern Mixture Models for Multivariate Longitudinal Data”, The Annals of Applied Statistics, 9(2), 777–800.MathSciNetCrossRefzbMATHGoogle Scholar
  3. ANDERSON, E. (1935), “The Irises of the Gaspé Peninsula”, Bulletin of the American Iris Society, 59, 2–5.Google Scholar
  4. ANDERSON, J.A. (1972), “Separate Sample Logistic Discrimination”, Biometrikam, 59(1), 19–35.MathSciNetCrossRefzbMATHGoogle Scholar
  5. ANDREWS, J.L., and MCNICHOLAS, P.D. (2012), “Model-Based Clustering, Classification, and Discriminant Analysis via Mixtures of Multivariate t-Distributions”, Statistics and Computing, 22(5), 1021–1029.MathSciNetCrossRefzbMATHGoogle Scholar
  6. ANDREWS, J.L., and MCNICHOLAS, P.D. (2014), “Variable Selection for Clustering and Classification”, Journal of Classification, 31(2), 136–153.MathSciNetCrossRefzbMATHGoogle Scholar
  7. ANDREWS, J.L., MCNICHOLAS, P.D., and SUBEDI, S. (2011), “Model-Based Classification via Mixtures of Multivariate t-Distributions”, Computational Statistics and Data Analysis, 55(1), 520–529.MathSciNetCrossRefzbMATHGoogle Scholar
  8. AZZALINI, A. (2013), sn: The Skew-Normal and Skew-t Distributions, R Package Version 0.4-18.Google Scholar
  9. BAUDRY, J.-P., and CELEUX, G. (2015), “EM for Mixtures”, Statistics and Computing, 25(4), 713–726.MathSciNetCrossRefzbMATHGoogle Scholar
  10. BIERNACKI, C., and LOURME, A. (2014), “Stable and Visualizable Gaussian Parsimonious Clustering Models”, Statistics and Computing, 24(6), 953–969.MathSciNetCrossRefzbMATHGoogle Scholar
  11. BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2000), “Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7), 719–725.CrossRefGoogle Scholar
  12. BÖHNING, D. (1995), “A Review of Reliable Algorithms for the Semi-Parametric Maximum Likelihood Estimator of a Mixture Distribution”, Journal of Statistical Planning and Inference, 47, 5–28.MathSciNetCrossRefzbMATHGoogle Scholar
  13. BÖHNING, D. (2000), Computer-Assisted Analysis of Mixtures and Applications: Metaanalysis, Disease Mapping and Others, London: Chapman & Hall/CRC.Google Scholar
  14. BÖHNING, D. (2003), “The EM Algorithm with Gradient Function Update for Discrete Mixtures with Known (Fixed) Number of Components”, Statistics and Computing, 13(3), 257–265.MathSciNetCrossRefGoogle Scholar
  15. BÖHNING, D., DIETZ, E., SCHAUB, R., SCHLATTMANN, P., and LINDSAY, B.G. (1994), “The Distribution of the Likelihood Ratio for Mixtures of Densities from the One-Parameter Exponential Family”, Annals of the Institute of Statistical Mathematics, 46(2), 373–388.CrossRefzbMATHGoogle Scholar
  16. BOUVEYRON, C., and BRUNET-SAUMARD, C. (2014), “Model-Based Clustering of High-Dimensional Data: A Review”, Computational Statistics and Data Analysis, 71, 52–78.MathSciNetCrossRefzbMATHGoogle Scholar
  17. BREIMAN, L., and FRIEDMAN, J.H. (1997), “Predicting Multivariate Responses in Multiple Linear Regression”, Journal of the Royal Statistical Society: Series B, 59(1), 3–54.MathSciNetCrossRefzbMATHGoogle Scholar
  18. BROWNE, R.P., and MCNICHOLAS, P.D. (2014a), “Estimating Common Principal Components in High Dimensions”, Advances in Data Analysis and Classification, 8(2), 217–226.Google Scholar
  19. BROWNE, R.P., and MCNICHOLAS, P.D. (2014b), “Orthogonal Stiefel Manifold Optimization for Eigen-Decomposed Covariance Parameter Estimation in Mixture Models”, Statistics and Computing, 24(2), 203–210.MathSciNetCrossRefzbMATHGoogle Scholar
  20. BROWNE, R.P., and MCNICHOLAS, P.D. (2015), mixture: Mixture Models for Clustering and Classification, R Package Version 1.1.Google Scholar
  21. BROWNE, R.P., SUBEDI, S., and MCNICHOLAS, P.D. (2013), “Constrained Optimization for a Subset of the Gaussian Parsimonious Clustering Models”, arXiv preprint arXiv:1306.5824.Google Scholar
  22. CAMPBELL, N.A., and MAHON, R.J. (1974), “A Multivariate Study of Variation in Two Species of Rock Crab of the Genus Leptograpsus”, Australian Journal of Zoology, 22(3), 417–425.CrossRefGoogle Scholar
  23. CELEUX, G., and DIEBOLT, J. (1988), “A Random Imputation Principle: The Stochastic EM Algorithm”, in Rapports de Recherche 901, INRIA.Google Scholar
  24. CELEUX, G., and GOVAERT, G. (1995), “Gaussian Parsimonious Clustering Models”, Pattern Recognition, 28(5), 781–793.CrossRefGoogle Scholar
  25. COOK, R.D., and WEISBERG, S. (1994), An Introduction to Regression Graphics, John Wiley & Sons.Google Scholar
  26. DANG, U.J., and MCNICHOLAS, P.D. (2015), “Families of Parsimonious Finite Mixtures of Regression Models”, in Advances in Statistical Models for Data Analysis, Studies in Classification, Data Analysis and Knowledge Organization, eds. I. Morlini, T. Minerva, and M. Vichi, Switzerland: Springer International Publishing, pp. 73–84.Google Scholar
  27. DANG, U.J., BROWNE, R.P., and MCNICHOLAS, P.D. (2015), “Mixtures of Multivariate Power Exponential Distributions”, Biometrics, 71(4), 1081–1089.MathSciNetCrossRefzbMATHGoogle Scholar
  28. DASGUPTA, A., and RAFTERY, A.E. (1998), “Detecting Features in Spatial Point Processes with Clutter via Model-Based Clustering”, Journal of the American Statistical Association, 93(441), 294–302.CrossRefzbMATHGoogle Scholar
  29. DEMPSTER, A., LAIRD, N., and RUBIN, D. (1977), “Maximum Likelihood from Incomplete Data via the EM Algorithm”, Journal of the Royal Statistical Society: Series B, 39(1), 1–38.MathSciNetzbMATHGoogle Scholar
  30. DESARBO, W.S., and CRON, W.L. (1988), “A Maximum Likelihood Methodology for Clusterwise Linear Regression”, Journal of Classification, 5(2), 249–282.MathSciNetCrossRefzbMATHGoogle Scholar
  31. FISHER, R.A. (1936), “The Use of Multiple Measurements in Taxonomic Problems”, Annals of Eugenics, 7(2), 179–188.CrossRefGoogle Scholar
  32. FLURY, B.N., and GAUTSCHI, W. (1986), “An Algorithm for Simultaneous Orthogonal Transformation of Several Positive Definite Matrices to Nearly Diagonal Form”, SIAM Journal on Scientific and Statistical Computing, 7(1), 169–184.MathSciNetCrossRefzbMATHGoogle Scholar
  33. FRALEY, C., and RAFTERY, A.E. (2002), “Model-Based Clustering, Discriminant Analysis, and Density Estimation”, Journal of the American Statistical Association, 97(458), 611–631.MathSciNetCrossRefzbMATHGoogle Scholar
  34. FRALEY, C., RAFTERY, A.E., MURPHY, T.B., and SCRUCCA, L. (2012), “Mclust Version 4 for R: Normal Mixture Modeling for Model-Based Clustering, Classification, and Density Estimation”, Technical Report 597, Department of Statistics, University of Washington, Seattle, WA.Google Scholar
  35. FRANCZAK, B.C., TORTORA, C., BROWNE, R.P., and MCNICHOLAS, P.D. (2015), “Unsupervised Learning viaMixtures of Skewed Distributions with Hypercube Contours” Pattern Recognition Letters, 58(1), 69–76.CrossRefGoogle Scholar
  36. GALIMBERTI, G., and SOFFRITTI,G. (2013), “A Multivariate Linear Regression Analysis Using Finite Mixtures of t Distributions”, Computational Statistics and Data Analysis, 71, 138–150.MathSciNetCrossRefzbMATHGoogle Scholar
  37. GALIMBERTI, G., SCARDOVI, E., and SOFFRITTI, G. (2015), “Using Mixtures in Seemingly Unrelated Linear Regression Models with Non-Normal Errors”, Statistics and Computing, 1–14.Google Scholar
  38. GERSHENFELD,N. (1997), “Nonlinear Inference and Cluster-Weighted Modeling”, Annals of the New York Academy of Sciences, 808(1), 18–24.CrossRefGoogle Scholar
  39. GRÜN, B., and LEISCH, F. (2008), “Flexmix Version 2: Finite Mixtures with Concomitant Variables and Varying and Constant Parameters”, R package version 2.3.13, Journal of Statistical Software, 28(4), 1–35.CrossRefGoogle Scholar
  40. HARTIGAN, J.A., and WONG, M.A. (1979), “A k-Means Clustering Algorithm”, Journal of the Royal Statistical Society: Series C, 28(1), 100–108.zbMATHGoogle Scholar
  41. HASTIE, T., and TIBSHIRANI, R. (1996), “Discriminant Analysis by Gaussian Mixtures”, Journal of the Royal Statistical Society: Series B, 58(1), 155–176.MathSciNetzbMATHGoogle Scholar
  42. HENNIG, C. (2000), “Identifiablity of Models for Clusterwise Linear Regression”, Journal of Classification, 17(2), 273–296.MathSciNetCrossRefzbMATHGoogle Scholar
  43. HENNIG, C., and LIAO, T.F. (2013), “How to Find an Appropriate Clustering for Mixed-Type Variables with Application to Socio-Economic Stratification (With Discussion)”, Journal of the Royal Statistical Society: Series C, 62(3), 309–369.MathSciNetCrossRefGoogle Scholar
  44. HUBERT, L., and ARABIE, P. (1985), “Comparing Partitions”, Journal of Classification, 2(1), 193–218.CrossRefzbMATHGoogle Scholar
  45. INGRASSIA, S. (2004), “A Likelihood-Based Constrained Algorithm for Multivariate Normal Mixture Models”, Statistical Methods and Applications, 13(2), 151–166.MathSciNetCrossRefGoogle Scholar
  46. INGRASSIA, S., and PUNZO, A. (2016), “Decision Boundaries for Mixtures of Regressions”, Journal of the Korean Statistical Society, DOI:  10.1016/j.jkss.2015.11.005.
  47. INGRASSIA, S., and ROCCI, R. (2007), “Constrained Monotone EM Algorithms for Finite Mixture of Multivariate Gaussians”, Computational Statistics and Data Analysis, 51(11), 5339–5351.MathSciNetCrossRefzbMATHGoogle Scholar
  48. INGRASSIA, S., MINOTTI, S.C., and VITTADINI, G. (2012), “Local Statistical Modeling via a Cluster-Weighted Approach with Elliptical Distributions”, Journal of Classification, 29(3), 363–401.MathSciNetCrossRefzbMATHGoogle Scholar
  49. INGRASSIA, S., MINOTTI, S.C., and PUNZO, A. (2014), “Model-Based Clustering via Linear Cluster-Weighted Models”, Computational Statistics and Data Analysis, 71, 159–182.MathSciNetCrossRefGoogle Scholar
  50. INGRASSIA, S., PUNZO, A., VITTADINI, G., and MINOTTI, S.C. (2015), “The Generalized Linear Mixed Cluster-Weighted Model”, Journal of Classification, 32(1), 85–113.MathSciNetCrossRefzbMATHGoogle Scholar
  51. KERIBIN, C. (1998), “Estimation Consistante de l’Ordre de Modèles deMélange”, Comptes Rendus de l’Académie des Sciences-Series I-Mathematics, 326(2), 243–248.Google Scholar
  52. KERIBIN, C. (2000), “Consistent Estimation of the Order of Mixture Models”, Sankhyā: The Indian Journal of Statistics, Series A, 62(1), 49–66.Google Scholar
  53. LEBRET, R., IOVLEFF, S., LANGROGNET, F., BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2015), “Rmixmod: The R Package of the Model-Based Unsupervised, Supervised, and Semi-Supervised Classification Mixmod Library”, Journal of Statistical Software, 67(6), 1–29.CrossRefGoogle Scholar
  54. LEE, S.X., and MCLACHLAN, G.J. (2013), “On Mixtures of Skew Normal and Skew t-Distributions”, Advances in Data Analysis and Classification, 7(3), 241–266.MathSciNetCrossRefzbMATHGoogle Scholar
  55. LEISCH, F. (2004), “FlexMix: A General Framework for Finite Mixture Models and Latent Class Regression in R”, Journal of Statistical Software, 11(8), 1–18.CrossRefGoogle Scholar
  56. LIN, T.-I., MCNICHOLAS, P.D., and HSIU, J.H. (2014), “Capturing Patterns via Parsimonious t Mixture Models”, Statistics and Probability Letters, 88, 80–87.MathSciNetCrossRefzbMATHGoogle Scholar
  57. LINDSAY, B.G. (1995), “Mixture Models: Theory, Geometry and Applications”, in NSFCBMS Regional Conference Series in Probability and Statistics, pp. 1–163.Google Scholar
  58. MACQUEEN, J. (1967), “Some Methods for Classification and Analysis of Multivariate Observations”, in Proceedings of the Fifth Berkeley Symposium onMathematical Statistics and Probability, Vol. 1, Oakland CA: University of California Press, pp. 281–297.Google Scholar
  59. MCLACHLAN, G.J., and BASFORD, K.E. (1988), Mixture Models: Inference and Applications to Clustering, New York: Marcel Dekker.Google Scholar
  60. MCLACHLAN, G.J., and PEEL, D. (2000), Finite Mixture Models, New York: John Wiley & Sons Inc.CrossRefzbMATHGoogle Scholar
  61. MCNICHOLAS, P.D. (2010), “Model-Based Classification Using Latent Gaussian Mixture Models”, Journal of Statistical Planning and Inference, 140(5), 1175–1181.MathSciNetCrossRefzbMATHGoogle Scholar
  62. MCNICHOLAS, P.D. (2016a), Mixture Model-Based Classification, Boca Raton FL: Chapman & Hall/CRC Press.Google Scholar
  63. MCNICHOLAS, P.D. (2016b),“Model-Based Clustering”, Journal of Classification 33, in press.Google Scholar
  64. MCNICHOLAS, P.D., MURPHY, T.B., MCDAID, A.F., and FROST, D. (2010), “Serial and Parallel Implementations of Model-Based Clustering via Parsimonious Gaussian Mixture Models”, Computational Statistics and Data Analysis, 54(3), 711–723.MathSciNetCrossRefzbMATHGoogle Scholar
  65. MURRAY, P.M., BROWNE, R.P., and MCNICHOLAS, P.D. (2014), “ Mixtures of Skew-t Factor Analyzers”, Computational Statistics and Data Analysis, 77, 326–335.Google Scholar
  66. O’HAGAN, A., MURPHY, T.B., GORMLEY, I.C., MCNICHOLAS, P.D., and KARLIS, D. (2016), “Clustering with the Multivariate Normal Inverse Gaussian Distribution”, Computational Statistics and Data Analysis, 93, 18–30.MathSciNetCrossRefGoogle Scholar
  67. PUNZO, A. (2014), “Flexible Mixture Modeling with the Polynomial Gaussian Cluster-Weighted Model”, Statistical Modelling, 14(3), 257–291.MathSciNetCrossRefGoogle Scholar
  68. PUNZO, A., and INGRASSIA, S. (2013), “On the Use of the Generalized Linear Exponential Cluster-Weighted Model to Assess Local Linear Independence in Bivariate Data”, QdS -Journal of Methodological and Applied Statistics, 15, 131–144.Google Scholar
  69. PUNZO, A., and INGRASSIA, S. (2015), “Clustering Bivariate Mixed-Type Data via the Cluster-Weighted Model”, Computational Statistics, 31(3), 989–1013.MathSciNetCrossRefzbMATHGoogle Scholar
  70. PUNZO, A., and MCNICHOLAS, P.D. (2016), “Parsimonious Mixtures of Multivariate Contaminated Normal Distributions”, Biometrical Journal, in press.Google Scholar
  71. PUNZO, A., BROWNE, R.P., and MCNICHOLAS, P.D. (2016), “Hypothesis Testing for MixtureModelling Selection”, Journal of Statistical Computation and Simulation, 86(14), 2797–2818.MathSciNetCrossRefGoogle Scholar
  72. R CORE TEAM(2015), R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.Google Scholar
  73. RAND,W.M. (1971). “Objective Criteria for the Evaluation of ClusteringMethods”, Journal of the American Statistical Association, 66(336), 846–850.CrossRefGoogle Scholar
  74. SCHWARZ, G. (1978), “Estimating the Dimension of a Model”, Annals of Statistics, 6(2), 461–464.MathSciNetCrossRefzbMATHGoogle Scholar
  75. SOFFRITTI, G., and GALIMBERTI, G. (2011), “Multivariate Linear Regression with Non-Normal Errors: A Solution Based on Mixture Models”, Statistics and Computing, 21(4), 523–536.MathSciNetCrossRefzbMATHGoogle Scholar
  76. STEINLEY, D. (2004), “Properties of the Hubert-Arabie Adjusted Rand Index”, Psychological Methods, 9(3), 386–396.CrossRefGoogle Scholar
  77. SUBEDI, S., PUNZO, A., INGRASSIA, S., and MCNICHOLAS, P.D. (2013), “Clustering and Classification via Cluster-Weighted Factor Analyzers”, Advances in Data Analysis and Classification, 7(1), 5–40.MathSciNetCrossRefzbMATHGoogle Scholar
  78. SUBEDI, S., PUNZO, A., INGRASSIA, S., and MCNICHOLAS, P.D. (2015), “Cluster-Weighted t-Factor Analyzers for Robust Model-Based Clustering and Dimension Reduction”, Statistical Methods and Applications, 24(4), 623–649.MathSciNetCrossRefzbMATHGoogle Scholar
  79. TEICHER, H. (1963), “Identifiability of Finite Mixtures”, The Annals of Mathematical Statistics, 34(4), 1265–1269.MathSciNetCrossRefzbMATHGoogle Scholar
  80. TITTERINGTON, D.M., SMITH, A.F.M., and MAKOV, U.E. (1985), Statistical Analysis of Finite Mixture Distributions, New York: Wiley.zbMATHGoogle Scholar
  81. VENABLES, W.N., and RIPLEY, B.D. (2002), Modern Applied Statistics with S (4th ed.), New York: Springer.Google Scholar
  82. VRBIK, I., and MCNICHOLAS, P.D. (2014), “Parsimonious Skew Mixture Models for Model-Based Clustering and Classification”, Computational Statistics and Data Analysis, 71, 196–210.MathSciNetCrossRefGoogle Scholar
  83. WEDEL, M. (2002), “Concomitant Variables in Finite Mixture Models”, Statistica Neerlandica, 56(3), 362–375.MathSciNetCrossRefzbMATHGoogle Scholar
  84. WEI, Y., and MCNICHOLAS, P.D. (2015), “Mixture Model Averaging for Clustering”, Advances in Data Analysis and Classification, 9(2), 197–217.MathSciNetCrossRefGoogle Scholar
  85. YAKOWITZ, S.J., and SPRAGINS, J.D. (1968), “On the Identifiability of Finite Mixtures”, The Annals of Mathematical Statistics, 39(1), 209–214.MathSciNetCrossRefzbMATHGoogle Scholar
  86. ZHOU, H., and LANGE, K.L. (2010), “On the Bumpy Road to the Dominant Mode”, Scandinavian Journal of Statistics, 37(4), 612–631.MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Classification Society of North America 2017

Authors and Affiliations

  • Utkarsh J. Dang
    • 1
    Email author
  • Antonio Punzo
    • 2
  • Paul D. McNicholas
    • 3
  • Salvatore Ingrassia
    • 2
  • Ryan P. Browne
    • 4
  1. 1.Department of Mathematical SciencesBinghamton University, State University of New YorkBinghamtonUSA
  2. 2.University of CataniaCataniaItaly
  3. 3.McMaster UniversityOntarioCanada
  4. 4.University of WaterlooOntarioCanada

Personalised recommendations