Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Clustering bivariate mixed-type data via the cluster-weighted model

  • 432 Accesses

  • 17 Citations

Abstract

The cluster-weighted model (CWM) is a mixture model with random covariates that allows for flexible clustering/classification and distribution estimation of a random vector composed of a response variable and a set of covariates. Within this class of models, the generalized linear exponential CWM is here introduced especially for modeling bivariate data of mixed-type. Its natural counterpart in the family of latent class models is also defined. Maximum likelihood parameter estimates are derived using the expectation-maximization algorithm and some computational issues are detailed. Through Monte Carlo experiments, the classification performance of the proposed model is compared with other mixture-based approaches, consistency of the estimators of the regression coefficients is evaluated, and several likelihood-based information criteria are compared for selecting the number of mixture components. An application to real data is also finally considered.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

References

  1. Akaike H (1973) Information theory and an extension of maximum likelihood principle. In: Petrov BN, Csaki F (eds) Second International Symposium on Information Theory. Akademiai Kiado, Budapest, pp 267–281

  2. Bagnato L, Punzo A (2013) Finite mixtures of unimodal beta and gamma densities and the \(k\)-bumps algorithm. Comput Stat 28(4):1571–1597

  3. Balakrishnan N, Lai C-D (2009) Continuous bivariate distributions. Springer, New York

  4. Banfield JD, Raftery AE (1993) Model-based Gaussian and non-Gaussian clustering. Biometrics 49(3):803–821

  5. Bermúdez L, Karlis D (2012) A finite mixture of bivariate Poisson regression models with an application to insurance ratemaking. Comput Stat Data Anal 56(12):3988–3999

  6. Biernacki C, Celeux G, Govaert G (2000) Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Trans Pattern Anal Mach Intell 22(7):719–725

  7. Biernacki C, Celeux G, Govaert G (2003) Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models. Comput Stat Data Anal 41(3–4):561–575

  8. Böhning D, Dietz E, Schaub R, Schlattmann P, Lindsay BG (1994) The distribution of the likelihood ratio for mixtures of densities from the one-parameter exponential family. Ann Inst Stat Math 46(2):373–388

  9. Bozdogan H (1994) Theory and methodology of time series analysis. In: Proceedings of the First US/Japan Conference on the Frontiers of Statistical Modeling: An Informational Approach, vol 1. Kluwer Academic Publishers, Dordrecht

  10. Bozdogan H (1987) Model selection and Akaike’s information criterion (AIC): the general theory and its analytical extensions. Psychometrika 52(3):345–370

  11. Browne RP, McNicholas PD (2012) Model-based clustering, classification, and discriminant analysis of data with mixed type. J Stat Plan Inference 142(11):2976–2984

  12. Celeux G, Hurn M, Robert CP (2000) Computational and inferential difficulties with mixture posterior distributions. J Am Stat Assoc 95(451):957–970

  13. Dempster A, Laird N, Rubin D (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Series B Methodol 39(1):1–38

  14. Escobar M, West M (1995) Bayesian density estimation and inference using mixtures. J Am Stat Assoc 90(430):577–588

  15. Fonseca JRS, Cardoso MGMS (2005) Retail clients latent segments. In: Progress in Artificial Intelligence. Springer, Berlin, pp 348–358

  16. Fonseca JRS (2008) The application of mixture modeling and information criteria for discovering patterns of coronary heart disease. J Appl Quant Methods 3(4):292–303

  17. Fonseca JRS (2010) On the performance of information criteria in latent segment models. World Acad Sci Eng Technol 63:2010

  18. Fraley C, Raftery AE, Murphy TB, Scrucca L (2012) mclust version 4 for R: normal mixture modeling for model-based clustering, classification, and density estimation. Technical report 597, Department of Statistics, University of Washington, Seattle, Washington, USA

  19. Frühwirth-Schnatter S (2006) Finite mixture and Markov switching models. Springer, New York

  20. Genest C, Neslehova J (2007) A primer on copulas for count data. Astin Bull 37(2):475–515

  21. Gershenfeld N (1997) Nonlinear inference and cluster-weighted modeling. Ann New York Acad Sci 808(1):18–24

  22. Grün B, Leisch F (2008) FlexMix version 2: finite mixtures with concomitant variables and varying and constant parameters. J Stat Softw 28(4):1–35

  23. Hennig C (2000) Identifiablity of models for clusterwise linear regression. J Classif 17(2):273–296

  24. Hennig C, Liao TF (2013) How to find an appropriate clustering for mixed type variables with application to socio-economic stratification. J R Stat Soc Series C Appl Stat 62(3):1–25

  25. Henning G (1989) Meanings and implications of the principle of local independence. Lang Test 6(1):95–108

  26. Hunt LA, Basford KE (1999) Fitting a mixture model to three-mode three-way data with categorical and continuous variables. J Classif 16(2):283–296

  27. Hunt LA, Jorgensen M (2011) Clustering mixed data. Wiley Interdiscip Rev Data Min Knowl Discov 1(4):352–361

  28. Hurvich CM, Tsai CL (1989) Regression and time series model selection in small samples. Biometrika 76(2):297–307

  29. Ingrassia S, Minotti SC, Vittadini G (2012) Local statistical modeling via the cluster-weighted approach with elliptical distributions. J Classif 29(3):363–401

  30. Ingrassia S, Minotti SC, Punzo A (2014) Model-based clustering via linear cluster-weighted models. Comput Stat Data Anal 71:159–182

  31. Ingrassia S, Punzo A, Vittadini G, Minotti SC (2015) The generalized linear mixed cluster-weighted model. J Classif 32(1):85–113

  32. Joe H (2005) Asymptotic efficiency of the two-stage estimation method for copula-based models. J Multivar Anal 94(2):401–419

  33. Jorgensen M, Hunt LA (1996) Mixture model clustering of data sets with categorical and continuous variables. In: Dowe DL, Korb KB, Oliver JJ (eds) Proceedings of the Conference: Information, Statistics and Induction in Science, Melbourne, Australia, 20–23 August, vol 96. River Edge, New Jersey, pp 375–384

  34. Karlis D, Xekalaki E (2003) Choosing initial values for the EM algorithm for finite mixtures. Computational Statistics & Data Analysis 41(3–4):577–590

  35. Kocherlakota S, Kocherlakota K (1992) Bivariate discrete distributions, volume 132 of statistics: a series of textbooks and monographs. Taylor & Francis, Cambridge

  36. Leisch F (2004) FlexMix: a general framework for finite mixture models and latent class regression in \({\sf R}\). J Stat Softw 11(8):1–18

  37. Lichman M (2013) UCI Machine Learning Repository, University of California, School of Information and Computer Science. Irvine, CA. http://archive.ics.uci.edu/ml

  38. Mazza A, Punzo A, Ingrassia S (2015) flexCWM: flexible cluster-weighted modeling. http://cran.r-project.org/web/packages/flexCWM/index.html

  39. McCullagh P, Nelder J (1989) Generalized linear models, 2nd edn. Chapman & Hall, Boca Raton

  40. McLachlan GJ, Peel D (2000) Finite mixture models. In: Applied probability and statistics: Wiley Series in Probability and Statistics. John Wiley & Sons, New York

  41. McLachlan GJ, Basford KE (1988) Mixture models: inference and applications to clustering, volume 84 of statistics series. Marcel Dekker, New York

  42. McQuarrie A, Shumway R, Tsai C-L (1997) The model selection criterion AICu. Stat Probab Lett 34(3):285–292

  43. Nelsen RB (2007) An introduction to copulas. Springer Series in Statistics. Springer, New York

  44. Punzo A (2014) Flexible mixture modeling with the polynomial Gaussian cluster-weighted model. Stat Modelling 14(3):257–291

  45. Punzo A, Ingrassia S (2015) Parsimonious generalized linear Gaussian cluster-weighted models. In: Morlini I, Minerva T, Vichi M (eds) Advances in Statistical Models for Data Analysis, Studies in Classification, Data Analysis and Knowledge Organization, Switzerland. Springer International Publishing, Forthcoming

  46. Punzo A, Ingrassia S (2013) On the use of the generalized linear exponential cluster-weighted model to asses local linear independence in bivariate data. QdS J Methodol Appl Stat 15:131–144

  47. Punzo A, McNicholas PD (2014) Robust clustering in regression analysis via the contaminated Gaussian cluster-weighted model. arXiv.org e-print arXiv.org e-print arXiv:1409.6019 available at: arXiv:1409.6019

  48. R Core Team (2013) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna

  49. Schlattmann P (2009) Medical applications of finite mixture models. Statistics for biology and health. Springer, Berlin

  50. Schwarz G (1978) Estimating the dimension of a model. Ann Stat 6(2):461–464

  51. Sklar M (1959) Fonctions de répartition à n dimensions et leurs marges. Publications de l’Institut de Statistique de l’Université de Paris 8:229–231

  52. Stephens M (2000) Dealing with label switching in mixture models. J R Stat Soc Series B Stat Methodol 62(4):795–809

  53. Subedi S, Punzo A, Ingrassia S, McNicholas PD (2013) Clustering and classification via cluster-weighted factor analyzers. Adv Data Anal Classif 7(1):5–40

  54. Subedi S, Punzo A, Ingrassia S, McNicholas PD (2015) Cluster-weighted \(t\)-factor analyzers for robust model-based clustering and dimension reduction. Stat Methods Appl 24 (in press)

  55. Titterington DM, Smith AFM, Makov UE (1985) Statistical analysis of finite mixture distributions. John Wiley & Sons, New York

  56. Tsanas A, Xifara A (2012) Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy Build 49:560–567

  57. Vermunt JK, Magidson J (2002) Latent class cluster analysis. In: Hagenaars JA, McCutcheon AL (eds) Applied latent class analysis. Cambridge University Press, Cambridge, pp 89–106

  58. Wedel M, DeSarbo WS (1995) A mixture likelihood approach for generalized linear models. J Classif 12(1):21–55

  59. Wedel M, Kamakura W (2000) Market segmentation: conceptual and methodological foundations, 2nd edn. Kluwer Academic Publishers, Boston

  60. Yao W (2012) Model based labeling for mixture models. Stat Comput 22(2):337–347

  61. Yao W, Wei Y, Yu C (2014) Robust mixture regression using the \(t\)-distribution. Comput Stat Data Anal 71:116–127

Download references

Acknowledgments

The authors acknowledge the financial support from the Grant “Finite mixture and latent variable models for causal inference and analysis of socio-economic data” (FIRB 2012-Futuro in ricerca) funded by the Italian Government (RBFR12SHVV).

Author information

Correspondence to Antonio Punzo.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Punzo, A., Ingrassia, S. Clustering bivariate mixed-type data via the cluster-weighted model. Comput Stat 31, 989–1013 (2016). https://doi.org/10.1007/s00180-015-0600-z

Download citation

Keywords

  • Mixture models with random covariates
  • Model-based clustering
  • Cluster-weighted models
  • Generalized linear models
  • Mixed-type data