Cluster-weighted models (CWMs) are a flexible family of mixture models for fitting the joint distribution of a random vector composed of a response variable and a set of covariates. CWMs act as a convex combination of the products of the marginal distribution of the covariates and the conditional distribution of the response given the covariates. In this paper, we introduce a broad family of CWMs in which the component conditional distributions are assumed to belong to the exponential family and the covariates are allowed to be of mixed-type. Under the assumption of Gaussian covariates, sufficient conditions for model identifiability are provided. Moreover, maximum likelihood parameter estimates are derived using the EM algorithm. Parameter recovery, classification assessment, and performance of some information criteria are investigated through a broad simulation design. An application to real data is finally presented, with the proposed model outperforming other well-established mixture-based approaches.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
AITKEN, A.C. (1926), “On Bernoulli’s Numerical Solution of Algebraic Equations”, in Proceedings of the Royal Society of Edinburgh, Vol. 46, pp. 289–305.
AKAIKE, H. (1973), “Information Theory and an Extension of Maximum Likelihood Principle”, in Second International Symposium on Information Theory, eds. B.N. Petrov and F. Csaki, Budapest: Akademiai Kiado, pp. 267–281.
BAGNATO, L., and PUNZO, A. (2013), “Finite Mixtures of Unimodal Beta and Gamma Densities and the k-bumps Algorithm”, Computational Statistics, 28(4), 1571–1597.
BAGNATO, L., GRESELIN, F., and PUNZO, A. (2014), “On the Spectral Decomposition in Normal Discriminant Analysis”, Communications in Statistics - Simulation and Computation, 43(6), 1471–1489.
BANFIELD, J.D., and RAFTERY, A.E. (1993), “Model-based Gaussian and non-Gaussian Clustering”, Biometrics, 49(3), 803–821.
BHATTACHARYYA, A. (1943), “On a Measure of Divergence Between Two Statistical Populations Defined by Their Probability Distributions”, Bulletin of the Calcutta Mathematical Society, 35(4), 99–109.
BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2000), “Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7), 719–725.
BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2003), “Choosing Starting Values for the EM Algorithm for Getting the Highest Likelihood in Multivariate Gaussian Mixture Models”, Computational Statistics and Data Analysis, 41(3-4), 561–575.
BOZDOGAN, H. (1987), “ Model Selection and Akaikes’s Information Criterion (AIC): The General Theory and Its Analytical Extensions”, Psychometrika, 52, 345–370.
BOZDOGAN, H. (1994), “Theory & Methodology of Time Series Analysis”, in Proceedings of the First US/Japan Conference on the Frontiers of Statistical Modeling: An Informational Approach (Vol. 1), Dordrecht: Kluwer Academic Publishers.
DEMPSTER, A.P., LAIRD, N.M., and RUBIN, D.B. (1977), “Maximum Likelihood From Incomplete Data Via the EM Algorithm”, Journal of the Royal Statistical Society. Series B (Methodological), 39(1), 1–38.
FOLLMANN, D.A., and LAMBERT, D. (1991), “Identifiability of Finite Mixtures of Logistic Regression Models”, Journal of Statistical Planning and Inference, 27(3), 375–381.
FONSECA, J.R.S. (2008), “The Application of Mixture Modeling and Information Criteria for Discovering Patterns of Coronary Heart Disease”, Journal of Applied Quantitative Methods, 3(4), 292–303.
FONSECA, J.R.S. (2010), “On the Performance of Information Criteria in Latent Segment Models”, World Academy of Science, Engineering and Technology, 63, 2010.
FONSECA, J.R.S., and CARDOSO, M.G.M.S. (2005), “Retail Clients Latent Segments”, in Progress in Artificial Intelligence, Berlin Heidelberg: Springer-Verlag, pp. 348–358.
FRÜHWIRTH-SCHNATTER, S. (2006), Finite Mixture and Markov Switching Models, New York: Springer.
GERSHENFELD, N. (1997), “Nonlinear Inference and Cluster-Weighted Modeling”, Annals of the New York Academy of Sciences, 808(1), 18–24.
GERSHENFELD, N. (1999), The Nature of Mathematical Modelling, Cambridge: Cambridge University Press.
GERSHENFELD, N., SCHÖNER, B., and METOIS, E. (1999), “Cluster-Weighted Modelling for Time-Series Analysis”, Nature, 397, 329–332.
GRESELIN, F., and PUNZO, A. (2013), “Closed Likelihood Ratio Testing Procedures to Assess Similarity of Covariance Matrices”, The American Statistician, 67(3), 117–128.
GRÜN, B., and LEISCH, F. (2008a), “Finite Mixtures of Generalized Linear Regression Models”, in Recent Advances in Linear Models and Related Areas - Essays in Honour of Helge Toutenburg Shalabh, ed. C. Heumann, Heidelberg: Springer Physica Verlag, pp. 205–230.
GRÜN, B., and LEISCH, F. (2008b), “ FlexMix Version 2: FiniteMixtures with Concomitant Variables and Varying and Constant Parameters”, Journal of Statistical Software, 28(4), 1–35.
HENNIG, C. (2000), “Identifiablity of Models for Clusterwise Linear Regression”, Journal of Classification, 17(2), 273–296.
HENNIG, C., and LIAO, T.F. (2013), “How to Find an Appropriate Clustering for Mixed Type Variables with Application to Socio-Economic Stratification”, Journal of the Royal Statistical Society: Series C (Applied Statistics), 62(3), 1–25.
HURVICH, C.M., and TSAI, C.L. (1989), “Regression and Time Series Model Selection in Small Samples”, Biometrika, 76(2), 297–307.
HWANG, H., MALHOTRA, N.K., KIM, Y., TOMIUK, M.A., and HONG, S. (2010), “A Comparative Study on Parameter Recovery of Three Approaches to Structural Equation Modeling”, Journal of Marketing Research, 47(4), 699–712.
INGRASSIA, S., MINOTTI, S.C., and VITTADINI, G. (2012), “Local Statistical Modeling Via the Cluster-Weighted Approach with Elliptical Distributions”, Journal of Classification, 29(3), 363–401.
INGRASSIA, S., MINOTTI, S.C., and PUNZO, A. (2014), “Model-Based Clustering Via Linear Cluster-Weighted Models”, Computational Statistics and Data Analysis, 71, 159–182.
KARLIS, D., and XEKALAKI, E. (2003), “Choosing Initial Values for the EM Algorithm for Finite Mixtures”, Computational Statistics and Data Analysis, 41(3–4), 577–590.
MAZZA, A., PUNZO, A., and INGRASSIA, S. (2013), flexCWM : Flexible Cluster-Weighted Modeling, available at http://cran.fhcrc.org/web/packages/flexCWM/index.html.
MCCULLAGH, P., and NELDER, J.A. (2000), Generalized Linear Models (2nd ed.), Boca Raton: Chapman and Hall.
MCLACHLAN, G.J. (1997), “On the EM Algorithm for Overdispersed Count Data”, Statistical Methods in Medical Research, 6(1), 76–98.
MCLACHLAN, G.J., and PEEL, D. (2000), Finite Mixture Models, New York: John Wiley and Sons.
MCNICHOLAS, P.D., MURPHY, T.B., MCDAID, A.F., and FROST, D. (2010), “Serial and Parallel Implementations of Model-Based Clustering Via Parsimonious Gaussian Mixture Models”, Computational Statistics and Data Analysis, 54(3), 711–723.
MCQUARRIE, A., SHUMWAY, R., and TSAI, C.L. (1997), “The Model Selection Criterion AICu”, Statistics and Probability Letters, 34(3), 285–292.
PUNZO, A. (2014), “Flexible Mixture Modeling with the Polynomial Gaussian Cluster-Weighted Model”, Statistical Modelling, 14(3), 257–291.
R CORE TEAM (2013), R: A Language and Environment for Statistical Computing, Vienna, Austria: R Foundation for Statistical Computing.
SCHÖNER, B. (2000), “Probabilistic Characterization and Synthesis of Complex Data Driven Systems”, Technical Report, Ph.D. Thesis, MIT, Cambridge.
SCHÖNER, B., and GERSHENFELD, N. (2001), “Cluster Weighted Modeling: Probabilistic Time Series Prediction, Characterization, and Synthesism”, in Nonlinear Dynamics and Statistics, ed. A. Mees, Boston: Birkhauser, pp. 365–385.
SCHWARZ, G. (1978), “Estimating the Dimension of a Model”, The Annals of Statistics, 6(2), 461–464.
SUBEDI, S., PUNZO, A., INGRASSIA, S., and MCNICHOLAS, P.D. (2013), “Clustering and Classification Via Cluster-Weighted Factor Analyzers”, Advances in Data Analysis and Classification, 7(1), 5–40.
TEICHER, H. (1963), “Identifiability of Finite Mixtures”, Annals of Mathematical Statistics, 34(4), 1265–1269.
TITTERINGTON, D.M., SMITH, A.F.M., and MAKOV, U.E. (1985), Statistical Analysis of Finite Mixture Distributions, New York: John Wiley and Sons.
TSANAS, A., and XIFARA, A. (2012), “Accurate Quantitative Estimation of Energy Performance of Residential Buildings Using Statistical Machine Learning Tools”, Energy and Buildings, 49, 560–567.
VERMUNT, J.K., and MAGIDSON, J. (2002), “Latent Class Cluster Analysis”, in Applied Latent Class Analysis, eds. J.A. Hagenaars and A.L. McCutcheon, Cambridge: Cambridge University Press, pp. 89–106.
WANG, P. (1994), “Mixed Regression Models for Discrete Data”, Technical Report, Ph.D. Thesis, University of British Columbia, Vancouver.
WANG, P., PUTERMAN, M.L., COCKBURN, M.L., and LE, N.D. (1996), “Mixed Poisson Regression Models with Covariate Dependent Rates”, Biometrics, 52(2), 381–400.
WEDEL, M. (2002), “Concomitant Variables in Finite Mixture Models”, Statistica Neerlandica, 56(3), 362–375.
WEDEL, M., and DE SARBO, W. (1995), “A Mixture Likelihood Approach for Generalized Linear Models”, Journal of Classification, 12(3), 21–55.
WEDEL, M., and KAMAKURA, W.A. (2001), Market Segmentation: Conceptual and Methodological Foundations (2nd ed.), Boston MA: Kluwer Academic Publishers.
The authors are grateful to the Editor and the reviewers for comments and suggestions which contributed to improve significantly the quality of the paper.
About this article
Cite this article
Ingrassia, S., Punzo, A., Vittadini, G. et al. The Generalized Linear Mixed Cluster-Weighted Model. J Classif 32, 85–113 (2015). https://doi.org/10.1007/s00357-015-9175-1
- Cluster-weighted models
- Model-based clustering
- Generalized linear models
- Mixed-type data