Deep Gaussian mixture models

Statistics and Computing volume 29, pages43–51(2019)Cite this article

Abstract

Deep learning is a hierarchical inference method formed by subsequent multiple layers of learning able to more efficiently describe complex relationships. In this work, deep Gaussian mixture models (DGMM) are introduced and discussed. A DGMM is a network of multiple layers of latent variables, where, at each layer, the variables follow a mixture of Gaussian distributions. Thus, the deep mixture model consists of a set of nested mixtures of linear models, which globally provide a nonlinear model able to describe the data in a very flexible way. In order to avoid overparameterized solutions, dimension reduction by factor models can be applied at each layer of the architecture, thus resulting in deep mixtures of factor analyzers.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3

References

  1. Baek, J., McLachlan, G., Flack, L.: Mixtures of factor analyzers with common factor loadings: applications to the clustering and visualization of high-dimensional data. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1298–1309 (2010)

    Article  Google Scholar 

  2. Baudry, J.-P., Raftery, A.E., Celeux, G., Lo, K., Gottardo, R.: Combining mixture components for clustering. J. Comput. Gr. Stat. 19(2), 332–353 (2010)

    MathSciNet  Article  Google Scholar 

  3. Celeux, G., Diebolt, J.: The SEM algorithm: a probabilistic teacher algorithm derived from the EM algorithm for the mixture problem. Comput. Stat. Q. 2(1), 73–82 (1985)

    Google Scholar 

  4. Forina, M., Armanino, C., Castino, M., Ubigli, M.: Multivariate data analysis as a discriminating method of the origin of wines. Vitis 25(3), 189–201 (1986)

    Google Scholar 

  5. Forina, M., Tiscornia, E.: Pattern-recognition methods in the prediction of Italian olive oil origin by their fatty-acid content. Anal. Chim. 72(3–4), 143–155 (1982)

    Google Scholar 

  6. Fraley, C., Raftery, A.: Model-based clustering, discriminant analysis and density estimation. J. Am. Stat. Assoc. 97, 611–631 (2002)

    MathSciNet  Article  MATH  Google Scholar 

  7. Hennig, C.: Methods for merging gaussian mixture components. Adv. Data Anal. Classif. 4(1), 3–34 (2010)

    MathSciNet  Article  MATH  Google Scholar 

  8. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  9. Li, J.: Clustering based on a multilayer mixture model. J. Comput. Gr. Stat. 14(3), 547–568 (2005)

    MathSciNet  Article  Google Scholar 

  10. Mardia, K.V., Kent, J.T., Bibby, J.M.: Multivariate Analysis. Academic Press, Oxford (1976)

    Google Scholar 

  11. McLachlan, G., Peel, D., Bean, R.: Modelling high-dimensional data by mixtures of factor analyzers. Comput. Stat. Data Anal. 41(3), 379–388 (2003)

    MathSciNet  Article  MATH  Google Scholar 

  12. McLachlan, G.J., Peel, D.: Finite Mixture Models. Wiley, Hoboken (2000)

    Google Scholar 

  13. Melnykov, V.: Merging mixture components for clustering through pairwise overlap. J. Comput. Gr. Stat. 25(1), 66–90 (2016)

    MathSciNet  Article  Google Scholar 

  14. Montanari, A., Viroli, C.: Heteroscedastic factor mixture analysis. Stat. Model. 10(4), 441–460 (2010)

    MathSciNet  Article  Google Scholar 

  15. Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  16. Scrucca, L., Fop, M., Murphy, T.B., Raftery, A.E.: mclust 5: Clustering, classification and density estimation using Gaussian finite mixture models. R Journal 8, 289–317 (2016)

    Google Scholar 

  17. Tang, Y., Hinton, G.E., Salakhutdinov, R.: Deep mixtures of factor analysers. In Langford, J., Pineau, J. (eds.) Proceedings of the 29th International Conference on Machine Learning (ICML-12), New York, NY, USA, pp. 505–512. ACM (2012)

  18. Taigman, Y., Yang, M., Ranzato, M.A., Wolf, L.: Deepface: Closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1701–1708 (2014)

  19. van den Oord, A., Schrauwen, B.: Factoring variations in natural images with deep gaussian mixture models. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 27, pp. 3518–3526. Curran Associates, Inc, Montreal, Quebec, Canada (2014)

  20. Viroli, C.: Dimensionally reduced model-based clustering through mixtures of factor mixture analyzers. J. Classif. 27(3), 363–388 (2010)

    MathSciNet  Article  MATH  Google Scholar 

  21. Wang, K., Ng, S.-K., McLachlan, G.J.: Multivariate skew t mixture models: applications to fluorescence-activated cell sorting data. In: Digital Image Computing: Techniques and Applications, 2009. DICTA’09., pp. 526–531. IEEE (2009)

  22. Wei, G.C., Tanner, M.A.: A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms. J. Am. Stat. Assoc. 85(411), 699–704 (1990)

    Article  Google Scholar 

Download references

Author information

Affiliations

  1. Department of Statistical Sciences, University of Bologna, Via Belle Arti 41, 40126, Bologna, Italy

    Cinzia Viroli

  2. Department of Mathematics, University of Queensland, St. Lucia, Brisbane, 4072, Australia

    Geoffrey J. McLachlan

Authors
  1. Cinzia Viroli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Geoffrey J. McLachlan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cinzia Viroli.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Viroli, C., McLachlan, G.J. Deep Gaussian mixture models. Stat Comput 29, 43–51 (2019). https://doi.org/10.1007/s11222-017-9793-z

Download citation

Keywords

  • Unsupervised classification
  • Mixtures of factor analyzers
  • Stochastic EM algorithm