Skip to main content
SpringerLink
Log in

Estimating number of components in Gaussian mixture model using combination of greedy and merging algorithm

Pattern Analysis and Applications volume 21, pages 181–192 (2018)Cite this article

Abstract

The brain must deal with a massive flow of sensory information without receiving any prior information. Therefore, when creating cognitive models, it is important to acquire as much information as possible from the data itself. Moreover, the brain has to deal with an unknown number of components (concepts) contained in a dataset without any prior knowledge. Most of the algorithms in use today are not able to effectively copy this strategy. We propose a novel approach based on neural modelling fields theory (NMF) to overcome this problem. The algorithm combines NMF and greedy Gaussian mixture models. The novelty lies in the combination of information criterion with the merging algorithm. The performance of the algorithm was compared with other well-known algorithms and tested both on artificial and real-world datasets.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Bar M (2003) A cortical mechanism for triggering top-down facilitation in visual object recognition. J Cogn Neurosci 15(4):600–609

    Article  Google Scholar 

  2. Schacter DL, Dobbins IG, Schnyer DM (2004) Specificity of priming: a cognitive neuroscience perspective. Nat Rev Neurosci 5(11):853–862

    Article  Google Scholar 

  3. Schacter DL, Addis DR (2007) The cognitive neuroscience of constructive memory: remembering the past and imagining the future. Philos Trans R Soc B Biol Sci 362(1481):773–786

    Article  Google Scholar 

  4. Perlovsky LI (2001) Neural neworks and intellect: using model-based concepts. Oxford University Press, New York

    Google Scholar 

  5. Perlovsky LI, Deming R, Illin R (2011) Emotional cognitive neural algorithms with engineering applications. Studies in computational intelligence. Springer, Berlin

    Book  Google Scholar 

  6. Fraley Ch, Raftery AE (1998) How many clusters? Which clustering method? Answers via model-based cluster analysis. Comput J 41:578–588

    Article  MATH  Google Scholar 

  7. Perlovsky LI (2009) Vague-to-crisp? Neural mechanism of perception. IEEE Trans Neural Netw 20:1363–1367

    Article  Google Scholar 

  8. Perlovsky LI (2007) Neural networks, fuzzy models and dynamic logic. Asp Autom Text Anal 209(2007):363–386

    Google Scholar 

  9. Perlovsky LI, McManus MM (1991) Maximum likelihood neural networks for sensor fusion and adaptive classification. Neural Netw 4(1):89–102

    Article  Google Scholar 

  10. Perlovsky LI (2005) Neural network with fuzzy dynamic logic. In: Proceedings of international joint conference on neural network, pp 3046–3051

  11. Deming R, Perlovsky LI (2008) Multi-target/multi-sensor tracking from optical data using modelling field theory. In: World congress on computational intelligence (WCCI). Hong Kong, China

  12. Cangelosi A, Tikhanoff V, Fontanari JF (2007) Integrating language and cognition: a cognitive robotics approach. Comput Intell Mag 2(3):65–70

    Article  Google Scholar 

  13. Figueiredo MAT, Jain AK (2002) Unsupervised learning of finite mixture models. IEEE Trans Pattern Anal Mach Intell 24(3):381–396

    Article  Google Scholar 

  14. Ueda N, Nakano R, Ghahramani Z, Hinton GE (1998) Split and merge EM algorithm for improving Gaussian mixture density estimates. In: Proceedings of IEEE workshop neural networks for signal processing, pp 274–283

  15. Li Y, Li L (2009) A novel split and merge EM algorithm for Gaussian mixture model. In: ICNC ’09. Fifth international conference on natural computation, pp 479–483

  16. Verbeek J, Vlassis M, Kröse B (2003) Efficient greedy learning for Gaussian mixture models. Neural Comput 5(2):469–485

    Article  MATH  Google Scholar 

  17. Vlassis N, Likas A (2002) A greedy EM algorithm for Gaussian mixture learning. Neural Process Lett 15:77–87

    Article  MATH  Google Scholar 

  18. Akaike H (1974) A new look at the statistical model identification. IEEE Trans Autom Control 19(6):716

    Article  MathSciNet  MATH  Google Scholar 

  19. Schwarz GE (1978) Estimating the dimension of a model. Ann Stat 6(2):461–464

    Article  MathSciNet  MATH  Google Scholar 

  20. Bouchard G, Celeux G (2006) Selection of generative models in classification. IEEE Trans Pattern Anal Mach Intell 28(4):544–554

    Article  Google Scholar 

  21. Celeux G, Soromenho G (1994) An entropy criterion for assessing the number of clusters in a mixture models. J Classif 13:195–212

    Article  MathSciNet  MATH  Google Scholar 

  22. Tenmoto H, Kudo M, Shimbo M (1998) MDL-based selection of the number of components in mixture models for pattern classification. Adv Pattern Recognit 1451(1998):831–836

    Article  Google Scholar 

  23. Lee Y, Lee KY, Lee J (2006) The estimating optimal number of GMM based on incremental k-means for speaker identifications. Int J Inf Technol 12(7):1119–1128

    Google Scholar 

  24. Likas A, Vlassis N, Verbeek JJ (2003) The global k-means clustering algorithm. Pattern Recognit 36:451–461

    Article  Google Scholar 

  25. Grim J, Novovicova J, Pudil P, Somol P, Ferri F (1987) Initialization normal mixutres of densities. Appl Stat 36(3):318–324

    Article  Google Scholar 

  26. Smyth P (2000) Model selection for probabilistic clustering using crossvalidated likelihood. Stat Comput 9:63

    Article  Google Scholar 

  27. McLachlan G (1987) On bootstraping the likelihood ratio test statistic for the number of components in a normal mixture. Appl Stat 36(3):318–324

    Article  Google Scholar 

  28. Pernkopf F, Bouchaffra D (2005) Genetic-based EM algorithm for learning gaussian mixture models. IEEE Trans Pattern Anal Mach Intell 27(8):1344–1348

    Article  Google Scholar 

  29. Ververidis D, Kotropoulos C (2008) Gaussian mixture modeling by exploiting the Mahalanobis distance. IEEE Trans Signal Process 56(7B):2797–2811

    Article  MathSciNet  MATH  Google Scholar 

  30. Bozdogan H, Sclove SL (1984) Multi-sample cluster analysis using Akaike’s information criterion. Ann Inst Stat Math 36:163–180

    Article  MATH  Google Scholar 

  31. Roberts S, Husmaier D, Rezek I, Penny W (1998) Bayesian approaches to Gaussian mixture modelling. IEEE Trans Pattern Anal Mach Intell 20(11):1133–1142

    Article  Google Scholar 

  32. Banfield J, Raftery A (1993) Model-based Gaussian and non-Gaussian clustering. Biometrics 49:803–821

    Article  MathSciNet  MATH  Google Scholar 

  33. Bozdogan H (1993) Choosing the number of component clusters in the mixture model using a new informational complexity criterion of the inverse Fisher information matrix. In: Opitz O, Lausen B, Klar R (eds) Information and Classification, Springer-Verlag, Berlin, pp 40–54

    Chapter  Google Scholar 

  34. Whindham M, Cutler A (1992) Information ratios for validating mixture analysis. J Am Stat Assoc 87:1188–1192

    Article  Google Scholar 

  35. Biernacki C, Celeux G (1999) An improvement of the NEC criterion for assessing the number of clusters in a mixture model. Pattern Recognit Lett 20(3):267–272

    Article  MATH  Google Scholar 

  36. Oliver JJ, Baxter RA, Wallace CS (1996) Unsupervised Learning using MML. In: Proceedings of the Thirteenth International Conference (ICML96), Morgan Kaufmann Publishers, SanFrancisco, CA, pp 364–372

    Google Scholar 

  37. Rissanen J (1989) Stochastic complexity in statisticalinquiry. World Scientific, Singapore

    MATH  Google Scholar 

  38. Biernacki C, Govaert G (1997) Using the classification likelihood to choose the number of clusters. Comput sci stat 29(2):451–457

    Google Scholar 

  39. Yang ZR, Zwolinski M (2001) A mutual information theory for adaptive mixture models. IEEE Trans Pattern Anal Mach Intell 23(4):1–8

    Google Scholar 

  40. Smyth MHC, Figueiredo MAT, Jain AK (2004) Simultaneous feature selection and clustering using mixture models. IEEE Trans Pattern Anal Mach Intell 26(9):1154–1166

    Article  Google Scholar 

  41. Franti P, Virmajoki O (2006) Iterative shrinking method for clustering problems. Pattern Recognit 39(5):761–765

    Article  MATH  Google Scholar 

  42. Frank A, Asuncion A (2010) UCI machine learning repository. http://archive.ics.uci.edu/ml. University of California, School of Information and Computer Science, Irvine

  43. Li J, Tao D (2013) Simple exponential family PCA. IEEE Trans Neural Netw Learn Syst 24(3):485–497

    Article  Google Scholar 

  44. Rasmussen CE (1999) The infinite Gaussian mixture model. In: Advances in Neural Information Processing Systems 12, MIT Press, Cambridge, MA, pp 554–560

  45. Gershman SJ, Blei DM (2012) A tutorial on Bayesian nonparametric models. J Math Psychol 56(1):1–12

    Article  MathSciNet  MATH  Google Scholar 

  46. Richardson S, Green PJ (1997) On Bayesian analysis of mixtures with an unknown number of components. J R Stat Soc B 59:731–792

    Article  MathSciNet  MATH  Google Scholar 

  47. Song M, Wang H (2005) Highly efficient incremental estimation of Gaussian mixture models for online data stream clustering. In: Defense and security. Proc. SPIE 5803, Intelligent Computing: Theory and Applications III, pp 174–183. doi:10.1117/12.601724

  48. Lv J, Yi Z, Tan K (2007) Determination of the number of principal directions in a biologically plausible PCA model. IEEE Trans Neural Netw Learn Syst 18(3):910–916

    Article  Google Scholar 

Download references

Acknowledgments

This research has been supported by SGS Grant No. 10/279/OHK3/3T/13, sponsored by the CTU in Prague, and by the research programme MSM6840770012 Transdisciplinary Research in the Area of Biomedical Engineering II of the CTU in Prague, sponsored by the Ministry of Education, Youth and Sports of the Czech Republic.

Author information

Authors and Affiliations

  1. Department of Cybernetics, Czech Technical University in Prague, Karlovo námestí 13, 121 35, Prague 2, Czech Republic

    Karla Štepánová

  2. Czech Institut of Informatics, Robotics and Cybernetics, Czech Technical University in Prague, Prague 2, Czech Republic

    Karla Štepánová & Michal Vavrečka

Authors
  1. Karla Štepánová
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michal Vavrečka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Karla Štepánová.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Štepánová, K., Vavrečka, M. Estimating number of components in Gaussian mixture model using combination of greedy and merging algorithm. Pattern Anal Applic 21, 181–192 (2018). https://doi.org/10.1007/s10044-016-0576-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-016-0576-5

Keywords

search

Navigation