Local Factor Analysis with Automatic Model Selection: A Comparative Study and Digits Recognition Application

  • Lei Shi
  • Lei Xu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4132)


A further investigation is made on an adaptive local factor analysis algorithm from Bayesian Ying-Yang (BYY) harmony learning, which makes parameter learning with automatic determination of both the component number and the factor number in each component. A comparative study has been conducted on simulated data sets and several real problem data sets. The algorithm has been compared with not only a recent approach called Incremental Mixture of Factor Analysers (IMoFA) but also the conventional two-stage implementation of maximum likelihood (ML) plus model selection, namely, using the EM algorithm for parameter learning on a series candidate models, and selecting one best candidate by AIC, CAIC, and BIC. Experiments have shown that IMoFA and ML-BIC outperform ML-AIC or ML-CAIC while the BYY harmony learning considerably outperforms IMoFA and ML-BIC. Furthermore, this BYY learning algorithm has been applied to the popular MNIST database for digits recognition with a promising performance.


Gaussian Mixture Model Minimum Description Length Handwritten Digit Digit Recognition Automatic Model Selection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Redner, R.A., Walker, H.F.: Mixture densities, maximum likelihood, and the EM algorithm. SIAM Review 26, 195–239 (1984)MATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Kambhatla, N., Leen, T.K.: Fast non-linear dimension reduction. In: Advances in NIPS 6. In: Advances in NIPS, vol. 6. Morgan Kaufmann, San Francisco (1994)Google Scholar
  3. 3.
    Hinton, G.E., Revow, M., Dayan, P.: Recognizing handwritten digits using mixtures of Linear models. In: Advances in NIPS, vol. 7. MIT Press, Cambridge (1995)Google Scholar
  4. 4.
    Akaike, H.: A new look at statistical model identification. IEEE Trans. Automatic Control 19, 716–723 (1974)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Barron, A., Rissanen, J.: The minimum description length priciple in coding and modeling. IEEE Trans. Information Theory 44, 2743–2760 (1998)MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Bozdogan, H.: Model selection and Akaike’s information criterion (AIC): the general theory and its analytical extensions. Psychometrika 52(3), 345–370 (1987)MATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Figueiredo, M.A.T., Jain, A.K.: Unsupervised learning of finite mixture models. IEEE Trans. Pattern Analysis and Machine Intelligence 24(3), 381–396 (2002)CrossRefGoogle Scholar
  8. 8.
    Rubin, D., Thayer, D.: EM algorithms for ML factor analysis. Psychometrika 47(1), 69–76 (1982)MATHCrossRefMathSciNetGoogle Scholar
  9. 9.
    Schwarz, G.: Estimating the dimension of a model. The Annals of Statistics 6(2), 461–464 (1978)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Ghahramani, Z., Beal, M.: Variational inference for Bayesian mixture of factor analysers. In: Advances in NIPS, vol. 12, pp. 449–455 (2000)Google Scholar
  11. 11.
    Salah, A.A., Alpaydin, E.: Incremental Mixtures of Factor Analysers. In: Proc. 17th Intl Conf. on Pattern Recognition, vol. 1, pp. 276–279 (2004)Google Scholar
  12. 12.
    LeCun, Y., et al.: Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  13. 13.
    Xu, L.: Multisets Modeling Learning: An Unified Theory for Supervised and Unsupervised Learning. In: Proc. IEEE ICNN 1994, Orlando, Florida, June 26-July 2, vol. I, pp. 315–320 (1994) (invited talk)Google Scholar
  14. 14.
    Xu, L.: Temporal BYY Encoding, Markovian State Spaces, and Space Dimension Determination. IEEE Trans. Neural Networks 15(5), 1276–1295 (2004)CrossRefGoogle Scholar
  15. 15.
    Xu, L.: Advances on BYY harmony learning: information theoretic perspective, generalized projection geometry, and independent factor auto-determination. IEEE Trans on Neural Networks 15(4), 885–902 (2004)CrossRefGoogle Scholar
  16. 16.
    Xu, L.: A Unified Perspective and New Results on RHT Computing, Mixture Based Learning, and Multi-learner Based Problem Solving. To appear in a special issue of Pattern Recognition (2006)Google Scholar
  17. 17.
    Xu, L.: Trends on Regularization and Model Selection in Statistical Learning: A Perspective from Bayesian Ying Yang Learning. In: Duch, W., Mandziuk, J., Zurada, J.M. (eds.) Challenges to Computational Intelligence. The Springers series - Studies in Computational Intelligence. Springer, Heidelberg (2006) (in press)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Lei Shi
    • 1
  • Lei Xu
    • 1
  1. 1.Chinese University of Hong KongShatin, NT, Hong Kong

Personalised recommendations