A Comparative Study on Selection of Cluster Number and Local Subspace Dimension in the Mixture PCA Models

  • Xuelei Hu
  • Lei Xu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3971)


How to determine the number of clusters and the dimensions of local principal subspaces is an important and challenging problem in various applications. Based on a probabilistic model of local PCA, this problem can be solved by one of existing statistical model selection criteria in a two-phase procedure. However, such a two-phase procedure is too time-consuming especially when there is no prior knowledge. The BYY harmony learning has provided a promising mechanism to make automatic model selection in parallel with parameter learning. This paper investigates the BYY harmony learning with automatic model selection on a mixture PCA model in comparison with three typical model selection criteria: AIC, CAIC, and MDL. This comparative study is made by experiments for different model selection tasks on simulated data sets under different conditions. Experiments have shown that automatic model selection by the BYY harmony learning are not only as good as or even better than conventional methods in terms of performances, but also considerably supervisory in terms of much less computational cost.


Model Selection Cluster Number Minimum Description Length Model Selection Criterion Parameter Learning 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Xu, L.: Beyond PCA Learning: From Linear to Nonlinear and From Global Representation to Local Representation. In: Proc. Intl. Conf. on Neural Information Processing (ICONIP 1994), Seoul, Korea, vol. 2, pp. 943–949 (1994)Google Scholar
  2. 2.
    Xu, L.: Theories for Unsupervised Learning: PCA and Its Nonlinear Extensions. In: Proc. of IEEE ICNN 1994, Orlando, Florida, vol. II, pp. 1252–1257 (1994)Google Scholar
  3. 3.
    Xu, L.: Bayesian-Kullback Coupled Ying-Yang Machines: Unified Leanings and New Results on Vector Quantization. In: Proc. Intl. Conf. on Neural Information Processing (ICONIP 1995), Beijing, China, pp. 977–988 (1995)Google Scholar
  4. 4.
    Tipping, M.E., Bishop, C.M.: Mixtures of Probabilistic Principal Component Analysers. Neural Computation 11, 443–482 (1999)CrossRefGoogle Scholar
  5. 5.
    Xu, L.: Independent Component Analysis and Extensions with Noise and Time: A Bayesian Ying-Yang Learning Perspective. Neural Information Processing Letters and Reviews 1, 1–52 (2003)Google Scholar
  6. 6.
    Akaike, H.: A New Look at Statistical Model Identification. IEEE Transactions on Automatic Control 19, 716–723 (1974)MATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Bozdogan, H.: Model Selection and Akaike’s Information Criterion (AIC): The General Theory and Its Analytical Extensions. Psychometrika 52, 345–370 (1987)MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Rissanen, J.: Modeling by Shortest Data Description. Automatica 14, 465–471 (1978)MATHCrossRefGoogle Scholar
  9. 9.
    Barron, A., Rissanen, J.: The Minimum Description Length Principle in Coding and Modeling. IEEE Trans. Information Theory 44, 2743–2760 (1998)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Xu, L.: Advances on BYY Harmony Learning: Information Theoretic Perspective, Generalized Projection Geometry, and Independent Factor Auto-determination. IEEE Trans on Neural Networks 15, 885–902 (2004)CrossRefGoogle Scholar
  11. 11.
    Xu, L.: Bayesian Ying Yang Learning (I): A Unified Perspective for Statistical Modeling. In: Zhong, N., Liu, J. (eds.) Intelligent Technologies for Information Analysis, pp. 615–659. Springer, Heidelberg (2004)Google Scholar
  12. 12.
    Xu, L.: Bayesian Ying Yang Learning (II): A New Mechanism for Model Selection and Regularization. In: Zhong, N., Liu, J. (eds.) Intelligent Technologies for Information Analysis, pp. 661–706. Springer, Heidelberg (2004)Google Scholar
  13. 13.
    Roweis, S.: EM Algorithms for PCA and SPCA. In: Jordan, M.I., Kearns, M.J., Solla, S.A. (eds.) Advances in Neural Information Processing Systems, vol. 10, pp. 626–632. The MIT Press, Cambridge (1998)Google Scholar
  14. 14.
    Sclove, S.L.: Some Aspects of Model-Selection Criteria. In: Bozdogan, H. (ed.) Proceedings of the First US/Japan Conference on the Frontiers of Statistical Modeling: An Informational Approach, vol. 2, pp. 37–67. Kluwer Academic Publishers, Dordrecht (1994)Google Scholar
  15. 15.
    Akaike, H.: Factor Analysis and AIC. Psychometrika 52, 317–332 (1987)MATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Xu, L.: BYY Harmony Learning, Structural RPCL, and Topological Self-Organizing on Mixture Models. Neural Networks 15, 1125–1151 (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Xuelei Hu
    • 1
  • Lei Xu
    • 2
  1. 1.Department of Computer Science and TechnologyNanjing University of Science and TechnologyNanjingChina
  2. 2.Department of Computer Science and EngineeringThe Chinese University of Hong KongShatin, NT, Hong Kong

Personalised recommendations