Mixtures of principal components Gaussians for density estimation in high dimension data spaces
An approximation of the gaussian model for density estimation in high-dimension data spaces is presented. The work is mainly motived by the need for a numerically tractable model in high dimension data spaces. The characteristic of the model is to restrict to the Principal Component of each local covariance matrix with the advantage of still being a probabilistic model. The likelihood of the local density is studied and an iterative algorithm is next proposed so as to learn the model. This latter is an adaptation of the well-known iterative Generalized Hebbian Algorithm. A comparison is done with related works based on Factorial Analysis. First experiments on handwritten digits are also reported.
KeywordsDensity estimation Approximation by mixtures Handwritten recognition
Unable to display preview. Download preview PDF.
- Dempster, A. P., Laird N. M. and Rubin, D. B. (1977), “Maximum Likelihood from incomplete data via the EM algorithm”, J. Royal Stat. Soc. B45(1), 47–50.Google Scholar
- Fukunaga K., (1990), “statistical pattern recognition”, second edition, Academic Press.Google Scholar
- Ghahramani Z., Hinton G. E. (1997), ”The EM algorithm for mixture of factorial analyzers”, University of Toronto CRG-TR-96-1. ftp://ftp.cs.toronto.edu/pub/zoubin/tr-96-l.ps.gzGoogle Scholar
- Kambhatla N., Leen T. K. (1995), “ Classifying with Gaussian Mixtures and clusters”, in Advances in Neural Information Processing Systems 7.Google Scholar
- Lemarié B., Gilloux M. and Leroux M.(1996), ”Handwritten Word recognition using Contextual Hybrid RBF networks/Hidden Markov Models”, in Advances in Neural Information Processing Systems 8.Google Scholar
- Lemarié B.,.(1998), ”Mélanges de gaussiennes en composantes principales pp our ('estimation de densité”, in Proceedings of RFIA, AFCET,France,1'Ih://ftp.srt-paste.frlpub/rmo/bernarcViiu98.ps.7..Google Scholar
- Mao J., Jain A. K. (1995), Artificial neural networks for feature extraction and multivariate data projection”, IEE transactions in neural networks, vol 6, no 2, 1995.Google Scholar
- Neal, R. N., Hinton, G. E. (1993), ”A new view of the EM algorithm that justifies Incremental and other variants”, Univeristy of Toronto, Dept. of computer Science, preprint.Google Scholar
- Ormoneit D., Tresp V. (1996), ”Improved Gaussian Mixture Density Estimates Using Bayesian Penalty Term and Network Averaging”, in Advances in Neural Information Processing Systems 8.Google Scholar
- Rao, C. R., “ Estimation and tests of signifiance in factor analysis”, Psychometrika, 20, 93–11, 1955.Google Scholar
- Tipping M. E., Bishop M. C, (06/1997), Mixtures of probabilistic principal components analyser”, Aston University NCRG Technical Report, http://neural-server.aston.ac.uk/cgi-binGoogle Scholar
- Xu I., Jordan M. I., (1996), “on converge properties of the EM algorithm for gaussian mixtures”, neural computation, 8,129–151.Google Scholar