Advertisement

Heteroscedastic Probabilistic Linear Discriminant Analysis with Semi-supervised Extension

  • Yu Zhang
  • Dit-Yan Yeung
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5782)

Abstract

Linear discriminant analysis (LDA) is a commonly used method for dimensionality reduction. Despite its successes, it has limitations under some situations, including the small sample size problem, the homoscedasticity assumption that different classes have the same Gaussian distribution, and its inability to produce probabilistic output and handle missing data. In this paper, we propose a semi-supervised and heteroscedastic extension of probabilistic LDA, called S2HPLDA, which aims at overcoming all these limitations under a common principled framework. Moreover, we apply automatic relevance determination to determine the required dimensionality of the low-dimensional space for dimensionality reduction. We empirically compare our method with several related probabilistic subspace methods on some face and object databases. Very promising results are obtained from the experiments showing the effectiveness of our proposed method.

Keywords

Face Recognition Unlabeled Data Small Sample Size Problem Recognition Error Rate Label Data Point 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Jolliffe, I.T.: Principal Component Analysis, 2nd edn. Springer, New York (2002)zbMATHGoogle Scholar
  2. 2.
    Fukunaga, K.: Introduction to Statistical Pattern Recognition. Academic Press, New York (1991)zbMATHGoogle Scholar
  3. 3.
    Chen, L., Liao, H., Ko, M., Lin, J., Yu, G.: A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recognition 33(10), 1713–1726 (2000)CrossRefGoogle Scholar
  4. 4.
    Krzanowski, W.J., Jonathan, P., McCarthy, W.V., Thomas, M.R.: Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data. Applied Statistics 44(1), 101–115 (1995)CrossRefzbMATHGoogle Scholar
  5. 5.
    Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7), 711–720 (1997)CrossRefGoogle Scholar
  6. 6.
    Ye, J., Li, Q.: A two-stage linear discirminant analysis via QR-decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(6), 929–941 (2005)CrossRefGoogle Scholar
  7. 7.
    Cevikalp, H., Neamtu, M., Wilkes, M., Barkana, A.: Discriminative common vectors for face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(1), 4–13 (2005)CrossRefGoogle Scholar
  8. 8.
    Wang, X., Tang, X.: Dual-space linear discriminant analysis for face recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, pp. 564–569 (2004)Google Scholar
  9. 9.
    Ye, J., Janardan, R., Li, Q.: Two-dimensional linear discriminant analysis. In: Advances in Neural Information Processing Systems 17, Vancouver, British Columbia, Canada, pp. 1529–1536 (2005)Google Scholar
  10. 10.
    Cai, D., He, X., Han, J.: Semi-supervised discriminant analysis. In: Proceedings of the IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil (2007)Google Scholar
  11. 11.
    Zhang, Y., Yeung, D.Y.: Semi-supervised discriminant analysis using robust path-based similarity. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska (2008)Google Scholar
  12. 12.
    Zhang, Y., Yeung, D.Y.: Semi-supervised discriminant analysis via CCCP. In: Proceedings of the 19th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Antwerp, Belgium, pp. 644–659 (2008)Google Scholar
  13. 13.
    Chapelle, O., Zien, A., Schölkopf, B. (eds.): Semi-Supervised Learning. MIT Press, Boston (2006)Google Scholar
  14. 14.
    Hastie, T., Tibshirani, R.: Discriminant analysis by Gaussian mixture. Journal of the Royal Statistical Society, Series B 58(1), 155–176 (1996)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Kumar, N., Andreou, A.G.: Heteroscedastic discriminant analysis and reduced rank HMMS for improved speech recognition. Speech Communication 26(4), 283–297 (1998)CrossRefGoogle Scholar
  16. 16.
    Loog, M., Duin, R.P.W.: Linear dimensionality reduction via a heteroscedastic extension of LDA: the Chernoff criterion. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(6), 732–739 (2004)CrossRefGoogle Scholar
  17. 17.
    Dai, G., Yeung, D.Y., Chang, H.: Extending kernel fisher discriminant analysis with the weighted pairwise Chernoff criterion. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 308–320. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  18. 18.
    Ioffe, S.: Probabilistic linear discriminant analysis. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 531–542. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  19. 19.
    Yu, S., Yu, K., Tresp, V., Kriegel, H.P., Wu, M.: Supervised probabilistic principal component analysis. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, pp. 464–473 (2006)Google Scholar
  20. 20.
    Prince, S.J.D., Elder, J.H.: Probabilistic linear discriminant analysis for inferences about identity. In: Proceedings of the 11th IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil, pp. 1–8 (2007)Google Scholar
  21. 21.
    Neal, R.M.: Bayesian Learning for Neural Network. Springer, New York (1996)CrossRefzbMATHGoogle Scholar
  22. 22.
    Tipping, M.E., Bishop, C.M.: Probabilistic principal component analysis. Journal of the Royal Statistic Society, B 61(3), 611–622 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Bach, F.R., Jordan, M.I.: A probabilistic interpretation of canonical correlation analysis. Technical Report 688, Department of Statistics, University of California, Berkeley (2005)Google Scholar
  24. 24.
    He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.J.: Face recognition using Laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(3), 328–340 (2005)CrossRefGoogle Scholar
  25. 25.
    Belkin, M., Niyogi, P., Sindhwani, V.: Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research 7, 2399–2434 (2006)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Zelnik-Manor, L., Perona, P.: Self-tuning spectral clustering. In: Advances in Neural Information Processing Systems 17, Vancouver, British Columbia, Canada, pp. 1601–1608 (2005)Google Scholar
  27. 27.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistic Society, B 39(1), 1–38 (1977)MathSciNetzbMATHGoogle Scholar
  28. 28.
    Tipping, M.E., Bishop, C.M.: Mixtures of probabilistic principal component analysers. Neural Computation 11(2), 443–482 (1999)CrossRefGoogle Scholar
  29. 29.
    Joachims, T.: Transductive inference for text classification using support vector machines. In: Proceedings of the Sixteenth International Conference on Machine Learning, San Francisco, CA, USA, pp. 200–209 (1999)Google Scholar
  30. 30.
    Bennett, K., Demiriz, A.: Semi-supervised support vector machines. In: Advances in Neural Information Processing Systems 11, Vancouver, British Columbia, Canada, pp. 368–374 (1998)Google Scholar
  31. 31.
    MacKay, D.J.C.: Comparison of approximate methods for handling hyperparameters. Neural Computation 11(5), 1035–1068 (1999)CrossRefGoogle Scholar
  32. 32.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  33. 33.
    Turk, M., Pentland, A.: Eigenfaces for recognition. Journal of Cognitive Neuroscience 3(1), 71–86 (1991)CrossRefGoogle Scholar
  34. 34.
    Sim, T., Baker, S., Bsat, M.: The CMU pose, illumination and expression database. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(12), 1615–1618 (2003)CrossRefGoogle Scholar
  35. 35.
    Nene, S.A., Nayar, S.K., Murase, H.: Columbia object image library (COIL-20). Technical Report 005, CUCS (1996)Google Scholar
  36. 36.
    Lawrence, N.D.: Probabilistic non-linear principal component analysis with Gaussian process latent variable models. Journal of Machine Learning Research 6, 1783–1816 (2005)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Yu Zhang
    • 1
  • Dit-Yan Yeung
    • 1
  1. 1.Hong Kong University of Science and TechnologyHong Kong

Personalised recommendations