Advertisement

Semi-supervised Discriminant Analysis Via CCCP

  • Yu Zhang
  • Dit-Yan Yeung
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5212)

Abstract

Linear discriminant analysis (LDA) is commonly used for dimensionality reduction. In real-world applications where labeled data are scarce, LDA does not work very well. However, unlabeled data are often available in large quantities. We propose a novel semi-supervised discriminant analysis algorithm called SSDA\(_{\mathit{CCCP}}\). We utilize unlabeled data to maximize an optimality criterion of LDA and use the constrained concave-convex procedure to solve the optimization problem. The optimization procedure leads to estimation of the class labels for the unlabeled data. We propose a novel confidence measure for selecting those unlabeled data points with high confidence. The selected unlabeled data can then be used to augment the original labeled data set for performing LDA. We also propose a variant of SSDA\(_{\mathit{CCCP}}\), called M-SSDA\(_{\mathit{CCCP}}\), which adopts the manifold assumption to utilize the unlabeled data. Extensive experiments on many benchmark data sets demonstrate the effectiveness of our proposed methods.

Keywords

Linear Discriminant Analysis Label Data Unlabeled Data Scatter Matrix Kernel Discriminant Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Fisher, R.A.: The use of multiple measurements in taxonomic problems. Annals of Eugenics 7, 179–188 (1936)Google Scholar
  2. 2.
    Rao, C.R.: The utilization of multiple measurements in problems of biological classification. Journal of the Royal Statictical Society 10, 159–203 (1948)zbMATHGoogle Scholar
  3. 3.
    Chen, L., Liao, H., Ko, M., Lin, J., Yu, G.: A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recognition 33(10), 1713–1726 (2000)CrossRefGoogle Scholar
  4. 4.
    Krzanowski, W.J., Jonathan, P., McCarthy, W.V., Thomas, M.R.: Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data. Applied Statistics 44(1), 101–115 (1995)zbMATHCrossRefGoogle Scholar
  5. 5.
    Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7), 711–720 (1997)CrossRefGoogle Scholar
  6. 6.
    Ye, J.P., Li, Q.: A two-stage linear discirminant analysis via QR-Decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(6), 929–941 (2005)CrossRefGoogle Scholar
  7. 7.
    Wang, X., Tang, X.: Dual-space linear discriminant analysis for face recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, pp. 564–569 (2004)Google Scholar
  8. 8.
    Jolliffe, I.T.: Principal Component Analysis. Springer, New York (1986)Google Scholar
  9. 9.
    Cevikalp, H., Neamtu, M., Wilkes, M., Barkana, A.: Discriminative common vectors for face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(1), 4–13 (2005)CrossRefGoogle Scholar
  10. 10.
    Ye, J.P., Janardan, R., Li, Q.: Two-dimensional linear discriminant analysis. In: Advances in Neural Information Processing Systems 17, Vancouver, British Columbia, Canada, pp. 1529–1536 (2004)Google Scholar
  11. 11.
    Baudat, G., Anouar, F.: Generalized discriminant analysis using a kernel approach. Neural Computation 12(10), 2385–2404 (2000)CrossRefGoogle Scholar
  12. 12.
    Ye, J.P., Xiong, T.: Computational and theoretical analysis of null space and orthogonal linear discriminant analysis. Journal of Machine Learning Research 7, 1183–1204 (2006)MathSciNetGoogle Scholar
  13. 13.
    Ye, J.P.: Least squares linear discriminant analysis. In: Proceedings of the Twenty-Fourth International Conference on Machine Learning, Corvalis, Oregon, USA, pp. 1087–1093 (2007)Google Scholar
  14. 14.
    Kim, S.J., Magnani, A., Boyd, S.: Optimal kernel selection in kernel fisher discriminant analysis. In: Proceedings of the Twenty-Third International Conference on Machine Learning, Pittsburgh, Pennsylvania, USA, pp. 465–472 (2006)Google Scholar
  15. 15.
    Ye, J.P., Chen, J., Ji, S.: Discriminant kernel and regularization parameter learning via semidefinite programming. In: Proceedings of the Twenty-Fourth International Conference on Machine Learning, Corvalis, Oregon, USA, pp. 1095–1102 (2007)Google Scholar
  16. 16.
    Zhu, X.: Semi-supervised learning literature survey. Technical Report 1530, Department of Computer Sciences, University of Wisconsin at Madison, Madison, WI (2006)Google Scholar
  17. 17.
    Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: Proceedings of the Workshop on Computational Learning Theory, Madison, Wisconsin, USA, pp. 92–100 (1998)Google Scholar
  18. 18.
    Bennett, K., Demiriz, A.: Semi-supervised support vector machines. In: Advances in Neural Information Processing Systems 11, Vancouver, British Columbia, Canada, pp. 368–374 (1998)Google Scholar
  19. 19.
    Joachims, T.: Transductive inference for text classification using support vector machines. In: Proceedings of the Sixteenth International Conference on Machine Learning, San Francisco, CA, USA, pp. 200–209 (1999)Google Scholar
  20. 20.
    Belkin, M., Niyogi, P., Sindhwani, V.: Manifold regularization: A geometric framework for learning from examples. Journal of Machine Learning Research 7, 2399–2434 (2006)MathSciNetGoogle Scholar
  21. 21.
    Zhou, D., Bousquet, O., Lal, T., Weston, J., Schölkopf, B.: Learning with local and global consistency. In: Advances in Neural Information Processing Systems 16, Vancouver, British Columbia, Canada (2003)Google Scholar
  22. 22.
    Zhu, X., Ghahramani, Z., Lafferty, J.: Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of the Twentieth International Conference on Machine Learning, Washington, DC, pp. 912–919 (2003)Google Scholar
  23. 23.
    Cai, D., He, X., Han, J.: Semi-supervised discriminant analysis. In: Proceedings of the IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil (2007)Google Scholar
  24. 24.
    Yuille, A., Rangarajan, A.: The concave-convex procedure. Neural Computation 15(4), 915–936 (2003)zbMATHCrossRefGoogle Scholar
  25. 25.
    Smola, A.J., Vishwanathan, S.V.N., Hofmann, T.: Kernel methods for missing variables. In: Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, Barbados (2005)Google Scholar
  26. 26.
    Fukunnaga, K.: Introduction to Statistical Pattern Recognition. Academic Press, New York (1991)Google Scholar
  27. 27.
    Yang, J., Yang, J.Y.: Why can LDA be performed in PCA transformed space? Pattern Recognition 36(2), 563–566 (2003)CrossRefGoogle Scholar
  28. 28.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, New York (2004)zbMATHGoogle Scholar
  29. 29.
    Zelnik-Manor, L., Perona, P.: Self-tuning spectral clustering. In: Advances in Neural Information Processing Systems 17, Vancouver, British Columbia, Canada, pp. 1601–1608 (2004)Google Scholar
  30. 30.
    Ye, J.P., Zhao, Z., Wu, M.: Discriminative k-means for clustering. In: Advances in Neural Information Processing Systems 20 (2007)Google Scholar
  31. 31.
    Asuncion, A., Newman, D.: UCI machine learning repository (2007)Google Scholar
  32. 32.
    Sim, T., Baker, S., Bsat, M.: The CMU pose, illumination and expression database. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(12), 1615–1618 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Yu Zhang
    • 1
  • Dit-Yan Yeung
    • 1
  1. 1.Hong Kong University of Science and Technology 

Personalised recommendations