Kernel Principal Component Analysis for Large Scale Data Set

  • Haixian Wang
  • Zilan Hu
  • Yu’e Zhao
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4113)


Kernel principal component analysis (KPCA) has provided an extremely powerful approach to extracting nonlinear features via kernel trick, and it has been suggested for a number of applications. Whereas the nonlinearity can be allowed by the utilization of Mercer kernels, the standard KPCA could only process limited number of training samples. For large scale data set, it may suffer from computational problem of diagonalizing large matrices, and occupy large storage space. In this paper, by choosing a subset of the entire training samples using Gram-Schmidt orthonormalization and incomplete Cholesky decomposition, we formulate KPCA as another eigenvalue problem of matrix whose size is much smaller than that of the kernel matrix. The theoretical analysis and experimental results on both artificial and real data have shown the advantages of the proposed method for performing KPCA in terms of computational efficiency and storage space, especially when the number of data points is large.


Training Sample Storage Space Large Scale Data Kernel Matrix Kernel Principal Component Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Schölkopf, B., Smola, A.J., Müller, K.R.: Nonlinear Component Analysis as A Kernel Eigenvalue Prolem. Neural Computation 10, 1299–1319 (1998)CrossRefGoogle Scholar
  2. 2.
    Mika, S., Schölkopf, B., Smola, A.J., Müller, K.R., Scholz, M., Rätsch, G.: Kernel PCA and De-noising in Feature Spaces. In: Kearns, M.S., Solla, S.A., Cohn, D.A. (eds.) Advances in Neural Information Processing Systems, Cambridge, vol. 11, pp. 536–542 (1999)Google Scholar
  3. 3.
    Rosipal, R., Girolami, M., Trejo, L.J., Cichocki, A.: Kernel PCA for Feature Extraction and De-noising in Nonlinear Regression. Neural Comput. Application 10, 231–243 (2001)MATHCrossRefGoogle Scholar
  4. 4.
    Kim, K.I., Jung, K., Kim, H.J.: Face Recognition Using Kernel Principal Component Analysis. IEEE Signal Proc. Letter 19, 40–42 (2002)Google Scholar
  5. 5.
    Yang, M.H.: Kernel Eigenfaces vs. Kernel Fisherfaces: Face Recognition Using Kernel Methods. In: Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, Washington, pp. 215–220 (2002)Google Scholar
  6. 6.
    Kong, H., Wang, L., Teoh, E.K., Li, X., Wang, J.G., Venkateswarlu, R.: Generalized 2D Principal Component Analysis for Face Image Representation and Recognition. Neural Networks 18, 585–594 (2005)CrossRefGoogle Scholar
  7. 7.
    Kim, K.I., Franz, M.O., Schölkopf, B.: Iterative Kernel Principal Component Analysis for Image Modelling. IEEE Trans. Patt. Anal. Mach. Intell. 27, 1351–1366 (2005)CrossRefGoogle Scholar
  8. 8.
    Rosipal, R., Girolami, M.: An Expectation-Maximization Approach to Nonlinear Component Analysis. Neural Computation 13, 505–510 (2001)MATHCrossRefGoogle Scholar
  9. 9.
    Zheng, W., Zou, C., Zhao, L.: An Improved Algorithm for Kernel Principal Component Analysis. Neural Proc. Letter 22, 49–56 (2005)CrossRefGoogle Scholar
  10. 10.
    Baudat, G., Anouar, F.: Kernel-Based Methods and Function Approximation. In: Proc. Int. Conf. Neural Networks, Washington, pp. 1244–1249 (2001)Google Scholar
  11. 11.
    Smola, A.J., Schölkopf, B.: Sparse Greedy Matrix Approximation for Machine Learning. In: Langley, P. (ed.) Proceedings of the Seventeenth International Conference on Machine Learning, pp. 911–918. Morgan Kaufmann, San Francisco (2000)Google Scholar
  12. 12.
    Tipping, M.: Sparse kernel Principal Component Analysis. In: Leen, T.K., Diettrich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing Systems, Cambridge, vol. 13, pp. 633–639 (2001)Google Scholar
  13. 13.
    Ouimet, M., Bengio, Y.: Greedy Spectral Embedding. In: Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, Barbade, pp. 253–260 (2005)Google Scholar
  14. 14.
    Franc, V., Hlaváč, V.: Greedy Algorithm for a Training Set Reduction in the Kernel Methods. In: Petkov, N., Westenberg, M.A. (eds.) CAIP 2003. LNCS, vol. 2756, pp. 426–433. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  15. 15.
    Cristianini, N., Lodhi, H., Shawe-Taylor, J.: Latent Semantic Kernels. J. Intell. Inf. Syst. 18, 127–152 (2002)CrossRefGoogle Scholar
  16. 16.
    Bach, F.R., Jordan, M.I.: Kernel Independent Component Analysis. J. Mach. Learn. Res. 3, 1–48 (2002)CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Haixian Wang
    • 1
  • Zilan Hu
    • 2
  • Yu’e Zhao
    • 3
  1. 1.Research Center for Learning ScienceSoutheast UniversityNanjingP.R. China
  2. 2.School of Mathematics and PhysicsAnhui University of TechnologyMaanshanP.R. China
  3. 3.Department of MathematicsQingdao UniversityQingdaoP.R. China

Personalised recommendations