Journal of Signal Processing Systems

, Volume 65, Issue 3, pp 403–412 | Cite as

Regularized Pre-image Estimation for Kernel PCA De-noising

Input Space Regularization and Sparse Reconstruction


The main challenge in de-noising by kernel Principal Component Analysis (PCA) is the mapping of de-noised feature space points back into input space, also referred to as “the pre-image problem”. Since the feature space mapping is typically not bijective, pre-image estimation is inherently illposed. As a consequence the most widely used estimation schemes lack stability. A common way to stabilize such estimates is by augmenting the cost function by a suitable constraint on the solution values. For de-noising applications we here propose Tikhonov input space distance regularization as a stabilizer for pre-image estimation, or sparse reconstruction by Lasso regularization in cases where the main objective is to improve the visual simplicity. We perform extensive experiments on the USPS digit modeling problem to evaluate the stability of three widely used pre-image estimators. We show that the previous methods lack stability in the is non-linear regime, however, by applying our proposed input space distance regularizer the estimates are stabilized with a limited sacrifice in terms of de-noising efficiency. Furthermore, we show how sparse reconstruction can lead to improved visual quality of the estimated pre-image.


Kernel PCA Pre-image Regularization De-noising Sparsity 



This work is supported in part by the Lundbeckfonden through the Center for Integrated Molecular Brain Imaging (Cimbi),


  1. 1.
    Arias, P., Randall, G., & Sapiro, G. (2007). Connecting the out-of-sample and pre-image problems in kernel methods. In IEEE computer society conference on computer vision and pattern recognition (pp. 1–8). Minneapolis, MN.Google Scholar
  2. 2.
    Bakir, G. H., Weston, J., & Schölkopf, B. (2004). Learning to find pre-images. In S. L. Thrun, & B. Schölkopf (Eds.), Advances in neural information processing systems (Vol. 16, pp. 449–456). Cambridge: MIT.Google Scholar
  3. 3.
    Burges, C. J. C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2, 121–167.CrossRefGoogle Scholar
  4. 4.
    Dambreville, S., Rathi, Y., & Tannenbaum, A. (2006). Statistical shape analysis using kernel PCA. In IS&T/SPIE symposium on electrical imaging.Google Scholar
  5. 5.
    Friedman, J. H. (2008). Fast sparse regression and classification. Tech. rep., Department of Statistics, Stanford University.Google Scholar
  6. 6.
    Hull, J. J. (1994). A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(5), 550–554.CrossRefGoogle Scholar
  7. 7.
    Kim, K. I., Franz, M. O., & Schölkopf, B. (2005). Iterative kernel principal component analysis for image modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(9), 1351–1366.CrossRefGoogle Scholar
  8. 8.
    Kwok, J. T. Y., & Tsang, I. W. H. (2004). The pre-image problem in kernel methods. IEEE Transactions on Neural Networks, 15(6), 1517–1525.CrossRefGoogle Scholar
  9. 9.
    Mika, S., Schölkopf, B., Smola, A., Müller, K. R., Scholz, M., & Rätsch, G. (1999). Kernel PCA and de-noising in feature spaces. In M. S. Kearns, S. A. Solla, & D. A. Cohn (Eds.), Advances in neural information processing systems (Vol. 11, pp. 536–542). Cambridge: MIT.Google Scholar
  10. 10.
    Nguyen, M. H., & la Torre, F. D. (2009). Robust kernel principal component analysis. Advances in Neural Information Processing Systems, 21, 1185–1192.Google Scholar
  11. 11.
    Rice, J. R., & White, J. S. (1964). Norms for smoothing and estimation. SIAM Review, 6, 243–256.MathSciNetMATHCrossRefGoogle Scholar
  12. 12.
    Schölkopf, B., Mika, S., Burges, C. J. C., Knirsch, P., Müller, K. R., Rätsch, G., et al. (1999). Input space versus feature space in kernel-based methods. IEEE Transactions On Neural Networks, 10(5), 1000–1017.CrossRefGoogle Scholar
  13. 13.
    Schölkopf, B., Smola, A., & Müller, K. R. (1998). Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10(5), 1299–1319.CrossRefGoogle Scholar
  14. 14.
    Schölkopf, B., Smola, A. J., Knirsch, P., & Burges, C. J. C. (1998). Fast approximation of support vector kernel expansions, and an interpretation of clustering as approximation in feature spaces. In P. Levi, M. Schanz, R.-J. Ahlers, & F. May (Eds.), Mustererkennung 1998, 20. DAGM-Symposium (pp. 125–132). Berlin: Springer.Google Scholar
  15. 15.
    Takahashi,T., & Kurita, T. (2002). Robust de-noising by kernel PCA. In ICANN ’02: Proceedings of the international conference on artificial neural networks (pp. 739–744). Berlin: Springer.Google Scholar
  16. 16.
    Teixeira, A. R., Tomé, A. M., Stadlthanner, K., & Lang, E. W. (2008). KPCA denoising and the pre-image problem revisited. Digital Signal Processing, 18(4), 568–580.CrossRefGoogle Scholar
  17. 17.
    Thorstensen, N., Segonne, F., & Keriven, R. (2008). Normalization and preimage problem in gaussian kernel PCA. In 2008 15th IEEE international conference on image processing—ICIP 2008, (pp. 741–744). San Diego, California.Google Scholar
  18. 18.
    Tibshirani, R. (1994). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, Series B, 58, 267–288.MathSciNetGoogle Scholar
  19. 19.
    Tikhonov, A. N., & Arsenin, V. Y. (1977). Solution of ill-posed problems. New York: Wiley.Google Scholar
  20. 20.
    Zheng, W. S., & Lai, J. H. (2006). Regularized locality preserving learning of pre-image problem in kernel principal component analysis. International Conference on Pattern Recognition, 2, 456–459.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  1. 1.DTU InformaticsTechnical University of DenmarkKgs. LyngbyDenmark

Personalised recommendations