Using Kernel PCA for Initialisation of Variational Bayesian Nonlinear Blind Source Separation Method

  • Antti Honkela
  • Stefan Harmeling
  • Leo Lundqvist
  • Harri Valpola
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3195)


The variational Bayesian nonlinear blind source separation method introduced by Lappalainen and Honkela in 2000 is initialised with linear principal component analysis (PCA). Because of the multilayer perceptron (MLP) network used to model the nonlinearity, the method is susceptible to local minima and therefore sensitive to the initialisation used. As the method is used for nonlinear separation, the linear initialisation may in some cases lead it astray. In this paper we study the use of kernel PCA (KPCA) in the initialisation. KPCA is a rather straightforward generalisation of linear PCA and it is much faster to compute than the variational Bayesian method. The experiments show that it can produce significantly better initialisations than linear PCA. Additionally, the model comparison methods provided by the variational Bayesian framework can be easily applied to compare different kernels.


Cost Function Independent Component Analysis Ensemble Learning Variational Bayesian Blind Signal Separation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hyvärinen, A., Karhunen, J., Oja, E.: Independent Component Analysis. J. Wiley, Chichester (2001)CrossRefGoogle Scholar
  2. 2.
    Jutten, C., Karhunen, J.: Advances in nonlinear blind source separation. In: Proc. of the 4th Int. Symp. on Independent Component Analysis and Blind Signal Separation (ICA 2003), pp. 245–256 (2003); Invited paper in the special session on nonlinear ICA and BSSGoogle Scholar
  3. 3.
    Schölkopf, B., Smola, A., Müller, K.-R.: Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation 10(5), 1299–1319 (1998)CrossRefGoogle Scholar
  4. 4.
    Harmeling, S., Ziehe, A., Kawanabe, M., Müller, K.-R.: Kernel-based nonlinear blind source separation. Neural Computation 15(5), 1089–1124 (2003)zbMATHCrossRefGoogle Scholar
  5. 5.
    Lappalainen, H., Honkela, A.: Bayesian nonlinear independent component analysis by multi-layer perceptrons. In: Girolami, M. (ed.) Advances in Independent Component Analysis, pp. 93–121. Springer, Berlin (2000)Google Scholar
  6. 6.
    Valpola, H., Oja, E., Ilin, A., Honkela, A., Karhunen, J.: Nonlinear blind source separation by variational Bayesian learning. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences E86-A(3), 532–541 (2003)Google Scholar
  7. 7.
    Hinton, G.E., van Camp, D.: Keeping neural networks simple by minimizing the description length of the weights. In: Proc. of the 6th Ann. ACM Conf. on Computational Learning Theory, Santa Cruz, CA, USA, pp. 5–13 (1993)Google Scholar
  8. 8.
    MacKay, D.J.C.: Developments in probabilistic modelling with neural networks – ensemble learning. In: Neural Networks: Artificial Intelligence and Industrial Applications. Proc. of the 3rd Annual Symposium on Neural Networks, pp. 191–198 (1995)Google Scholar
  9. 9.
    Valpola, H., Östman, T., Karhunen, J.: Nonlinear independent factor analysis by hierarchical models. In: Proc. 4th Int. Symp. on Independent Component Analysis and Blind Signal Separation (ICA2003), Nara, Japan, pp. 257–262 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Antti Honkela
    • 1
  • Stefan Harmeling
    • 2
  • Leo Lundqvist
    • 1
  • Harri Valpola
    • 1
  1. 1.Neural Networks Research CentreHelsinki University of TechnologyEspooFinland
  2. 2.Fraunhofer FIRST.IDABerlinGermany

Personalised recommendations