Unsupervised Slow Subspace-Learning from Stationary Processes

  • Andreas Maurer
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4264)


We propose a method of unsupervised learning from stationary, vector-valued processes. A low-dimensional subspace is selected on the basis of a criterion which rewards data-variance (like PSA) and penalizes the variance of the velocity vector, thus exploiting the short-time dependencies of the process. We prove error bounds in terms of the β-mixing coefficients and consistency for absolutely regular processes. Experiments with image recognition demonstrate the algorithms ability to learn geometrically invariant feature maps.


Kernel Principal Component Analysis Stationary Stochastic Process Batch Algorithm Slow Feature Analysis Minimal Subspace 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Benveniste, A., Métevier, M., Priouret, P.: Adaptive Algorithms and Stochastic Approximations. Springer, Heidelberg (1987)MATHGoogle Scholar
  2. 2.
    Földiák, P.: Learning invariance from transformation sequences. Neural Computation 3, 194–200 (1991)CrossRefGoogle Scholar
  3. 3.
    Lozano, A.C., Kulkarni, S.R., Shapire, R.E.: Convergence and consistency of regularized boosting algorithms with stationary, β-mixing observations. In: Advances in Neural Information Processing Systems 18 (2006)Google Scholar
  4. 4.
    Manton, J.H., Helmke, U., Mareels, I.M.Y.: A dual purpose principal and minor component flow. Systems & Control Letters 54, 759–769 (2005)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Maurer, A.: Bounds for linear multi-task learning. JMLR 7, 117–139 (2006)MathSciNetGoogle Scholar
  6. 6.
    Maurer, A.: Generalization Bounds for Subspace Selection and Hyperbolic PCA. In: Saunders, C., Grobelnik, M., Gunn, S., Shawe-Taylor, J. (eds.) SLSFS 2005. LNCS, vol. 3940, pp. 185–197. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  7. 7.
    McDiarmid, C.: Concentration. Probabilistic Methods of Algorithmic Discrete Mathematics, pp. 195–248. Springer, Berlin (1998)Google Scholar
  8. 8.
    Meir, R.: Nonparametric time series prediction through adaptive model selection. Machine Learning 39, 5–34 (2000)MATHCrossRefGoogle Scholar
  9. 9.
    Oja, E.: Principal component analysis. In: Arbib, M.A. (ed.) The Handbook of Brain Theory and Neural Networks, pp. 910–913. MIT Press, Cambridge (2002)Google Scholar
  10. 10.
    Mika, S., Schölkopf, B., Smola, A., Müller, K.-R., Scholz, M., Rätsch, G.: Kernel PCA and De-noising in Feature Spaces. In: Advances in Neural Information Processing Systems 11 (1998)Google Scholar
  11. 11.
    Shawe-Taylor, J., Christianini, N.: Estimating the moments of a random vector. In: Proceedings of GRETSI 2003 Conference, vol. I, pp. 47–52 (2003)Google Scholar
  12. 12.
    Reed, M., Simon, B.: Functional Analysis, part I. In: Methods of Mathematical Physics. Academic Press, London (1980)Google Scholar
  13. 13.
    Rio, E.: Théorie asymptotique des processus al éatoires faiblement dépendants. Springer, Heidelberg (2000)Google Scholar
  14. 14.
    Simon, B.: Trace Ideals and Their Applications. Cambridge University Press, London (1979)MATHGoogle Scholar
  15. 15.
    Shawe-Taylor, J., Williams, C.K.I., Cristianini, N., Kandola, J.S.: On the eigenspectrum of the Gram matrix and the generalization error of kernel-PCA. IEEE Transactions on Information Theory 51(7), 2510–2522 (2005)CrossRefMathSciNetGoogle Scholar
  16. 16.
    Vidyasagar, M.: Learning and generalization with applications to neural networks. Springer, London (2003)Google Scholar
  17. 17.
    Wiskott, L., Sejnowski, T.: Slow feature analysis: Unsupervised learning of invariances. Neural Computation 14, 715–770 (2003)CrossRefGoogle Scholar
  18. 18.
    Yan, W., Helmke, U., Moore, J.B.: Global analysis of Oja’s flow for neural networks. IEEE Trans. on Neural Networks 5(5), 674–683 (1994)CrossRefGoogle Scholar
  19. 19.
    Yu, B.: Rate of convergence for empirical processes of stationary mixing sequences. Annals of Probability 22, 94–116 (1994)MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Andreas Maurer
    • 1
  1. 1. München

Personalised recommendations