Abstract
This paper develops a kernelized slow feature analysis (SFA) algorithm. SFA is an unsupervised learning method to extract features which encode latent variables from time series. Generative relationships are usually complex, and current algorithms are either not powerful enough or tend to over-fit. We make use of the kernel trick in combination with sparsification to provide a powerful function class for large data sets. Sparsity is achieved by a novel matching pursuit approach that can be applied to other tasks as well. For small but complex data sets, however, the kernel SFA approach leads to over-fitting and numerical instabilities. To enforce a stable solution, we introduce regularization to the SFA objective. Versatility and performance of our method are demonstrated on audio and video data sets.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Assmann, P.F., Nearey, T.M., Bharadwaj, S.: Analysis and classification of a vowel database. Canadian Acoustics 36(3), 148–149 (2008)
Becker, S., Hinton, G.E.: A self-organizing neural network that discovers surfaces in randomdot stereograms. Nature 355(6356), 161–163 (1992)
Berkes, P., Wiskott, L.: Slow feature analysis yields a rich repertoire of complex cell properties. Journal of Vision 5, 579–602 (2005)
Bray, A., Martinez, D.: Kernel-based extraction of Slow features: Complex cells learn disparity and translation invariance from natural images. In: Neural Information Processing Systems, vol. 15, pp. 253–260 (2002)
Csató, L., Opper, M.: Sparse on-line gaussian processes. Neural Computation 14(3), 641–668 (2002)
Davison, A.J.: Real-time simultaneous localisation and mapping with a single camera. In: IEEE International Conference on Computer Vision, pp. 1403–1410 (2003)
Einhäuser, W., Hipp, J., Eggert, J., Körner, E., König, P.: Learning viewpoint invariant object representations using temporal coherence principle. Biological Cybernetics 93(1), 79–90 (2005)
Földiák, P.: Learning invariance from transformation sequences. Neural Computation 3(2), 194–200 (1991)
Franzius, M., Sprekeler, H., Wiskott, L.: Slowness and sparseness leads to place, head-direction, and spatial-view cells. PLoS Computational Biology 3(8), e166 (2007)
Fukumizu, K., Bach, F.R., Gretton, A.: Statistical consistency of kernel canonical correlation analysis. Journal of Machine Learning Research 8, 361–383 (2007)
Huke, J.P.: Embedding nonlinear dynamical systems: A guide to takens’ theorem. Technical report, University of Manchester (2006)
Hussain, Z., Shawe-Taylor, J.: Theory of matching pursuit. In: Advances in Neural Information Processing Systems, vol. 21, pp. 721–728 (2008)
Lowe, D.G.: Object recognition from local scale-invariant features. In: International Conference on Computer Vision, pp. 1150–1157 (1999)
Mallat, S., Zhang, Z.: Matching pursuits with time-frequency dictionaries. IEEE Transactions On Signal Processing 41, 3397–3415 (1993)
Meyn, S.P., Tweedie, R.L.: Markov chains and stochastic stability. Springer, London (1993)
Schölkopf, B., Smola, A., Müller, K.-R.: Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation 10(5), 1299–1319 (1998)
Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)
Smola, A.J., Schölkopf, B.: Sparse greedy matrix approximation for machine learning. In: Proceedings to the 17th International Conference Machine Learning, pp. 911–918 (2000)
Stone, J.V.: Blind source separation using temporal predictability. Neural Computation 13(7), 1559–1574 (2001)
Takens, F.: Detecting strange attractors in turbulence. Dynamical Systems and Turbulence, 366–381 (1981)
Wahba, G.: Spline Models for Observational Data. Society for Industrial and Applied Mathematics, Philadelphia (1990)
Wiskott, L.: Slow feature analysis: A theoretical analysis of optimal free responses. Neural Computation 15(9), 2147–2177 (2003)
Wiskott, L., Sejnowski, T.: Slow feature analysis: Unsupervised learning of invariances. Neural Computation 14(4), 715–770 (2002)
Wyss, R., König, P., Verschure, P.F.M.J.: A model of the ventral visual system based on temporal stability and local memory. PLoS Biology 4(5), e120 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Böhmer, W., Grünewälder, S., Nickisch, H., Obermayer, K. (2011). Regularized Sparse Kernel Slow Feature Analysis. In: Gunopulos, D., Hofmann, T., Malerba, D., Vazirgiannis, M. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2011. Lecture Notes in Computer Science(), vol 6911. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23780-5_25
Download citation
DOI: https://doi.org/10.1007/978-3-642-23780-5_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-23779-9
Online ISBN: 978-3-642-23780-5
eBook Packages: Computer ScienceComputer Science (R0)