Regularized Principal Manifolds

  • Alex J. Smola
  • Robert C. Williamson
  • Sebastian Mika
  • Bernhard Schölkopf
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1572)


Many settings of unsupervised learning can be viewed as quantization problems — the minimization of the expected quantization error subject to some restrictions. This allows the use of tools such as regularization from the theory of (supervised) risk minimization for unsupervised settings. Moreover, this setting is very closely related to both principal curves and the generative topographic map.

We explore this connection in two ways: 1) we propose an algorithm for finding principal manifolds that can be regularized in a variety of ways. Experimental results demonstrate the feasibility of the approach. 2) We derive uniform convergence bounds and hence bounds on the learning rates of the algorithm. In particular, we give good bounds on the covering numbers which allows us to obtain a nearly optimal learning rate of order \( O(m^{ - \tfrac{1} {2} + \alpha } ) \) for certain types of regularization operators, where m is the sample size and α an arbitrary positive constant.


Quantization Error Unsupervised Learning Regularization Term Reproduce Kernel Hilbert Space Kernel Principal Component Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    P. Bartlett, T. Linder, and G. Lugosi. The minimax distortion redundancy in empirical quantizer design. IEEE Transactions on Information Theory, 44(5):1802–1813, 1998.zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    C.M. Bishop, M. Svensén, and C.K.I. Williams. GTM: The generative topographic mapping. Neural Computation, 10(1):215–234, 1998.CrossRefGoogle Scholar
  3. 3.
    B. Carl and I. Stephani. Entropy, compactness, and the approximation of operators. Cambridge University Press, Cambridge, UK, 1990.zbMATHGoogle Scholar
  4. 4.
    R. Der, U. Steinmetz, B. Balzuweit, and G. Schü ürmann. Nonlinear principal component analysis. University of Leipzig, Preprint,, 1998.
  5. 5.
    M. Hamermesh. Group theory and its applications to physical problems. Addison Wesley, Reading, MA, 2 edition, 1962. Reprint by Dover, New York, NY.Google Scholar
  6. 6.
    T. Hastie and W. Stuetzle. Principal curves. Journal of the American Statistical Association, 84(406):502–516, 1989.zbMATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    B. Kégl, A. Krzyżak, T. Linder, and K. Zeger. Learning and design of principal curves. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1999.
  8. 8.
    B. Schölkopf, A. Smola, and K.-R. Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10:1299–1319, 1998.CrossRefGoogle Scholar
  9. 9.
    A.J. Smola, B. Schölkopf, and K.-R. Müller. The connection between regularization operators and support vector kernels. Neural Networks, 11:637–649, 1998.CrossRefGoogle Scholar
  10. 10.
    A.N. Tikhonov and V.Y. Arsenin. Solution of Ill-Posed Problems. Winston, Washington, DC, 1977.Google Scholar
  11. 11.
    V.N. Vapnik. Estimation of Dependences Based on Empirical Data. Springer-Verlag, Berlin, 1982.zbMATHGoogle Scholar
  12. 12.
    C.K.I. Williams. Prediction with gaussian processes: From linear regression to linear prediction and beyond. Learning and Inference in Graphical Models, 1998.Google Scholar
  13. 13.
    R.C. Williamson, A.J. Smola, and B. Schölkopf. Generalization performance of regularization networks and support vector machines via entropy numbers of compact operators. NeuroCOLT NC-TR-98-019, Royal Holloway College, 1998.Google Scholar
  14. 14.
    A. Yuille and N. Grzywacz. The motion coherence theory. In Proceedings of the International Conference on Computer Vision, pages 344–354, Washington, D.C., December 1988. IEEE Computer Society Press.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1999

Authors and Affiliations

  • Alex J. Smola
    • 1
  • Robert C. Williamson
    • 2
  • Sebastian Mika
    • 1
  • Bernhard Schölkopf
    • 1
  1. 1.GMD FIRSTBerlinGermany
  2. 2.Department of EngineeringAustralian National UniversityCanberraAustralia

Personalised recommendations