Machine Learning

, Volume 46, Issue 1–3, pp 53–70 | Cite as

Hierarchical Learning in Polynomial Support Vector Machines

  • Sebastian Risau-Gusman
  • Mirta B. Gordon


We study the typical properties of polynomial Support Vector Machines within a Statistical Mechanics approach that takes into account the number of high order features relative to the input space dimension. We analyze the effect of different features' normalizations on the generalization error, for different kinds of learning tasks. If the normalization is adequately selected, hierarchical learning of features of increasing order takes place as a function of the training set size. Otherwise, the performance worsens, and there is no hierarchical learning at all.

learning theory support vector machines 


  1. Buhot, A. & Gordon, M. B. (1999). Learning properties of support vector machines. In M. Verleysen (Ed.), Proceedings of ESANN'99-European Symposium on Artificial Neural Networks, Brussels: D facto Publications.Google Scholar
  2. Buhot, A., Torres Moreno, J-M., & Gordon, M. B. (1997). Finite size scaling of the bayesian perceptron. Phys. Rev. E, 55, 7434-7440.Google Scholar
  3. Cortes C. & Vapnik, V. (1995). Support vector networks. Machine Learning, 20, 273-297.Google Scholar
  4. Dietrich, R., Opper, M., & Sompolinsky, H. (1999). Statistical mechanics of support vector networks. Phys. Rev. Lett., 82, 2975-2978.Google Scholar
  5. Gardner, E. & Derrida, B. (1988). Optimal storage properties of neural networks models. J. Phys. A, 21, 271-284.Google Scholar
  6. Gordon, M. B. & Grempel, D. R. (1995). Learning with a temperature-dependent algorithm. Europhys. Lett., 29, 257-262.Google Scholar
  7. Marangi, C., Biehl, M., & Solla, S. (1995). Supervised learning from clustered input examples. Europhys. Lett., 30, 117-122.Google Scholar
  8. Mzard, M., Parisi, G., & Virasoro, M. A. (1987). Spin glasses and beyond. Singapore: World Scientific.Google Scholar
  9. Monasson, R. (1993). Storage of spatially correlated patterns in autoassociative memories. J. Phys. A, 25, 1141-1152.Google Scholar
  10. Nadler, W. & Fink, W. (1997). Finite size scaling in neural networks. Phys. Rev. Lett., 78, 555-558.Google Scholar
  11. Opper, M. & Kinzel, W. (1995). Statistical mechanics of generalization. In E. Domany & J. L. van Hemmen (Eds.), Models of neural networks III. Physics of Neural Networks Series, Berlin: Springer-Verlag.Google Scholar
  12. Opper, M., Kinzel, W., Kleinz, J., & Nehl, R. (1990). On the ability of the optimal perceptron to generalise. J. Phys. A: Math. Gen., 23, L-581-L-586.Google Scholar
  13. Risau-Gusman, S. & Gordon, M. B. (2000). Understanding stepwise generalization of support vector machines: A toy model. In S. A. Solla, T. K. Leen, & K.-R. Müller (Eds.), Advances in neural information processing systems 12, MIT Press.Google Scholar
  14. Schroder, M. & Urbanczik, R. (1998). Comment on “finite size scaling in neural networks. ” Phys. Rev. Lett., 80, 4109.Google Scholar
  15. Seung, H. S., Sompolinsky, H. S., & Tishby, N. (1992). Statistical mechanics of learning from examples. Phys. Rev. A, 45, 6056-6090.Google Scholar
  16. Smola, A. (1999). Program available upon request to Scholar
  17. Talagrand, M. (1998). Self averaging and the space of interactions in neural networks. Random Structures and Algorithms, 14, 199-213.Google Scholar
  18. Vanderbei, R. J. (1994). Loqo, an interior point code for quadratic programming. Technical Report SOR-94-15, Princeton University.Google Scholar
  19. Vapnik, V. (1995). The nature of statistical learning theory. New York: Springer Verlag.Google Scholar
  20. Vapnik, V. & Chapelle, O. (2000). Bounds on error expectation for support vector machines. Neural Computation, 12, 2013-2036.Google Scholar
  21. Watkin, T. L. H., Rau, A., & Biehl, M. (1993). The statistical mechanics of learning a rule. Rev. Mod. Phys., 65, 499-555.Google Scholar
  22. Yoon, H. & Oh, J.-H. (1998). Learning of higher order perceptrons with tunable complexities. J. Phys. A: Math. Gen., 31, 7771-7784.Google Scholar

Copyright information

© Kluwer Academic Publishers 2002

Authors and Affiliations

  • Sebastian Risau-Gusman
    • 1
  • Mirta B. Gordon
    • 1
  1. 1.DRFMC/SPSMS CEA GrenobleGrenoble Cedex 09France

Personalised recommendations