Neural Processing Letters

, Volume 21, Issue 3, pp 175–188

An Incremental Learning Strategy for Support Vector Regression



Support vector machine (SVM) provides good generalization performance but suffers from a large amount of computation. This paper presents an incremental learning strategy for support vector regression (SVR). The new method firstly formulates an explicit expression of ||W||2 by constructing an orthogonal basis in feature space together with a basic Hilbert space identity, and then finds the regression function through minimizing the formula of ||W||2 rather than solving a convex programming problem. Particularly, we combine the minimization of ||W||2 with kernel selection that can lead to good generalization performance. The presented method not only provides a novel way for incremental SVR learning, but opens an opportunity for model selection of SVR as well. An artificial data set, a benchmark data set and a real-world data set are employed to evaluate the method. The simulations support the feasibility and effectiveness of the proposed approach.


incremental learning kernel selection regression support vector machine 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barzilay, O., Brailovsky, V. 1999On domain knowledge and feature selection using a support vector machinePattern Recognition Letters20475484Google Scholar
  2. 2.
    Blake, C. L. and Merz, C. J.: UCI Repository of machine learning databases, University of California, Department of Information and Computer Science. 1998. http://www.ics.uci.deu/~mlearn/MLRRepository.htmlGoogle Scholar
  3. 3.
    Brown, M., Grundy, W., Lin, D., Cristianini, N., Sugnet, C., Furey, T., Ares, M. and Haussler, D.: Knowledge-based analysis of microarray gene expression data by using support vector machines, In: Proceedings of The National Academy of Sciences of The United States of America, 97: 262–267, (2000)Google Scholar
  4. 4.
    Campbell, C., Cristianini, N. and Smola, A.: Query learning with large margin classifiers, In: Proceedings 17th International Conference on Machine Learning. 111–118, San Francisco, CA: Morgan Kaufmann, (2000)Google Scholar
  5. 5.
    Cauwenberghs, G. and Poggio, T.: Incremental and decremental support vector machine learning. Advances in Neural Information Processing Systems, 13: MIT Press, (2001)Google Scholar
  6. 6.
    Chapelle, O., Vapnik, V. 2000

    Model selection for support vector machines

    Smola, A.Leen, T.Müller, K. eds. Advances in Neural Information Processing Systems 12MIT PressCambridge, MA230236
    Google Scholar
  7. 7.
    Cristianini, N. and Shawe-Taylor, J.: An Introduction to Support Vector Machines, Cambridge University Press, 2000Google Scholar
  8. 8.
    Drucker, H., Wu, D., Vapnik, V. 1999Support vector machines for span categorizationIEEE Trans. Neural Networks1010481054Google Scholar
  9. 9.
    Fine, S. and Scheinberg, K.: Incremental Learning and Selective Sampling via Parametric Optimization Framework for SVM. In Machine Learning, Kluwer Academic, 2002.Google Scholar
  10. 10.
    Frieß, T.-T. Chistianini, N. and Campbell, C.: The kernel adatron algorithm: A fast and simple learning procedure for support vector machines. In: J. Shavlik (ed.), Proceedings of the The 15th International Conference of Machine Learning. San Francisco, 188–196, CA: Morgan Kaufmann (1998)Google Scholar
  11. 11.
    Gunn, S. R.: Support vector machines for classification and regression, Technical Report, Image Speech and Intelligent System Research Group, University of Southampton, 1997Google Scholar
  12. 12.
    Joachims, T.: Text categorization with support vector machines: Learning with many relevant features, In: Proceedings of the European Conference on Machine Learning, 137–142, Berlin: Springer (1998)Google Scholar
  13. 13.
    Joachims, T. 1998Make large-scale support vector machine learning practical. Advances in kernel methods: support vector machinesMIT PressCambridge, MA169184Google Scholar
  14. 14.
    Martin, M.: On-line support vector machines for function approximation, (2002) Scholar
  15. 15.
    Osuna, E. Freund, R. and Girosi, F.: An improved training algorithm for support vector machines. Proceedings of the IEEE Workshop on Neural Networks for Signal Processing, 276–285, (1997)Google Scholar
  16. 16.
    Platt, J.C. 1998Fast training of support vector machines using sequential minimum optimization, Advances in kernel methods: support vector machinesMIT PressCambridge, MA185208Google Scholar
  17. 17.
    Poggio, T., Girosi, F. 1990Networks for approximation and learningProceedings on IEEE7814811497Google Scholar
  18. 18.
    Vapnik, V. 1995The Nature of Statistical Learning TheoryJohn Wiley & SonsNew YorkGoogle Scholar
  19. 19.
    Vapnik, V. 1998Statistical Learning TheoryJohn Wiley & SonsNew YorkGoogle Scholar
  20. 20.
    Vapnik, V., Golowich, S., Smola, A. 1997

    Support Vector Method for Function Approximation, Regression Estimation, and Signal Processing

    Mozer, M.Jordan, M.Petsche, T. eds. Neural Information Processing SystemsMIT PressCambridge, MA
    Google Scholar
  21. 21.
    Wang, W. J., Xu, Z. B., Lu, W. Z. 2003Three improved neural network models for air quality forecastingEngineering Computations20192210Google Scholar

Copyright information

© Springer 2005

Authors and Affiliations

  1. 1.Institute of System Engineering, Faculty of Computer and Information TechnologyShanxi UniversityTaiyuanPeople’s Republic of China

Personalised recommendations