Bootstrap Based Pattern Selection for Support Vector Regression

  • Dongil Kim
  • Sungzoon Cho
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5012)


Support Vector Machine (SVM) results in a good generalization performance by employing the Structural Risk Minimization (SRM) principle. However, one drawback is O(n 3) training time complexity. In this paper, we propose a pattern selection method designed specifically for Support Vector Regression (SVR). In SVR training, only a few patterns called support vectors are used to construct the regression model while other patterns are not used at all. The proposed method tries to select patterns which are likely to become support vectors. With multiple bootstrap samples, we estimate the likelihood of each pattern to become a support vector. The proposed method automatically determines the appropriate number of patterns selected by estimating the expected number of support vectors. Through the experiments involving twenty datasets, the proposed method resulted in the best accuracy among the competing methods.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1995)zbMATHGoogle Scholar
  2. 2.
    Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge (2000)Google Scholar
  3. 3.
    Drucker, H., Burges, C.J.C., Kaufman, L., Smola, A., Vapnik, V.: Support Vector Regression Machines. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing System, vol. 9. MIT Press, Cambridge (1997)Google Scholar
  4. 4.
    Platt, J.C.: Fast Training of Support Vector Machines Using Sequential Minimal Optimization. In: Advanced in Kernel Methods; Support Vector Machines, pp. 185–208. MIT Press, Cambridge (1999)Google Scholar
  5. 5.
    Almeida, M.B., Braga, A., Braga, J.P.: SVM-KM: Speeding SVMs Learning with a Priori Cluster Selection and k-Means. In: Proc. of the 6th Brazilian Symposium on Neural Networks, pp. 162–167 (2000)Google Scholar
  6. 6.
    Shin, H., Cho, S.: Neighborhood Property based Pattern Selection for SVM. Neural Computation 19(3), 816–855 (2007)zbMATHCrossRefGoogle Scholar
  7. 7.
    Bakir, G.H., Bottou, L., Weston, J.: Breaking SVM Complexity with Cross-Training. In: Advances in Neural Information Processing Systems, vol. 17, pp. 81–88 (2005)Google Scholar
  8. 8.
    Joachims, T.: Training Linear SVMs in Linear Time. In: Proc. of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 217–226 (2006)Google Scholar
  9. 9.
    Wang, W., Xu, Z.: A Heuristic Training for Support Vector Regression. Neurocomputing 61, 259–275 (2004)CrossRefGoogle Scholar
  10. 10.
    Sun, J., Cho, S.: Pattern Selection for Support Vector Regression based on Sparsity and Variability. In: 2006 IEEE International Joint Conference on Neural Networks (IJCNN), pp. 559–602 (2006)Google Scholar
  11. 11.
    Kim, D., Cho, S.: ε–tube based Pattern Selection for Support Vector Machines. In: Ng, W.-K., Kitsuregawa, M., Li, J., Chang, K. (eds.) PAKDD 2006. LNCS (LNAI), vol. 3918, pp. 215–224. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  12. 12.
    Kim, D., Lee, H., Cho, S.: Response Modeling with Support Vector Regression. Expert Systems with Applications 34(2), 1102–1108 (2008)CrossRefMathSciNetGoogle Scholar
  13. 13.
    Chalimourda, A., Schölkopf, B., Smola, A.: Experimentally Optimal ν in Support Vector Regression for Different Noise Models and Parameter Settings. Neural Networks 17, 127–141 (2004)zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Dongil Kim
    • 1
  • Sungzoon Cho
    • 1
  1. 1.Seoul National UniversitySeoulKorea

Personalised recommendations