Infinite Ensemble Learning with Support Vector Machines

  • Hsuan-Tien Lin
  • Ling Li
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3720)


Ensemble learning algorithms such as boosting can achieve better performance by averaging over the predictions of base hypotheses. However, existing algorithms are limited to combining only a finite number of hypotheses, and the generated ensemble is usually sparse. It is not clear whether we should construct an ensemble classifier with a larger or even infinite number of hypotheses. In addition, constructing an infinite ensemble itself is a challenging task. In this paper, we formulate an infinite ensemble learning framework based on SVM. The framework can output an infinite and nonsparse ensemble, and can be used to construct new kernels for SVM as well as to interpret some existing ones. We demonstrate the framework with a concrete application, the stump kernel, which embodies infinitely many decision stumps. The stump kernel is simple, yet powerful. Experimental results show that SVM with the stump kernel is usually superior than boosting, even with noisy data.


Support Vector Machine Gaussian Kernel Decision Boundary Ensemble Learning Support Vector Machine Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Machine Learning: Proceedings of the Thirteenth International Conference, pp. 148–156 (1996)Google Scholar
  2. 2.
    Rosset, S., Zhu, J., Hastie, T.: Boosting as a regularized path to a maximum margin classifier. Journal of Machine Learning Research 5, 941–973 (2004)MathSciNetGoogle Scholar
  3. 3.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55, 119–139 (1997)zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Vapnik, V.N.: Statistical Learning Theory. John Wiley & Sons, New York (1998)zbMATHGoogle Scholar
  5. 5.
    Lin, H.T.: Infinite ensemble learning with support vector machines. Master’s thesis, California Institute of Technology (2005)Google Scholar
  6. 6.
    Schölkopf, B., Smola, A.: Learning with Kernels. MIT Press, Cambridge (2002)Google Scholar
  7. 7.
    Rätsch, G., Onoda, T., Müller, K.: Soft margins for AdaBoost. Machine Learning 42, 287–320 (2001)zbMATHCrossRefGoogle Scholar
  8. 8.
    Demiriz, A., Bennett, K.P., Shawe-Taylor, J.: Linear programming boosting via column generation. Machine Learning 46, 225–254 (2002)zbMATHCrossRefGoogle Scholar
  9. 9.
    Meir, R., Rätsch, G.: An introduction to boosting and leveraging. In: Mendelson, S., Smola, A.J. (eds.) Advanced Lectures on Machine Learning. LNCS (LNAI), vol. 2600, pp. 118–183. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  10. 10.
    Freund, Y., Schapire, R.E.: A short introduction to boosting. Journal of Japanese Society for Artificial Intelligence 14, 771–780 (1999)Google Scholar
  11. 11.
    Reed, M., Simon, B.: Functional Analysis. In: Methods of Modern Mathematical Physics, Revised and enlarged edn. Academic Press, London (1980)Google Scholar
  12. 12.
    Holte, R.C.: Very simple classification rules perform well on most commonly used datasets. Machine Learning 11, 63–91 (1993)zbMATHCrossRefGoogle Scholar
  13. 13.
    Berg, C., Christensen, J.P.R., Ressel, P.: Harmonic Analysis on Semigroups: Theory of Positive Definite and Related Functions. Springer, New York (1984)zbMATHGoogle Scholar
  14. 14.
    Lin, H.T., Lin, C.J.: A study on sigmoid kernels for SVM and the training of non-PSD kernels by SMO-type methods. Technical report, National Taiwan University (2003)Google Scholar
  15. 15.
    Chang, C.C., Lin, C.J.: Training ν-support vector classifiers: Theory and algorithms. Neural Computation 13, 2119–2147 (2001)zbMATHCrossRefGoogle Scholar
  16. 16.
    Keerthi, S.S., Lin, C.J.: Asymptotic behaviors of support vector machines with Gaussian kernel. Neural Computation 15, 1667–1689 (2003)zbMATHCrossRefGoogle Scholar
  17. 17.
    Hsu, C.W., Chang, C.C., Lin, C.J.: A practical guide to support vector classification. Technical report, National Taiwan University (2003)Google Scholar
  18. 18.
    Chang, C.C., Lin, C.J.: LIBSVM: A library for support vector machines, Software (2001), available at
  19. 19.
    Breiman, L.: Prediction games and arcing algorithms. Neural Computation 11, 1493–1517 (1999)CrossRefGoogle Scholar
  20. 20.
    Hettich, S., Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998), Downloadable at

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Hsuan-Tien Lin
    • 1
  • Ling Li
    • 1
  1. 1.Learning Systems GroupCalifornia Institute of TechnologyUSA

Personalised recommendations