Smooth Boosting Using an Information-Based Criterion

  • Kohei Hatano
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4264)


Smooth boosting algorithms are variants of boosting methods which handle only smooth distributions on the data. They are proved to be noise-tolerant and can be used in the “boosting by filtering” scheme, which is suitable for learning over huge data. However, current smooth boosting algorithms have rooms for improvements: Among non-smooth boosting algorithms, real AdaBoost or InfoBoost, can perform more efficiently than typical boosting algorithms by using an information-based criterion for choosing hypotheses. In this paper, we propose a new smooth boosting algorithm with another information-based criterion based on Gini index. we show that it inherits the advantages of two approaches, smooth boosting and information-based approaches.


Loss Function Gini Index Training Error Huge Data Weak Hypothesis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aslam, J.A.: Improving algorithms for boosting. In: Proc. 13th Annu. Conference on Comput. Learning Theory, pp. 200–207 (2000)Google Scholar
  2. 2.
    Balcazar, J.L., Dai, Y., Watanabe, O.: Provably fast training algorithms for support vector machines. In: Proceedings of IEEE International Conference on Data Mining (ICDM 2001), pp. 43–50 (2001)Google Scholar
  3. 3.
    Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. Wadsworth International Group (1984)Google Scholar
  4. 4.
    Dasgupta, S., Long, P.M.: Boosting with diverse base classifiers. In: Schölkopf, B., Warmuth, M.K. (eds.) COLT/Kernel 2003. LNCS, vol. 2777, pp. 273–287. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  5. 5.
    Domingo, C., Gavaldà, R., Watanabe, O.: Adaptive sampling methods for scaling up knowledge discovery algorithms. Data Mining and Knowledge Discovery 6(2), 131–152 (2002)MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Domingo, C., Watanabe, O.: MadaBoost: A modification of AdaBoost. In: Proceedings of 13th Annual Conference on Computational Learning Theory, pp. 180–189 (2000)Google Scholar
  7. 7.
    Domingos, P., Hulten, G.: Mining high-speed data streams. In: Terano, T., Chen, A.L.P. (eds.) PAKDD 2000. LNCS, vol. 1805. Springer, Heidelberg (2000)Google Scholar
  8. 8.
    Feller, W.: An introduction to probability theory and its applications. Wiley, Chichester (1950)MATHGoogle Scholar
  9. 9.
    Freund, Y.: An improved boosting algorithm and its implications on learning complexity. In: Proc. 5th Annual ACM Workshop on Computational Learning Theory, pp. 391–398. ACM Press, New York (1992)CrossRefGoogle Scholar
  10. 10.
    Freund, Y.: Boosting a weak learning algorithm by majority. Information and Computation 121(2), 256–285 (1995)MATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55(1), 119–139 (1997)MATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Freund, Y.: An adaptive version of the boost by majority algorithm. In: COLT 1999: Proceedings of the twelfth annual conference on Computational learning theory, pp. 102–113 (1999)Google Scholar
  13. 13.
    Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Annals of Statisitics 2, 337–374 (2000)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Gavinsky, D.: Optimally-smooth adaptive boosting and application to agnostic learning. Journal of Machine Learning Research (2003)Google Scholar
  15. 15.
    Hatano, K., Warmuth, M.K.: Boosting versus covering. In: Advances in Neural Information Processing Systems 16 (2003)Google Scholar
  16. 16.
    Hatano, K., Watanabe, O.: Learning r-of-k functions by boosting. In: Ben-David, S., Case, J., Maruoka, A. (eds.) ALT 2004. LNCS, vol. 3244, pp. 114–126. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  17. 17.
    Hatano, K.: Smooth boosting using an information-based criterion. Technical Report DOI-TR-225, Department of Informatics, Kyushu University (2006)Google Scholar
  18. 18.
    Kearns, M., Mansour, Y.: On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and System Sciences 58(1), 109–128 (1999)MATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Mansour, Y., McAllester, D.A.: Boosting using branching programs. Journal of Computer and System Sciences 64(1), 103–112 (2002)MATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Schapire, R.E., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the margin: a new explanation for the effectiveness of voting methods. The Annals of Statistics 26(5), 1651–1686 (1998)MATHCrossRefMathSciNetGoogle Scholar
  21. 21.
    Schapire, R.E.: The strength of weak learnability. Machine Learning 5(2), 197–227 (1990)Google Scholar
  22. 22.
    Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. Machine Learning 37(3), 297–336 (1999)MATHCrossRefGoogle Scholar
  23. 23.
    Scheffer, T., Wrobel, S.: Finding the most interesting patterns in a database quickly by using sequential sampling. Journal of Machine Learning Research 3, 833–862 (2003)MATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    Serfling, R.J.: Approximation theorems of mathematical statistics. Wiley, Chichester (1980)MATHCrossRefGoogle Scholar
  25. 25.
    Servedio, R.A.: Smooth boosting and learning with malicious noise. In: Helmbold, D.P., Williamson, B. (eds.) COLT 2001 and EuroCOLT 2001. LNCS, vol. 2111, pp. 473–489. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  26. 26.
    Takimoto, E., Koya, S., Maruoka, A.: Boosting based on divide and merge. In: Ben-David, S., Case, J., Maruoka, A. (eds.) ALT 2004. LNCS, vol. 3244, pp. 127–141. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  27. 27.
    Valiant, L.G.: A theory of the learnable. Communications of the ACM 27(11), 1134–1142 (1984)MATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Kohei Hatano
    • 1
  1. 1.Department of InformaticsKyushu University 

Personalised recommendations