MCS 2003: Multiple Classifier Systems pp 15-24 | Cite as
Boosting with Averaged Weight Vectors
Abstract
AdaBoost [5] is a well-known ensemble learning algorithm that constructs its constituent or base models in sequence. A key step in AdaBoost is constructing a distribution over the training examples to create each base model. This distribution, represented as a vector, is constructed to be orthogonal to the vector of mistakes made by the previous base model in the sequence [7]. The idea is to make the next base model’s errors uncorrelated with those of the previous model. Some researchers have pointed out the intuition that it is probably better to construct a distribution orthogonal to the mistake vectors of all the previous base models, but that this is not always possible [7]. We present an algorithm that attempts to come as close as possible to this goal in an efficient manner. We present experimental results demonstrating significant improvement over AdaBoost and the Totally Corrective boosting algorithm [7], which also attempts to satisfy this goal.
Keywords
Neural Network Ensemble Computational Learn Theory Decision Stump Base Train Average Weight VectorPreview
Unable to display preview. Download preview PDF.
References
- 1.Eric Bauer and Ron Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning, 36:105–139, Sep. 1999.CrossRefGoogle Scholar
- 2.C. Blake, E. Keogh, and C.J. Merz. UCI repository of machine learning databases, 1999. (URL: http://www.ics.uci.edu/~mlearn/MLRepository.html).Google Scholar
- 3.Y. Censor and A. Lent. An iterative row-action method for interval convex programming. Journal of Optimization Theory and Applications, 34(3):321–353, 1981.MATHCrossRefMathSciNetGoogle Scholar
- 4.Thomas G. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine Learning, 40:139–158, Aug. 2000.CrossRefGoogle Scholar
- 5.Y. Freund and R. Schapire. Experiments with a new boosting algorithm. In Proceedings of the Thirteenth International Conference on Machine Learning, pages 148–156, Bari, Italy, 1996. Morgan Kaufmann.Google Scholar
- 6.Michael J. Kearns and Umesh V. Vazirani. Introduction to Computational Learning Theory. MIT Press, Cambridge, MA, 1994.Google Scholar
- 7.Jyrki Kivinen and Manfred K. Warmuth. Boosting as entropy projection. In Proceedings of the Twelfth Annual Conference on Computational Learning Theory, pages 134–144, 1999.Google Scholar
- 8.A. Krogh and J. Vedelsby. Neural network ensembles, cross validation and active learning. In G. Tesauro, D. S. Touretzky, and T. K. Leen, editors, Advances in Neural Information Processing Systems-7, pages 231–238. M.I.T. Press, 1995.Google Scholar
- 9.Samuel Kutin and Partha Niyogi. The interaction of stability and weakness in adaboost. Technical Report TR-2001-30, University of Chicago, October 2001.Google Scholar
- 10.Nikunj C. Oza. Online Ensemble Learning. PhD thesis, The University of California, Berkeley, CA, Dec 2001.Google Scholar
- 11.K. Tumer and J. Ghosh. Analysis of decision boundaries in linearly combined neural classifiers. Pattern Recognition, 29(2):341–348, February 1996.CrossRefGoogle Scholar