A Comparison of Model Aggregation Methods for Regression
Combining machine learning models is a means of improving overall accuracy. Various algorithms have been proposed to create aggregate models from other models, and two popular examples for classification are Bagging and AdaBoost. In this paper we examine their adaptation to regression, and benchmark them on synthetic and real-world data. Our experiments reveal that different types of AdaBoost algorithms require different complexities of base models. They outperform Bagging at their best, but Bagging achieves a consistent level of success with all base models, providing a robust alternative.
KeywordsLoss Function Base Learner Training Error Weighted Median AdaBoost Algorithm
Unable to display preview. Download preview PDF.
- 2.Freund, Y. and R. E. Schapire, “A Decision-Theoretic Generalization of On-line Learning and an Application to Boosting”, European Conf. on Computational Learning Theory, pp. 23–37, 1995.Google Scholar
- 3.Freund, Y. and R. E. Schapire, “Experiments with a New Boosting Algorithm”, International Conf. on Machine Learning, pp. 148–156, 1996.Google Scholar
- 4.Ridgeway, G., D. Madigan and T. Richardson, “Boosting methodology for regression problems”, Proc. of Artificial Intelligence and Statistics, pp. 152–161, 1999.Google Scholar
- 5.Drucker, H., “Improving regressors using boosting techniques”, Proc. 14th International Conf. on Machine Learning, pp. 107–115, Morgan Kaufmann, San Francisco, CA, 1997.Google Scholar
- 6.Zemel, R. S. and T. Pitassi, “A Gradient-Based Boosting Algorithm for Regression Problems”, Adv. in Neural Information Processing Systems, Vol. 13, 2001.Google Scholar
- 7.Friedman, J. H., Greedy Function Approximation: a Gradient Boosting Machine, Tech. Rep. 7, Stanford University, Dept. of Statistics, 1999.Google Scholar
- 8.Duffy, N. and D. Helmbold, “Leveraging for Regression”, Proc. 13th Annual Conf. on Computational Learning Theory, pp. 208–219, Morgan Kaufmann, San Francisco, CA, 2000.Google Scholar
- 9.Rätsch, G., M. Warmuth, S. Mika, T. Onoda, S. Lemm and K.-R. Müller, “Barrier Boosting”, Proc. 13th Annual Conference on Computational Learning Theory, 2000.Google Scholar
- 11.Blake, C. and P. M. Murphy, “UCI Repository of Machine Learning Databases”, http://www.ics.uci.edu/&~mlearn/MLRepository.html.Google Scholar
- 12.Hosmer, D. and S. Lemeshow, Applied Logistic Regression, John Wiley & Sons Inc., 2nd edn., 2000.Google Scholar