Machine Learning

, Volume 24, Issue 2, pp 123–140 | Cite as

Bagging predictors

  • Leo Breiman
Article

Abstract

Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.

Keywords

Aggregation Bootstrap Averaging Combining 

References

  1. Belsley, D., Kuh, E., & Welsch, R. (1980). “Regression Diagnostics”, John Wiley and Sons.Google Scholar
  2. Breiman, L. (1994). Heuristics of instability in model selection, Technical Report, Statistics Department, University of California at Berkeley (to appear, Annals of Statistics).Google Scholar
  3. Breiman, L., Friedman, J., Olshen, R., & Stone, C. (1984). “Classification and Regression Trees”, Wadsworth.Google Scholar
  4. Breiman, L. & Friedman, J. (1985). Estimating optimal transformations in multiple regression and correlation (with discussion), Journal of the American Statistical Association, 80, 580–619.Google Scholar
  5. Breiman, L. & Spector, P (1992). Submodel Selection and Evaluation in Regression-the X-Random Case, International Statistical Review, 3, 291–319Google Scholar
  6. Buntine, W. (1991). “Learning classification trees”, Artificial Intelligence Frontiers in Statistics, ed D.J. Hand, Chapman and Hall, London, 182–201.Google Scholar
  7. Dietterich, T.G. & Bakiri, G. (1991). Error-correcting output codes: A general method for improving multiclass inductive learning programs, Proceedings of the Ninth National Conference on Artificial Intelligence (AAAI-91), Anaheim, CA: AAAI Press.Google Scholar
  8. Efron, B., & Tibshirani, R. (1993). “An Introduction to the Bootstrap”. Chapman and Hall.Google Scholar
  9. Friedman, J. (1991). Multivariate adaptive regression splines (with discussion), Annals of Statistics, 19, 1–141.Google Scholar
  10. Heath, D., Kasif, S., & Salzberg, S. (1993). k-dt: a multi-tree learning method. Proceedings of the Second International Workshop on Multistrategy Learning, 1002–1007, Chambery, France, Morgan Kaufman.Google Scholar
  11. Kwok, S., & Carter, C. (1990). Multiple decision trees, Uncertainty in Artificial Intelligence 4, ed. Shachter, R., Levitt, T., Kanal, L., and Lemmer, J., North-Holland, 327–335.Google Scholar
  12. Michie, D., Spiegelhalter, D.J. & Taylor, C.C. (1994). Machine Learning, Neural and Statistical Classification. Ellis Horwood Limited.Google Scholar
  13. Olshen, R., Gilpin, A., Henning, H., LeWinter, M., Collins, D., & Ross, J. (1985). Twelve-month prognosis following myocardial infarction: Classification trees, logistic regression, and stepwise linear discrimination, Proceedings of the Berkeley conference in honor of Jerzy Neyman and Jack Kiefer, L. Le Cam;R. Olshen, (Ed), Wadsworth, 245–267.Google Scholar
  14. Smith, J., Everhart, J., Dickson, W., Knowler, W., & Johannes, R. (1988). Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Symposium on Computer Applications and Medical Care 261–265. IEEE Computer Society Press.Google Scholar
  15. Sigillito, V. G., Wing, S. P., Hutton, L. V., & Baker, K. B. (1989). Classification of radar returns from the ionosphere using neural networks. Johns Hopkins APL Technical Digest, 10, 262–266.Google Scholar
  16. Wolberg, W. & Mangasarian, O (1990). Multisurface method of pattern separation for medical diagnosis applied to breast cytology, Proceedings of the National Academy of Sciences, U.S.A., Volume 87, December 1990, pp 9193–9196.Google Scholar

Copyright information

© Kluwer Academic Publishers 1996

Authors and Affiliations

  • Leo Breiman
    • 1
  1. 1.Statistics DepartmentUniversity of CaliforniaBerkeley

Personalised recommendations