Artificial Intelligence Review

, Volume 35, Issue 3, pp 223–240 | Cite as

Combining bagging, boosting, rotation forest and random subspace methods

Article

Abstract

Bagging, boosting, rotation forest and random subspace methods are well known re-sampling ensemble methods that generate and combine a diversity of learners using the same learning algorithm for the base-classifiers. Boosting and rotation forest algorithms are considered stronger than bagging and random subspace methods on noise-free data. However, there are strong empirical indications that bagging and random subspace methods are much more robust than boosting and rotation forest in noisy settings. For this reason, in this work we built an ensemble of bagging, boosting, rotation forest and random subspace methods ensembles with 6 sub-classifiers in each one and then a voting methodology is used for the final prediction. We performed a comparison with simple bagging, boosting, rotation forest and random subspace methods ensembles with 25 sub-classifiers, as well as other well known combining methods, on standard benchmark datasets and the proposed technique had better accuracy in most cases.

Keywords

Data mining Machine learning Pattern recognition Ensembles of classifiers 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bauer E, Kohavi R (1999) An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach Learn 36: 105–139CrossRefGoogle Scholar
  2. Bosch A, Daelemans W (1999) Memory-based morphological analysis. In: Proceedings of 37th annual meeting of the ACL. University of Maryland, pp 285–292 (http://ilk.kub.nl/~antalb/ltuia/week10.html)
  3. Breiman L (1996) Bagging predictors. Mach Learn 24(3): 123–140MathSciNetMATHGoogle Scholar
  4. Breiman L (2001) Random forests. Mach Learn 45(1): 5–32MATHCrossRefGoogle Scholar
  5. Bryll R, Gutierrez-Osuna R, Quek F (2003) Attribute bagging: improving accuracy of classifier ensemble by using random feature subsets. Pattern Recognit 36: 1291–1302MATHCrossRefGoogle Scholar
  6. Bühlman P, Yu B (2002) Analyzing bagging. Ann Stat 30: 927–961CrossRefGoogle Scholar
  7. Buja WS (2006) Observations on bagging. Statistica Sinica 16: 323–351MathSciNetMATHGoogle Scholar
  8. Cai Q-T, Peng C-Y, Zhang C-S (2008) A weighted subspace approach for improving bagging performance. In: IEEE ICASSP, pp 3341–3344Google Scholar
  9. Dietterich T (2000) An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Mach Learn 40: 139–157CrossRefGoogle Scholar
  10. Dietterich TG (2001) Ensemble methods in machine learning. In: Kittler J, Roli F (eds) Multiple classifier systems. LNCS, vol 1857, pp 1–15Google Scholar
  11. Domingos P, Pazzani M (1997) On the optimality of the simple Bayesian classifier under zero-one loss. Mach Learn 29: 103–130MATHCrossRefGoogle Scholar
  12. Frank E, Witten IH (1998) Generating Accurate Rule Sets Without Global Optimization. In: Fifteenth international conference on machine learning, pp 144–151Google Scholar
  13. Frank A, Asuncion A (2010) UCI machine learning repository [http://www.archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science
  14. Freidman JH, Hall P (2007) On bagging and nonlinear estimation. J Stat Plan Inference 137(3): 669–683CrossRefGoogle Scholar
  15. Freidman J, Hastie T, Tibshirani R (2000) Additive logistic regression: a statistical view of boosting. Ann Stat 28: 307–337Google Scholar
  16. Freund Y, Robert E Schapire (1996) Experiments with a new boosting algorithm. In: Proceedings of ICML’96, pp 148–156Google Scholar
  17. Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1): 119–139MathSciNetMATHCrossRefGoogle Scholar
  18. Fumera G, Roli F, Serrau A (2005) Dynamics of variance reduction in bagging and other techniques based on randomisation. MCS 2005, LNCS vol 3541, pp 316–325Google Scholar
  19. Fumera G, Roli F, Serrau A (2008) A theoretical analysis of bagging as a linear combination of classifiers. IEEE Trans Pattern Anal Mach Intell 30(7):1293–1299CrossRefGoogle Scholar
  20. Garcıa-Pedrajas N, Ortiz-Boyer D (2008) Boosting random subspace method. Neural Netw 21: 1344–1362CrossRefGoogle Scholar
  21. Grandvalet Y (2004) Bagging equalizes influence. Mach Learn 55: 251–270MATHCrossRefGoogle Scholar
  22. Ho TK (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8): 832–844CrossRefGoogle Scholar
  23. Kotsiantis S, Pierrakeas C, Pintelas P (2003) Preventing student dropout in distance learning systems using machine learning techniques. In: Proceedings of 7th international conference on knowledge-based intelligent information and engineering systems (KES), Oxford, Sept. 3–5, Lecture notes series, vol 2774. Springer, pp 267–274Google Scholar
  24. Kuncheva LI, Skurichina M, Duin RPW (2002) An experimental study on diversity for bagging and boosting with linear classifiers. Inf Fusion 3: 245–258CrossRefGoogle Scholar
  25. Latinne P, Debeir O, Decaestecker Ch (2000) Mixing bagging and multiple feature subsets to improve classification accuracy of decision tree combination. In: Proceedings of the 10th Belgian-Dutch conference on machine learning, Tilburg UniversityGoogle Scholar
  26. Melville P, Mooney R (2003) Constructing diverse classifier ensembles using artificial training examples. In: Proceedings of IJCAI-2003, pp 505–510, Acapulco, Mexico, August 2003Google Scholar
  27. Murthy RPW (1998) Automatic construction of decision trees from data: a multi-disciplinary survey. Data Min Knowl Discov 2: 345–389MathSciNetCrossRefGoogle Scholar
  28. Opitz D, Maclin R (1999) Popular ensemble methods: an empirical study. Artif Intell Res 11: 169–198MATHGoogle Scholar
  29. Panov P, Dzeroski S (2007) Combining bagging and random subspaces to create better ensembles, IDA 2007, LNCS 4723, pp 118–129Google Scholar
  30. Redpath DB, Lebart K (2005) Boosting feature selection. In: The third international conference on advances in pattern recognition, Bath, UK, Springer, Berlin, pp 305–314Google Scholar
  31. Rodríguez JJ, Kuncheva LI, Alonso CJ (2006) Rotation forest: a new classifier ensemble method. IEEE Trans Pattern Anal Mach Intell 28(10): 1619–1630CrossRefGoogle Scholar
  32. Salzberg S (1997) On comparing classifiers: pitfalls to avoid and a recommended approach. Data Min Knowl Discov 1: 317–328CrossRefGoogle Scholar
  33. Schapire RE, Freund Y, Bartlett P, Lee WS (1998) Boosting the margin: a new explanation for the effectiveness of voting methods. Ann Stat 26: 1651–1686MathSciNetMATHCrossRefGoogle Scholar
  34. Schapire RE, Singer Y (1999) Improved boosting algorithms using confidence-rated predictions. Mach Learn 37: 297–336MATHCrossRefGoogle Scholar
  35. Shirai S, Kudo M, Nakamura A (2008) Bagging, random subspace method and biding. SSPR&SPR 2008, LNCS vol 5342, pp 801–810Google Scholar
  36. Stefanowski J (2007) Combining answers of sub-classifiers in the bagging-feature ensembles, RSEISP 2007, LNAI vol 4585, pp 574–583Google Scholar
  37. Ting KM, Witten IH (1997) Stacking bagged and dagged models. In: Fourteenth international conference on machine learning, San Francisco, CA, pp 367–375Google Scholar
  38. Webb GI (2000) MultiBoosting: a technique for combining boosting and wagging. Mach Learn 40: 159–196CrossRefGoogle Scholar
  39. Yin X-C, Liu C-P, Zhi H (2005) Feature combination using boosting. Pattern Recognit Lett 26: 2195–2205CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of PatrasPatrasGreece

Personalised recommendations