Selective Ensemble of Decision Trees

  • Zhi-Hua Zhou
  • Wei Tang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2639)


An ensemble is generated by training multiple component learners for a same task and then combining their predictions. In most ensemble algorithms, all the trained component learners are employed in constituting an ensemble. But recently, it has been shown that when the learners are neural networks, it may be better to ensemble some instead of all of the learners. In this paper, this claim is generalized to situations where the component learners are decision trees. Experiments show that ensembles generated by a selective ensemble algorithm, which selects some of the trained C4.5 decision trees to make up an ensemble, may be not only smaller in the size but also stronger in the generalization than ensembles generated by non-selective algorithms.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Asker L., Maclin R.: Ensembles as a sequence of classifiers. In: Proceedings of the 15th International Joint Conference on Artificial Intelligence (1997) 860–865.Google Scholar
  2. 2.
    Bauer E., Kohavi R.: An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Machine Learning 36 (1999) 105–139.CrossRefGoogle Scholar
  3. 3.
    Blake C., Keogh E., Merz C. J.: UCI repository of machine learning databases []. Department of Information and Computer Science, University of California, Irvine, CA, 1998.Google Scholar
  4. 4.
    Breiman L.: Bagging predictors. Machine Learning 24 (1996) 123–140.zbMATHMathSciNetGoogle Scholar
  5. 5.
    Breiman L.: Arcing classifiers. Annals of Statistics 26 (1998) 801–849.zbMATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Cherkauer K. J.: Human expert level performance on a scientific image analysis task by a system using combined artificial neural networks. In: Proceedings of the AAAI-96 Workshop on Integrating Multiple Models for Improving and Scaling Machine Learning Algorithms (1996) 15–21.Google Scholar
  7. 7.
    Cunningham P., Carney J., Jacob S.: Stability problems with artificial neural networks and the ensemble solution. Artificial Intelligence in Medicine 20 (2000) 217–225.CrossRefGoogle Scholar
  8. 8.
    Dietterich T. G.: An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Machine Learning 40 (2000) 139–157.CrossRefGoogle Scholar
  9. 9.
    Drucker H., Schapire R., Simard P.: Improving performance in neural networks using a boosting algorithm. In: Hanson S. J., Cowan J. D., Giles C. L. (eds.): Advances in Neural Information Processing Systems 5, Morgan Kaufmann, San Mateo, CA (1993) 42–49.Google Scholar
  10. 10.
    Efron B., Tibshirani R.: An Introduction to the Bootstrap. Chapman & Hall, New York (1993).zbMATHGoogle Scholar
  11. 11.
    Freund Y., Schapire R. E.: A decision-theoretic generalization of on-line learning and an application to boosting. In: Proceedings of the 2nd European Conference on Computational Learning Theory (1995) 23–37.Google Scholar
  12. 12.
    Goldberg D. E.: Genetic Algorithm in Search, Optimization and Machine Learning. Addison-Wesley, Reading (1989).Google Scholar
  13. 13.
    Gutta S., Wechsler H.: Face recognition using hybrid classifier systems. In: Proceedings of the International Conference on Neural Networks (1996) 1017–1022.Google Scholar
  14. 14.
    Harries M.: Boosting a strong learner: evidence against the minimum margin. In: Proceedings of the 16th International Conference on Machine Learning (1999) 171–179.Google Scholar
  15. 15.
    Hu X.: Using rough sets theory and database operations to construct a good ensemble of classifiers for data mining applications. In: Proceedings of the IEEE International Conference on Data Mining (2001) 233–240.Google Scholar
  16. 16.
    Huang F. J., Zhou Z.-H., Zhang H.-J., Chen T. H.: Pose invariant face recognition. In: Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition (2000) 245–250.Google Scholar
  17. 17.
    Mao J.: A case study on bagging, boosting and basic ensembles of neural networks for OCR. In: Proceedings of the International Joint Conference on Neural Networks (1998) 1828–1833.Google Scholar
  18. 18.
    Margineantu D., Dietterich T. G.: Pruning adaptive boosting. In: Proceedings of the 14th International Conference on Machine Learning (1997) 211–218.Google Scholar
  19. 19.
    Opitz D., Maclin R.: Popular ensemble methods: an empirical study. Journal of Artificial Intelligence Research 11 (1999) 169–198.zbMATHGoogle Scholar
  20. 20.
    Quinlan J. R.: Bagging, boosting, and C4.5. In: Proceedings of the 13th National Conference on Artificial Intelligence (1996) 725–730.Google Scholar
  21. 21.
    Quinlan J. R.: Miniboosting decision trees.
  22. 22.
    Tamon C., Xiang J.: On the boosting pruning problem. In: Proceedings of the 11th European Conference on Machine Learning (2000) 404–412.Google Scholar
  23. 23.
    Ting K. M., Witten I. H.: Issues in stacked generalization. Journal of Artificial Intelligence Research 10 (1999) 271–289.zbMATHGoogle Scholar
  24. 24.
    Webb G. I.: MultiBoosting: a technique for combining boosting and wagging. Machine Learning 40 (2000) 159–196.CrossRefMathSciNetGoogle Scholar
  25. 25.
    Wolpert D.: Stacked generalization. Neural Networks 5 (1992) 241–259.CrossRefGoogle Scholar
  26. 26.
    Zhou Z.-H., Jiang Y., Yang Y.-B., Chen S.-F.: Lung cancer cell identification based on artificial neural network ensembles. Artificial Intelligence in Medicine 24 (2002) 25–36.zbMATHCrossRefGoogle Scholar
  27. 27.
    Zhou Z.-H., Wu J., Tang W.: Ensembling neural networks: many could be better than all. Artificial Intelligence 137 (2002) 239–263.zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Zhi-Hua Zhou
    • 1
  • Wei Tang
    • 1
  1. 1.National Laboratory for Novel Software TechnologyNanjing UniversityNanjingChina

Personalised recommendations