Using Diversity in Preparing Ensembles of Classifiers Based on Different Feature Subsets to Minimize Generalization Error

  • Gabriele Zenobi
  • Pádraig Cunningham
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2167)


It is well known that ensembles of predictors produce better accuracy than a single predictor provided there is diversity in the ensemble. This diversity manifests itself as disagreement or ambiguity among the ensemble members. In this paper we focus on ensembles of classifiers based on different feature subsets and we present a process for producing such ensembles that emphasizes diversity (ambiguity) in the ensemble members. This emphasis on diversity produces ensembles with low generalization errors from ensemble members with comparatively high generalization error. We compare this with ensembles produced focusing only on the error of the ensemble members (without regard to overall diversity) and find that the ensembles based on ambiguity have lower generalization error. Further, we find that the ensemble members produced focusing on ambiguity have less features on average that those based on error only. We suggest that this indicates that these ensemble members are local learners.


Feature Selection Ensemble Member Feature Subset Generalization Error Ensemble Size 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Breiman, L., (1996) Bagging predictors. Machine Learning, 24:123–140.zbMATHMathSciNetGoogle Scholar
  2. Byrne, S., Cunningham, P., Barry, A., Graham, I., Delaney T., Corrigan, O.I., (2000) Using Neural Nets for Decision Support in Prescription and Outcome Prediction in Anticoagulation Drug Therapy, N. Lavrac, S. Miksch (eds.): The Fifth Workshop on Intelligent Data Analysis in Medicine and Pharmacology (IDAMAP-2000).Google Scholar
  3. Cherkauer, K.J. (1995) Stuffing Mind into Computer: Knowledge and Learning for Intelligent Systems. Informatica 19:4 (501–511) Nov. 1995Google Scholar
  4. Condorcet, Marquis J. A. (1781) Sur les elections par scrutiny, Histoire de l’Academie Royale des Sciences, 31–34.Google Scholar
  5. Cunningham, P., Carney, J., (2000) Diversity versus Quality in Classification Ensembles based on Feature Selection, 11th European Conference on Machine Learning (ECML 2000), Lecture Notes in Artificial Intelligence, R. López de Mántaras and E. Plaza, (eds) pp109–116, Springer Verlag.Google Scholar
  6. Cunningham, P., & Zenobi, G., (2001) Case Representation Issues for Case-Based Reasoning from Ensemble Research, to be presented at ICCBR 2001.Google Scholar
  7. Guerra-Salcedo, C., Whitley, D., (1999a). Genetic Approach for Feature Selection for Ensemble Creation. in GECCO-99: Proceedings of the Genetic and Evolutionary Computation Conference, Banzhaf, W., Daida, J., Eiben, A. E., Garzon, M. H., Honavar, V., Jakiela, M., & Smith, R. E. (eds.). Orlando, Florida USA, pp236–243, San Francisco, CA: Morgan Kaufmann.Google Scholar
  8. Guerra-Salcedo, C., Whitley, D., (1999b). Feature Selection Mechanisms for Ensemble Creation: A Genetic Search Perspective, in Data Mining with Evolutionary Algorithms: Research Directions. Papers from the AAAI Workshop. Alex A. Freitas (Ed.) Technical Report WS-99-06. AAAI Press, 1999.Google Scholar
  9. Hansen, L.K., Salamon, P., (1990) Neural Network Ensembles, IEEE Pattern Analysis and Machine Intelligence, 1990. 12, 10, 993–1001.CrossRefGoogle Scholar
  10. Ho, T.K., (1998a) The Random Subspace Method for Constructing Decision Forests, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 8, 832–844.CrossRefGoogle Scholar
  11. Ho, T.K., (1998b) Nearest Neighbours in Random Subspaces, Proc. Of 2nd International Workshop on Statistical Techniques in Pattern Recognition, A. Amin, D. Dori, P. Puil, H. Freeman, (eds.) pp640–648, Springer Verlag LNCS 1451.Google Scholar
  12. Kohavi, R. & John, G.H., (1998) The Wrapper Approach, in Feature Selection for Knowledge Discovery and Data Mining, H. Liu & H. Motoda (eds.), Kluwer Academic Publishers, pp33–50.Google Scholar
  13. Krogh, A., Vedelsby, J., (1995) Neural Network Ensembles, Cross Validation and Active Learning, in Advances in Neural Information Processing Systems 7, G. Tesauro, D. S. Touretsky, T. K. Leen, eds., pp231–238, MIT Press, Cambridge MA.Google Scholar
  14. Liu Y., Yao X. (1999) Ensemble learning via negative correlation, Neural Networks 12, 1999.Google Scholar
  15. Nitzan, S.I., Paroush, J., (1985) Collective Decision Making. Cambridge: Cambridge University Press.Google Scholar
  16. Opitz D., Shavlik J., (1996) Generating Accurate and diverse members of a Neural Network Ensemble, Advances in Neural Information Processing Systems, pp. 535–543, Denver, CO. MIT Press. 1996.Google Scholar
  17. Tumer, K., and Ghosh, J., (1996) Error Correlation and Error Reduction in Ensemble Classifiers, Connection Science, Vol. 8, No. 3 & 4, pp 385–404.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Gabriele Zenobi
    • 1
  • Pádraig Cunningham
    • 1
  1. 1.Department of Computer ScienceTrinity College DublinIreland

Personalised recommendations