Feature Selection for Ensembles of Simple Bayesian Classifiers
A popular method for creating an accurate classifier from a set of training data is to train several classifiers, and then to combine their predictions. The ensembles of simple Bayesian classifiers have traditionally not been a focus of research. However, the simple Bayesian classifier has much broader applicability than previously thought. Besides its high classification accuracy, it also has advantages in terms of simplicity, learning speed, classification speed, storage space, and incrementality. One way to generate an ensemble of simple Bayesian classifiers is to use different feature subsets as in the random subspace method. In this paper we present a technique for building ensembles of simple Bayesian classifiers in random subspaces. We consider also a hill-climbing-based refinement cycle, which improves accuracy and diversity of the base classifiers. We conduct a number of experiments on a collection of real-world and synthetic data sets. In many cases the ensembles of simple Bayesian classifiers have significantly higher accuracy than the single “global” simple Bayesian classifier. We consider several methods for integration of simple Bayesian classifiers. The dynamic integration better utilizes ensemble diversity than the static integration.
KeywordsFeature Selection Ensemble Member Feature Subset Base Classifier Iterative Refinement
Unable to display preview. Download preview PDF.
- 3.Brodley, C., Lane, T.: Creating and exploiting coverage and diversity. In: Proc. AAAI-96 Workshop on Integrating Multiple Learned Models (1996) 8–14.Google Scholar
- 4.Cunningham, P.: Diversity versus quality in classification ensembles based on feature selection. Tech. Report TCD-CS-2000-02, Dept. of Computer Science, Trinity College Dublin, Ireland (2000).Google Scholar
- 5.Dietterich, T. G.: Ensemble Learning Methods. In: M.A. Arbib (ed.), Handbook of Brain Theory and Neural Networks, 2nd ed., MIT Press (2001).Google Scholar
- 7.Elkan C.: Boosting and naïve Bayesian learning. Tech. Report CS97-557, Dept. of CS and Engineering, Un-ty of California, San Diego, USA (1997).Google Scholar
- 10.Kohavi, R., Sommerfield, D., Dougherty, J.: Data mining using MLC++: a machine learning library in C++. Tools with Artificial Intelligence, IEEE CS Press (1996) 234–245.Google Scholar
- 11.Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation, and active learning. In D. Touretzky, T. Leen (eds.), Advances in Neural Information Processing Systems, Vol. 7, Cambridge, MA, MIT Press (1995) 231–238.Google Scholar
- 12.Opitz, D.: Feature selection for ensembles. In: Proc. 16th National Conf. on Artificial Intelligence, AAAI (1999) 379–384.Google Scholar
- 13.Pedersen, T.: A simple approach to building ensembles of naive Bayesian classifiers for word sense disambiguation. In: Proc. 1st Annual Meeting of the North American Chapter of the Association for Computational Linguistics, Seattle, WA (2000) 63–69.Google Scholar
- 16.Skurichina, M., Duin, R.P.W.: Bagging and the random subspace method for redundant feature spaces. In: J. Kittler, F. Roli (eds.), Proc. 2nd Int. Workshop on Multiple Classifier Systems MCS 2001, Cambridge, UK (2001) 1–10.Google Scholar
- 17.Tsymbal, A., Puuronen, S., Skrypnyk, I.: Ensemble feature selection with dynamic integration of classifiers, In: Proc. Int. ICSC Congress on Computational Intelligence Methods and Applications CIMA’2001, Bangor, Wales, U.K. (2001).Google Scholar