Boosting Feature Selection
It is possible to reduce the error rate of a single classifier using a classifier ensemble. However, any gain in performance is undermined by the increased computation of performing classification several times. Here the Adaboost FS algorithm is proposed which builds on two popular areas of ensemble research: Adaboost and Ensemble Feature Selection (EFS). The aim of Adaboost FS is to reduce the number of features used by each base classifer and hence the overall computation required by the ensemble. To do this the algorithm combines a regularised version of Boosting Adaboost Reg  with a floating feature search for each base classifier.
Adaboost FS is compared using four benchmark data sets to Adaboost All , which uses all features and to Adaboost RSM , which uses a random selection of features. Performance is assessed based on error rate, ensemble error and diversity, and the total number of features used for classification. Results show that Adaboost FS achieves a lower error rate and higher diversity than Adaboost All , and achieves a lower error rate and comparable diversity to Adaboost RSM . However, over the other methods Adaboost FS produces a significant reduction in the number of features required for classification in each base classifier and the entire ensemble.
KeywordsFeature Selection Feature Subset Base Classifer Benchmark Dataset Ensemble Size
Unable to display preview. Download preview PDF.
- 2.Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Proc. 13th International Conference on Machine Learning, pp. 148–156 (1996)Google Scholar
- 3.Schapire, R., Freund, Y., Bartlett, P., Lee, W.: Boosting the margin: A new explanation for the effectiveness of voting methods. The Annuals of Statistics, 1651–1686 (1998)Google Scholar
- 5.Quinlan, J.R.: Bagging, boosting and c4.5. In: Proceedings of the Thirteenth National Conference on Artificial Intelligence, pp. 725–730 (1996)Google Scholar
- 7.Tieu, K., Viola, P.: Boosting image retrieval. In: IEEE Conf. on Computer Vision and Pattern Recognition, pp. 228–235 (2000)Google Scholar
- 11.Guerra-Salcedo, C., Whitley, D.: Feature selection mechanisms for ensemble creation: a genetic search perspective. In: AAAI 1999 (1999)Google Scholar
- 16.Blake, C., Merz, C.: UCI repository of machine learning databases (1998)Google Scholar
- 17.Feiss, J.: Statistical methods for rates and proportions (1981)Google Scholar