Abstract
Feature selection for ensembles has shown to be an effective strategy for ensemble creation due to its ability of producing good subsets of features, which make the classifiers of the ensemble disagree on difficult cases. In this paper we present an ensemble feature selection approach based on a hierarchical multi-objective genetic algorithm. The algorithm operates in two levels. Firstly, it performs feature selection in order to generate a set of classifiers and then it chooses the best team of classifiers. In order to show its robustness, the method is evaluated in two different contexts: supervised and unsupervised feature selection. In the former, we have considered the problem of handwritten digit recognition while in the latter, we took into account the problem of handwritten month word recognition. Experiments and comparisons with classical methods, such as Bagging and Boosting, demonstrated that the proposed methodology brings compelling improvements when classifiers have to work with very low error rates.
Keywords
- Feature Selection
- Recognition Rate
- Feature Subset Selection
- Perform Feature Selection
- Handwritten Digit Recognition
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, access via your institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Davies, D.L., Bouldin, D.W.: A cluster separation measure. IEEE Trans. on Pattern Analysis and Machine Intelligence 1(224-227), 550–554 (1979)
Deb, K.: Multi-Objective Optimization using Evolutionary Algorithms, 2nd edn. John Wiley and Sons, Chichester (2002)
Dy, J.G., Brodley, C.E.: Feature subset selection and order identification for unsupervised learning. In: Proc. 17th International Conference on Machine Learning, Stanford University, CA (July 2000)
Emmanouilidis, C., Hunter, A., MacIntyre, J.: A multiobjective evolutionary setting for feature selection and a commonality-based crossover operator. In: Proc. of Congress on Evolutionary Computation, vol. 1, pp. 309–316 (2000)
Gerra-Salcedo, C., Whitley, D.: Genetic approach to feature selection for ensemble creatin. In: Proc. of Genetic and Evolutionary Computation Conference, pp. 236–243 (1999)
Ho, T.K.: The random subspace method for constructing decision forests. IEEE Trans. on Pattern Analysis and Machine Intelligence 20(8), 832–844 (1998)
Krogh, A., Vedelsby, J.: Neural networks ensembles, cross validation, and active learning. In: Tesauro, G., et al. (eds.) Advances in Neural Information Processing Systems 7, pp. 231–238. MIT Press, Cambridge (1995)
Kudo, M., Sklansky, J.: Comparision of algorithms that select features for pattern classifiers. Pattern Recognition 33(1), 25–41 (2000)
Moody, J., Utans, J.: Principled architecture selection for neural networks: Application to corporate bond rating prediction. In: Moody, J., Hanson, S.J., Lippmann, R.P. (eds.) Advances in Neural Information Processing Systems 4, Morgan Kaufmann, San Francisco (1991)
Morita, M., Oliveira, L.S., Sabourin, R., Bortolozzi, F., Suen, C.Y.: An HMM-MLP hybrid system to recognize handwritten dates. In: Proc. of International Joint Conference on Neural Networks, Honolulu, USA, pp. 867–872. IEEE Press, Los Alamitos (2002)
Morita, M., Sabourin, R., Bortolozzi, F., Suen, C.Y.: Unsupervised feature selection using multi-objective genetic algorithms for handwritten word recognition. In: Procs of the 7th ICDAR, pp. 666–670. IEEE Computer Society Press, Los Alamitos (2003)
Morita, M., El Yacoubi, A., Sabourin, R., Bortolozzi, F., Suen, C.Y.: Handwritten month word recognition on Brazilian bank cheques. In: Proc. 6th ICDAR, pp. 972–976 (2001)
Oliveira, L.S., Sabourin, R., Bortolozzi, F., Suen, C.Y.: Automatic recognition of handwritten numerical strings: A recognition and verification strategy. IEEE Trans. on Pattern Analysis and Machine Intelligence 24(11), 1438–1454 (2002)
Oliveira, L.S., Sabourin, R., Bortolozzi, F., Suen, C.Y.: A methodology for feature selection using multi-objective genetic algorithms for handwritten digit string recognition. International Journal of Pattern Recognition and Artificial Intelligence 17(6), 903–930 (2003)
Optiz, D.W.: Feature selection for ensembles. In: Proc. of 16th International Conference on Artificial Intelligence, pp. 379–384 (1999)
Partridge, D., Yates, W.B.: Engineering multiversion neural-net systems. Neural Computation 8(4), 869–893 (1996)
Siedlecki, W., Sklansky, J.: A note on genetic algorithms for large scale on feature selection. Pattern Recognition Letters 10, 335–347 (1989)
Tsymbal, A., Puuronen, S., Patterson, D.W.: Ensemble feature selection with the simple Bayesian classification. Information Fusion 4, 87–100 (2003)
Tumer, K., Ghosh, J.: Error correlation and error reduction in ensemble classifiers. Connection Science 8(3-4), 385–404 (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Oliveira, L.S., Morita, M., Sabourin, R., Bortolozzi, F. (2005). Multi-objective Genetic Algorithms to Create Ensemble of Classifiers. In: Coello Coello, C.A., Hernández Aguirre, A., Zitzler, E. (eds) Evolutionary Multi-Criterion Optimization. EMO 2005. Lecture Notes in Computer Science, vol 3410. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-31880-4_41
Download citation
DOI: https://doi.org/10.1007/978-3-540-31880-4_41
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-24983-2
Online ISBN: 978-3-540-31880-4
eBook Packages: Computer ScienceComputer Science (R0)