Effects of Many Feature Candidates in Feature Selection and Classification
We address the problems of analyzing many feature candidates when performing feature selection and error estimation on a limited data set. A Monte Carlo study of multivariate normal distributed data has been performed to illustrate the problems. Two feature selection methods are tested: Plus- 1- Minus- 1 and Sequential Forward Floating Selection. The simulations demonstrate that in order to find the correct features, the number of features initially analyzed is an important factor, besides the number of samples. Moreover, the sufficient ratio of number of training samples to feature candidates is not a constant. It depends on the number of feature candidates, training samples and the Mahalanobis distance between the classes. The two feature selection methods analyzed gave the same result. Furthermore, the simulations demonstrate how the leave-one-out error estimate can be a highly biased error estimate when feature selection is performed on the same data as the error estimation. It may even indicate complete separation of the classes, while no real difference between the classes exists.
KeywordsError Estimate Feature Selection Training Sample Mahalanobis Distance Feature Selection Method
- 2.R. O Duda and P. E Hart. Pattern classification and scene analysis. A Wiley-interscience publication, first edition, 1973.Google Scholar
- 3.R. P. W. Duin. A matlab toolbox for pattern recognition. Technical Report Version 3.0, Delft University of Technology, 2000.Google Scholar
- 10.H. Schulerud. The influence of feature selection on error estimates in linear discriminant analysis. Submittet to Pattern Recognition.Google Scholar
- 11.S. D. Stearns. On selecting features or pattern classifiers. Proc. Third Intern. Conf. Pattern Recognition, pages 71–75, 1976.Google Scholar