Speeding Up the Wrapper Feature Subset Selection in Regression by Mutual Information Relevance and Redundancy Analysis
A hybrid filter/wrapper feature subset selection algorithm for regression is proposed. First, features are filtered by means of a relevance and redundancy filter using mutual information between regression and target variables. We introduce permutation tests to find statistically significant relevant and redundant features. Second, a wrapper searches for good candidate feature subsets by taking the regression model into account. The advantage of a hybrid approach is threefold. First, the filter provides interesting features independently from the regression model and, hence, allows for an easier interpretation. Secondly, because the filter part is computationally less expensive, the global algorithm will faster provide good candidate subsets compared to a stand-alone wrapper approach. Finally, the wrapper takes the bias of the regression model into account, because the regression model guides the search for optimal features. Results are shown for the ‘Boston housing’ and ‘orange juice’ benchmarks based on the multilayer perceptron regression model.
KeywordsFeature Selection Mean Square Error Mutual Information Orange Juice Feature Subset
Unable to display preview. Download preview PDF.
- 4.Van Dijck, G., Van Hulle, M.M., Wevers, M.: Hierarchical Feature Subset Selection for Features Computed from the Continuous Wavelet Transform. In: 2005 IEEE Workshop on Machine Learning for Signal Processing, pp. 81–86 (2005)Google Scholar
- 8.Francois, D., Wertz, V., Verleysen, M.: The Permutation Test for Feature Selection by Mutual Information. In: European Symposium on Artificial Neural Networks, pp. 239–244 (2006)Google Scholar
- 9.John, G., Kohavi, R., Pfleger, K.: Irrelevant Features and the Subset Selection Problem. In: Proc. of the Eleventh Int. Conf. on Machine Learning, pp. 121–129 (1994)Google Scholar
- 11.Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press Inc., New York (1997)Google Scholar