Analysis and Insights into the Variable Selection Problem

  • Amir F. Atiya
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4233)


In many large applications a large number of input variables is initially available, and a subset selection step is needed to select the best few to be be used in the subsequent classification or regression step. The designer initially screens the inputs for the ones that have good predictive ability and that are not too much correlated with the other selected inputs. In this paper, we study how the predictive ability of the inputs, viewed individually, reflect on the performance of the group (i.e. what are the chances that as a group they perform well). We also study the effect of “irrelevant” inputs. We develop a formula for the distribution of the change in error due to adding an irrelevant input. This can be a useful reference. We also study the role of correlations and their effect on group performance. To study these issues, we first perform a theoretical analysis for the case of linear regression problems. We then follow with an empirical study for nonlinear regression models such as neural networks.


Mean Square Error Group Performance Benchmark Problem Scatter Diagram Random Input 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Anderson, T.W.: An Introduction to Multivariate Statistical Analysis. John Wiley & Sons, Chichester (2003)MATHGoogle Scholar
  2. 2.
    DELVE Data Repository,
  3. 3.
    Dong, M., Kothari, R.: Feature subset selection using a new definition of classifiability. Pattern Recognition Letters 23, 1215–1225 (2003)CrossRefGoogle Scholar
  4. 4.
    Harrell, F.: Regression Modeling Strategies. Springer, Heidelberg (2001)MATHGoogle Scholar
  5. 5.
    Jain, A., Zongker, D.: Feature Selection: Evaluation, Application, and Small Sample Performance. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(2), 153–158 (1997)CrossRefGoogle Scholar
  6. 6.
    Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artificial Intelligence 97(1-2), 273–324 (1997)MATHCrossRefGoogle Scholar
  7. 7.
    Naylor, A., Sell, G.: Linear Operator Theory in Science and Engineering. Springer, Heidelberg (1982)MATHGoogle Scholar
  8. 8.
    Portinale, L., Saitta, L.: Feature selection, University of Dortmund Technical Report, Dortmund, GermanyGoogle Scholar
  9. 9.
    Yoo, S.G., Chong, K.T., Yi, S.Y.: Neural netowrk modeling of transmission rate control factor for multimedia transmission using the Interent. In: Zhang, Y., Tanaka, K., Yu, J.X., Wang, S., Li, M. (eds.) APWeb 2005. LNCS, vol. 3399, pp. 851–862. Springer, Heidelberg (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Amir F. Atiya
    • 1
  1. 1.Dept Computer Engineering Cairo University GizaEgypt

Personalised recommendations