Design Choices and Theoretical Issues for Relative Feature Importance, a Metric for Nonparametric Discriminatory Power

  • Hilary J. Holz
  • Murray H. Loew
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1876)


We have developed relative feature importance (RFI), a metric for the classifier-independent ranking of features. Previously, we have shown the metric to rank accurately features for a wide variety of artificial and natural problems, for both two-class and multi-class problems. In this paper, we present the design of the metric, including both theoretical considerations and statistical analysis of the possible components.


discriminatory power feature selection feature extraction feature analysis non-parametric classifier-independent relative feature importance multi-class 


  1. 1.
    I. Chang and M.H. Loew, “Pattern Recognition with New Class Discovery,” Proceedings of the IEEE Computer Society Conference on Pattern Recognition and Computer Vision, pp. 438–443., 1991.Google Scholar
  2. 2.
    M. Ben-Bassat, “f-Entropies, Probability of Error, and Feature Selection,” Information and Control, vol. 39, pp. 227–242, 1978.zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    T.M. Cover, “The Best Two Independent Measurements are Not the Two Best,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 4, no. 1, pp. 116–117, Jan. 1974.zbMATHGoogle Scholar
  4. 4.
    K. Fukunaga and J.M. Mantock, “Nonparametric Discriminant Analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 5, no. 6, pp. 671–678, Nov. 1983.zbMATHCrossRefGoogle Scholar
  5. 5.
    G.F. Hughes, “On the mean accuracy of statistical pattern recognizers,” IEEE Transactions on Information Theory, vol. IT-14, pp. 55–63, Jan. 1968.Google Scholar
  6. 6.
    R.O. Duda and P.E. Hart, Pattern Classification and Scene Analysis, New York: Wiley & Sons, 1973.zbMATHGoogle Scholar
  7. 7.
    R.D. Short and K. Fukunaga, “The Optimal Distance Measure for Nearest Neighbor Classification,” IEEE Transactions on Information Theory, vol. IT-27, no. 5, pp. 622–627, Sept. 1981.CrossRefMathSciNetGoogle Scholar
  8. 8.
    H.J. Holz and M.H. Loew, “Multi-class classifier-independent feature analysis,” Pattern Recognition Letters, vol. 18, no. 11–13, pp. 1219–1224, Nov. 1997.CrossRefGoogle Scholar
  9. 9.
    K. Fukunaga, Introduction to Statistical Pattern Recognition: 2nd Edition, Academic Press, Inc. 1990.Google Scholar
  10. 10.
    T.M. Cover and P.E. Hart, “Nearest neighbor pattern classification,” IEEE Transactions on Information Theory, vol. IT-13, pp. 21–27, Jan. 1967.Google Scholar
  11. 11.
    J.M. Van Campenhout, “The Arbitrary Relation Between Probability of Error and Measurement Subset,” Journal of the American Statistical Association, vol. 75, no. 369, pp. 104–109, March 1980.zbMATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    P.M. Narendra and K. Fukunaga, “A branch and bound algorithm for feature subset selection,” IEEE Transactions on Computers, vol. 26, no. 9, pp. 917–922, Sept. 1977.zbMATHCrossRefGoogle Scholar
  13. 13.
    N.L. Johnson and F.C. Leone, Statistics and Experimental Design in Engineering and Physical Sciences, pp. 614–661, New York: Wiley, 1977.zbMATHGoogle Scholar
  14. 14.
    H.J. Holz and M.H. Loew, “Relative Feature Importance: A Classifier-Independent Approach to Feature Selection,” Pattern Recognition in Practice IV, E.S. Gelsema and L.N. Kanal, eds, pp. 473–487, Elsevier Science B.V., 1994.Google Scholar
  15. 15.
    -, “Non-Parametric Discriminatory Power,” Proceedings of the 1994 IEEE-IMS Workshop on Information Theory and Statistics, pp. 65, Alexandria, VA, 1994.Google Scholar
  16. 16.
    -, “Validation of Relative Feature Importance Using a Natural Data Set,” 15th International Conference on Pattern Recognition, Barcelona, Spain, September 2000.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2000

Authors and Affiliations

  • Hilary J. Holz
    • 1
  • Murray H. Loew
    • 1
  1. 1.Department of Electrical and Computer EngineeringThe George Washington UniversityWashington, DC

Personalised recommendations