A computationally fast variable importance test for random forests for high-dimensional data

  • Silke Janitza
  • Ender Celik
  • Anne-Laure Boulesteix
Regular Article


Random forests are a commonly used tool for classification and for ranking candidate predictors based on the so-called variable importance measures. These measures attribute scores to the variables reflecting their importance. A drawback of variable importance measures is that there is no natural cutoff that can be used to discriminate between important and non-important variables. Several approaches, for example approaches based on hypothesis testing, were developed for addressing this problem. The existing testing approaches require the repeated computation of random forests. While for low-dimensional settings those approaches might be computationally tractable, for high-dimensional settings typically including thousands of candidate predictors, computing time is enormous. In this article a computationally fast heuristic variable importance test is proposed that is appropriate for high-dimensional data where many variables do not carry any information. The testing approach is based on a modified version of the permutation variable importance, which is inspired by cross-validation procedures. The new approach is tested and compared to the approach of Altmann and colleagues using simulation studies, which are based on real data from high-dimensional binary classification settings. The new approach controls the type I error and has at least comparable power at a substantially smaller computation time in the studies. Thus, it might be used as a computationally fast alternative to existing procedures for high-dimensional data settings where many variables do not carry any information. The new approach is implemented in the R package vita.


Gene selection Feature selection Random forests Variable importance Variable selection Variable importance test 

Mathematics Subject Classification

62F07 65C60 62-07 

Supplementary material

11634_2016_270_MOESM1_ESM.pdf (671 kb)
Supplementary material 1 (pdf 671 KB)


  1. Alon U, Barkai N, Notterman DA, Gish K, Ybarra S, Mack D, Levine AJ (1999) Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc Natl Acad Sci 96:6745–6750CrossRefGoogle Scholar
  2. Altmann A, Toloşi L, Sander O, Lengauer T (2010) Permutation importance: a corrected feature importance measure. Bioinformatics 26:1340–1347CrossRefGoogle Scholar
  3. Boulesteix A-L (2015) Ten simple rules for reducing overoptimistic reporting in methodological computational research. PLoS Comput Biol 4:e1004191CrossRefGoogle Scholar
  4. Boulesteix AL, Bender A, Bermejo JL, Strobl C (2012) Random forest Gini importance favours SNPs with large minor allele frequency: assessment, sources and recommendations. Brief Bioinform 13:292–304CrossRefGoogle Scholar
  5. Breiman L (2001) Random forests. Mach Learn 45:5–32CrossRefMATHGoogle Scholar
  6. Dettling M, Bühlmann P (2003) Boosting for tumor classification with gene expression data. Bioinformatics 19:1061–1069CrossRefGoogle Scholar
  7. Díaz-Uriarte R, De Andres SA (2006) Gene selection and classification of microarray data using random forest. BMC Bioinform 7:3CrossRefGoogle Scholar
  8. Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA et al (1999) Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286:531–537CrossRefGoogle Scholar
  9. Gregorutti B, Michel B, Saint-Pierre P (2013) Correlation and variable importance in random forests. arXiv preprint arXiv:1310.5726
  10. Hapfelmeier A, Ulm K (2013) A new variable selection approach using random forests. Comput Stat Data Anal 60:50–69MathSciNetCrossRefGoogle Scholar
  11. Hothorn T, Hornik K, Zeileis A (2006) Unbiased recursive partitioning: a conditional inference framework. J Comput Graph Stat 15:651–674MathSciNetCrossRefGoogle Scholar
  12. Huynh-Thu VA, Saeys Y, Wehenkel L, Geurts P (2012) Statistical interpretation of machine learning-based feature importance scores for biomarker discovery. Bioinformatics 28:1766–1774CrossRefGoogle Scholar
  13. Ishwaran H (2007) Variable importance in binary regression trees and forests. Electron J Stat 1:519–537MathSciNetCrossRefMATHGoogle Scholar
  14. Ishwaran H, Kogalur UB, Blackstone EH, Lauer MS (2008) Random survival forests. Ann Appl Stat 2:841–860MathSciNetCrossRefMATHGoogle Scholar
  15. Janitza S, Strobl C, Boulesteix AL (2013) An AUC-based permutation variable importance measure for random forests. BMC Bioinform 14:119CrossRefGoogle Scholar
  16. Janitza S, Tutz G, Boulesteix A-L (2016) Random forest for ordinal responses: prediction and variable selection. Comput Stat Data Anal 96:57–73MathSciNetCrossRefGoogle Scholar
  17. Kim H, Loh W-Y (2001) Classification trees with unbiased multiway splits. J Am Stat Assoc 96:589–604MathSciNetCrossRefGoogle Scholar
  18. Liaw A, Wiener M (2002) Classification and regression by randomForest. R News 2:18–22Google Scholar
  19. Louppe G, Wehenkel L, Sutera A, Geurts P (2013) Understanding variable importances in forests of randomized trees. In: Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ (eds) Advances in neural information processing systems, pp 431–439Google Scholar
  20. Molinaro AM, Carriero N, Bjornson R, Hartge P, Rothman N, Chatterjee N (2011) Power of data mining methods to detect genetic associations and interactions. Hum Hered 72:85–97CrossRefGoogle Scholar
  21. Nicodemus K (2011) Letter to the editor: on the stability and ranking of predictors from random forest variable importance measures. Brief Bioinform 12:369–373CrossRefGoogle Scholar
  22. Nicodemus K, Malley J (2009) Predictor correlation impacts machine learning algorithms: implications for genomic studies. Bioinformatics 25:1884–1890CrossRefGoogle Scholar
  23. Pepe M (2004) The statistical evaluation of medical tests for classification and prediction. Oxford University Press, USAMATHGoogle Scholar
  24. Phipson B, Smyth G (2010) Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn. Stat Appl Genet Mol Biol 9:1544–6115MathSciNetMATHGoogle Scholar
  25. Polak P, Karlić R, Koren A, Thurman R, Sandstrom R, Lawrence MS, Reynolds A, Rynes E, Vlahoviček K, Stamatoyannopoulos JA et al (2015) Cell-of-origin chromatin organization shapes the mutational landscape of cancer. Nature 518:360–364CrossRefGoogle Scholar
  26. Pomeroy SL, Tamayo P, Gaasenbeek M, Sturla LM, Angelo M, McLaughlin ME, Kim JY, Goumnerova LC, Black PM, Lau C et al (2002) Prediction of central nervous system embryonal tumour outcome based on gene expression. Nature 415:436–442CrossRefGoogle Scholar
  27. Prosperi MC, Marinho S, Simpson A, Custovic A, Buchan IE (2014) Predicting phenotypes of asthma and eczema with machine learning. BMC Med Genomics 7:S7CrossRefGoogle Scholar
  28. Reif DM, Motsinger-Reif AA, McKinney BA, Rock MT, Crowe J, Moore JH (2009) Integrated analysis of genetic and proteomic data identifies biomarkers associated with adverse events following smallpox vaccination. Genes Immun 10:112–119CrossRefGoogle Scholar
  29. Schwarz DF, König IR, Ziegler A (2010) On safari to random jungle: a fast implementation of random forests for high-dimensional data. Bioinformatics 26:1752–1758CrossRefGoogle Scholar
  30. Singh D, Febbo PG, Ross K, Jackson DG, Manola J, Ladd C, Tamayo P, Renshaw AA, D’Amico AV, Richie JP et al (2002) Gene expression correlates of clinical prostate cancer behavior. Cancer Cell 1:203–209CrossRefGoogle Scholar
  31. Strobl C, Boulesteix A-L, Kneib T, Augustin T, Zeileis A (2008) Conditional variable importance for random forests. BMC Bioinform 9:307CrossRefGoogle Scholar
  32. Strobl C, Boulesteix AL, Zeileis A, Hothorn T (2007) Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinform 8:25CrossRefGoogle Scholar
  33. Strobl C, Malley J, Tutz G (2009) An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychol Methods 14:323–348CrossRefGoogle Scholar
  34. Strobl C, Zeileis A (2008) Danger: high power!—exploring the statistical properties of a test for random forest variable importance. In: Brito P (ed) Proceedings of the 18th international conference on computational statistics, Porto, Portugal (CD-ROM). Springer, Heidelberg, pp 59–66Google Scholar
  35. Szymczak S, Holzinger E, Dasgupta A, Malley JD, Molloy AN, Mills JL, Brody LC, Stambolian D, Bailey-Wilson JE (2016) r2VIM: a new variable selection method for random forests in genome-wide association studies. BioData Min 9:7CrossRefGoogle Scholar
  36. Tan AC, Gilbert D (2003) Ensemble machine learning on gene expression data for cancer classification. Appl Bioinform 2:S75–S83Google Scholar
  37. Tang R, Sinnwell JP, Li J, Rider DN, de Andrade M, Biernacka JM (2009) Identification of genes and haplotypes that predict rheumatoid arthritis using random forests. BMC Proc 3:S68CrossRefGoogle Scholar
  38. van’t Veer LJ, Dai H, Van De Vijver MJ, He YD, Hart AA, Mao M, Peterse HL, van der Kooy K, Marton MJ, Witteveen AT et al (2002) Gene expression profiling predicts clinical outcome of breast cancer. Nature 415:530–536Google Scholar
  39. Wang H, Yang F, Luo Z (2016) An experimental study of the intrinsic stability of random forest variable importance measures. BMC Bioinform 17:60CrossRefGoogle Scholar
  40. Wang-Sattler R, Yu Z, Herder C, Messias AC, Floegel A, He Y, Heim K, Campillos M, Holzapfel C, Thorand B et al (2012) Novel biomarkers for pre-diabetes identified by metabolomics. Mol Syst Biol 8:615. doi:10.1038/msb.2012.43
  41. Wright MN, Ziegler A (2016) ranger: a fast implementation of random forests for high dimensional data in C++ and R. J Stat Softw (in press)Google Scholar
  42. Yatsunenko T, Rey FE, Manary MJ, Trehan I, Dominguez-Bello MG, Contreras M, Magris M, Hidalgo G, Baldassano RN, Anokhin AP et al (2012) Human gut microbiome viewed across age and geography. Nature 486:222–227Google Scholar
  43. Zhu R, Zeng D, Kosorok MR (2015) Reinforcement learning trees. JASA 110:1770–1784MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Department of Medical Informatics, Biometry and EpidemiologyUniversity of MunichMunichGermany

Personalised recommendations