Extensions of stability selection using subsamples of observations and covariates
- 266 Downloads
We introduce extensions of stability selection, a method to stabilise variable selection methods introduced by Meinshausen and Bühlmann (J R Stat Soc 72:417–473, 2010). We propose to apply a base selection method repeatedly to random subsamples of observations and subsets of covariates under scrutiny, and to select covariates based on their selection frequency. We analyse the effects and benefits of these extensions. Our analysis generalizes the theoretical results of Meinshausen and Bühlmann (J R Stat Soc 72:417–473, 2010) from the case of half-samples to subsamples of arbitrary size. We study, in a theoretical manner, the effect of taking random covariate subsets using a simplified score model. Finally we validate these extensions on numerical experiments on both synthetic and real datasets, and compare the obtained results in detail to the original stability selection method.
KeywordsVariable selection Stability selection Subsampling
We are extremely grateful to Nicolai Meinshausen and Peter Bühlmann for communicating to us the R-code used by Meinshausen and Bühlmann (2010) as well as for numerous discussions. We are indebted to Richard Samworth and Rajen Shah for numerous discussions and for hosting the first author during part of this work. We thank Maurilio Gutzeit for helping us with part of the numerical experiments.
- Bolasso, F.B.: Model consistent Lasso estimation through the bootstrap. In: Proceedings of 25th International Conference on Machine Learning (ICML), pp. 33–40. ACM (2008)Google Scholar
- Beinrucker, A., Dogan, U., Blanchard, G.: Early stopping for mutual information based feature selection. In: Proceedings of 21st International Conference on Pattern Recognition (ICPR), pp. 975–978 (2012a)Google Scholar
- Beinrucker, A., Dogan, U., Blanchard, G.: A simple extension of stability feature selection. In: Pattern Recognition, vol. 7476 of Lecture Notes in Computer Science, pp. 256–265. Springer, New York (2012b)Google Scholar
- Escudero, G., Marquez, L., Rigau, G.: Boosting applied to word sense disambiguation. In: Proceedings of European Conference on Machine Learning (ECML), pp. 129–141 (2000)Google Scholar
- Guyon, I.: Feature Extraction: Foundations and Applications, vol. 207. Springer, New York (2006)Google Scholar
- Hastie, T., Efron, B.: LARS: Least Angle Regression, Lasso and Forward Stagewise (2012). URL http://CRAN.R-project.org/package=lars. R package version 1.1
- He, Z., Yu, W.: Stable feature selection for biomarker discovery. Comput. Biol. Chem. 34(4), 215–225 (2010)Google Scholar
- Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)
- MASH Consortium. The MASH project. http://www.mash-project.eu (2012). [Online; Accessed 19 Mar 2013]