The structured elastic net for quantile regression and support vector classification
In view of its ongoing importance for a variety of practical applications, feature selection via ℓ1-regularization methods like the lasso has been subject to extensive theoretical as well empirical investigations. Despite its popularity, mere ℓ1-regularization has been criticized for being inadequate or ineffective, notably in situations in which additional structural knowledge about the predictors should be taken into account. This has stimulated the development of either systematically different regularization methods or double regularization approaches which combine ℓ1-regularization with a second kind of regularization designed to capture additional problem-specific structure. One instance thereof is the ‘structured elastic net’, a generalization of the proposal in Zou and Hastie (J. R. Stat. Soc. Ser. B 67:301–320, 2005), studied in Slawski et al. (Ann. Appl. Stat. 4(2):1056–1080, 2010) for the class of generalized linear models.
In this paper, we elaborate on the structured elastic net regularizer in conjunction with two important loss functions, the check loss of quantile regression and the hinge loss of support vector classification. Solution paths algorithms are developed which compute the whole range of solutions as one regularization parameter varies and the second one is kept fixed.
The methodology and practical performance of our approach is illustrated by means of case studies from image classification and climate science.
KeywordsDouble regularization Elastic net Quantile regression Solution path Support vector classification
Unable to display preview. Download preview PDF.
- Bradley, P., Mangasarian, O.: Feature selection via concave minimization and support vector machines. In: International Conference on Machine Learning (1998) Google Scholar
- Christianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge (2000) Google Scholar
- El Anbari, M., Mkhadri, A.: Penalized regression combing the L 1 norm and a correlation based penalty. Technical report, Université Paris Sud 11 (2008) Google Scholar
- Landau, S., Ellison-Wright, I., Bullmore, E.: Tests for a difference in timing of physiological response between two brain regions measured by using functional magnetic resonance imaging. Appl. Stat. 63–82, 53 (2003) Google Scholar
- Ramsay, J., Silverman, B.: Functional Data Analysis. Springer, New York (2006) Google Scholar
- Rosset, S.: Bi-level path following for cross validated solution of kernel quantile regression. In: International Conference on Machine Learning (2008) Google Scholar
- Schölkopf, B., Smola, A.: Learning with Kernels. MIT Press, Cambridge (2002) Google Scholar
- Tibshirani, R.: Regression shrinkage and variable selection via the lasso. J. R. Stat. Soc. Ser. B 58, 671–686 (1996) Google Scholar
- Wood, S.: R package gamair: Data for “GAMs: An Introduction with R”, Version 0.0-4. Available from www.r-project.org (2006)
- Zhu, J., Rosset, S., Hastie, T., Tibshirani, R.: L 1 norm support vector machine. Adv. Neural Inf. Process. Syst. 16, 55–63 (2003) Google Scholar