The Group-Lasso: ℓ1, ∞ Regularization versus ℓ1,2 Regularization
The ℓ1, ∞ norm and the ℓ1,2 norm are well known tools for joint regularization in Group-Lasso methods. While the ℓ1,2 version has been studied in detail, there are still open questions regarding the uniqueness of solutions and the efficiency of algorithms for the ℓ1, ∞ variant. For the latter, we characterize the conditions for uniqueness of solutions, we present a simple test for uniqueness, and we derive a highly efficient active set algorithm that can deal with input dimensions in the millions. We compare both variants of the Group-Lasso for the two most common application scenarios of the Group-Lasso, one is to obtain sparsity on the level of groups in “standard” prediction problems, the second one is multi-task learning where the aim is to solve many learning problems in parallel which are coupled via the Group-Lasso constraint. We show that both version perform quite similar in “standard” applications. However, a very clear distinction between the variants occurs in multi-task settings where the ℓ1,2 version consistently outperforms the ℓ1, ∞ counterpart in terms of prediction accuracy.
Unable to display preview. Download preview PDF.
- 2.Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. J. Roy. Stat. Soc. B, 49–67 (2006)Google Scholar
- 5.Argyriou, A., Evgeniou, T., Pontil, M.: Multi-task feature learning. In: Advances in Neural Information Processing Systems, vol. 19. MIT Press, Cambridge (2007)Google Scholar
- 8.Schmidt, M., Murphy, K., Fung, G., Rosales, R.: Structure learning in random fields for heart motion abnormality detection. In: CVPR (2008)Google Scholar
- 9.Quattoni, A., Carreras, X., Collins, M., Darrell, T.: An efficient projection for l 1 ∞ regularization. In: 26th Intern. Conference on Machine Learning (2009)Google Scholar
- 10.Liu, H., Palatucci, M., Zhang, J.: Blockwise coordinate descent procedures for the multi-task lasso, with applications to neural semantic basis discovery. In: 26th Intern. Conference on Machine Learning (2009)Google Scholar
- 12.McCullaghand, P., Nelder, J.: Generalized Linear Models. Chapman & Hall, Boca Raton (1983)Google Scholar