Abstract
In this contribution we consider the problem of regression estimation. We elaborate on a framework based on functional analysis giving rise to structured models in the context of reproducing kernel Hilbert spaces. In this setting the task of input selection is converted into the task of selecting functional components depending on one (or more) inputs. In turn the process of learning with embedded selection of such components can be formalized as a convex-concave problem. This results in a practical algorithm that can be implemented as a quadratically constrained quadratic programming (QCQP) optimization problem. We further investigate the mechanism of selection for the class of linear functions, establishing a relationship with LASSO.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Pelckmans, K., Goethals, I., De Brabanter, J., Suykens, J., De Moor, B.: Componentwise Least Squares Support Vector Machines. In: Wang, L. (ed.) Support Vector Machines: Theory and Applications, pp. 77–98. Springer, Heidelberg (2005)
Tibshirani, R.: Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological) 58(1), 267–288 (1996)
Chen, S.: Basis Pursuit. PhD thesis, Department of Statistics, Stanford University (November 1995)
Lanckriet, G., Cristianini, N., Bartlett, P., El Ghaoui, L., Jordan, M.: Learning the Kernel Matrix with Semidefinite Programming. The Journal of Machine Learning Research 5, 27–72 (2004)
Tsang, I.: Efficient hyperkernel learning using second-order cone programming. IEEE transactions on neural networks 17(1), 48–58 (2006)
Ben-Tal, A., Nemirovski, A.: Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications. Society for Industrial Mathematics (2001)
Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)
Cucker, F., Zhou, D.: Learning Theory: An Approximation Theory Viewpoint (Cambridge Monographs on Applied & Computational Mathematics). Cambridge University Press, New York (2007)
Aronszajn, N.: Theory of reproducing kernels. Transactions of the American Mathematical Society 68, 337–404 (1950)
Berlinet, A., Thomas-Agnan, C.: Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer Academic Publishers, Dordrecht (2004)
Wahba, G.: Spline Models for Observational Data. CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 59. SIAM, Philadelphia (1990)
Gu, C.: Smoothing spline ANOVA models. Series in Statistics (2002)
Chen, Z.: Fitting Multivariate Regression Functions by Interaction Spline Models. J. of the Royal Statistical Society. Series B (Methodological) 55(2), 473–491 (1993)
Light, W., Cheney, E.: Approximation theory in tensor product spaces. Lecture Notes in Math., vol. 1169 (1985)
Takemura, A.: Tensor Analysis of ANOVA Decomposition. Journal of the American Statistical Association 78(384), 894–900 (1983)
Huang, J.: Functional ANOVA models for generalized regression. J. Multivariate Analysis 67, 49–71 (1998)
Lin, Y.: Tensor Product Space ANOVA Models. The Annals of Statistics 28(3), 734–755 (2000)
Grant, M., Boyd, S., Ye, Y.: CVX: Matlab Software for Disciplined Convex Programming (2006)
Donoho, D., Johnstone, J.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (2003)
Miller, A.: Subset Selection in Regression. CRC Press, Boca Raton (2002)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Signoretto, M., Pelckmans, K., Suykens, J.A.K. (2008). Quadratically Constrained Quadratic Programming for Subspace Selection in Kernel Regression Estimation. In: Kůrková, V., Neruda, R., Koutník, J. (eds) Artificial Neural Networks - ICANN 2008. ICANN 2008. Lecture Notes in Computer Science, vol 5163. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87536-9_19
Download citation
DOI: https://doi.org/10.1007/978-3-540-87536-9_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-87535-2
Online ISBN: 978-3-540-87536-9
eBook Packages: Computer ScienceComputer Science (R0)