Abstract
We examine the interplay of optimization and machine learning. Great progress has been made in machine learning by cleverly reducing machine learning problems to convex optimization problems with one or more hyper-parameters. The availability of powerful convex-programming theory and algorithms has enabled a flood of new research in machine learning models and methods. But many of the steps necessary for successful machine learning models fall outside of the convex machine learning paradigm. Thus we now propose framing machine learning problems as Stackelberg games. The resulting bilevel optimization problem allows for efficient systematic search of large numbers of hyper-parameters. We discuss recent progress in solving these bilevel problems and the many interesting optimization challenges that remain. Finally, we investigate the intriguing possibility of novel machine learning models enabled by bilevel programming.
This work was supported in part by the Office of Naval Research under grant no. N00014-06-1-0014. The authors are grateful to Professor Olvi Mangasarian for his suggestions on the penalty approach.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bennett, K., Hu, J., Ji, X., Kunapuli, G., Pang, J.: Model selection via bilevel optimization. In: International Joint Conference on Neural Networks (IJCNN 2006), pp. 1922–1929 (2006)
Kunapuli, G., Bennett, K., Hu, J., Pang, J.: Bilevel model selection for support vector machines. In: Hansen, P., Pardolos, P. (eds.) CRM Proceedings and Lecture Notes. American Mathematical Society (in press, 2008)
Bracken, J., McGill, J.: Mathematical programs with optimization problems in the constraints, vol. 21, pp. 37–44 (1973)
Luo, Z., Pang, J., Ralph, D.: Mathematical Programs With Equilibrium Constraints. Cambridge University Press, Cambridge (1996)
Facchinei, F., Pang, J.: Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer, New York (2003)
Outrata, J., Kocvara, M., Zowe, J.: Nonsmooth Approach to Optimization Problems with Equilibrium Constraints: Theory, Applications and Numerical Results. Kluwer Academic Publishers, Dordrecht (1998)
Dempe, S.: Foundations of Bilevel Programming. Kluwer Academic Publishers, Dordrecht (2002)
Dempe, S.: Annotated bibliography on bilevel programming and mathematical programs with equilibrium constraints. Optimization 52, 333–359 (2003)
Ralph, D., Wright, S.: Some properties of regularization and penalization schemes for mpecs. Optimization Methods and Software 19, 527–556 (2004)
Mangasarian, O.: Misclassification minimization. Journal of Global Optimization 5, 309–323 (1994)
Bennett, K.P., Mangasarian, O.L.: Bilinear separation of two sets in n-space. Computational Optimization and Applications 2, 207–227 (1993)
Fletcher, R., Leyffer, S.: Nonlinear programming without a penalty function. Mathematical Programming 91, 239–269 (2002)
Fletcher, R., Leyffer, S.: User manual for filtersqp Tech. Report NA/181, Department of Mathematics, University of Dundee (1999), http://www-unix.mcs.anl.gov/leyffer/papers/SQP_manual.pdf
Gill, P., Murray, W., Saunders, M.: User’s guide for snopt version 6: A fortran package for large-scale nonlinear programming (2002)
Huang, X., Yang, X., Teo, K.: Partial augmented lagrangian method and mathematical programs with complementarity constraints. Journal of Global Optimization 35, 235–254 (2006)
Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of Machine Learning Research 3, 1157–1182 (2003)
Demiriz, A., Bennett, K., Breneman, C., Embrecht, M.: Support vector regression methods in cheminformatics. Computer Science and Statistics 33 (2001)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Bennett, K.P., Kunapuli, G., Hu, J., Pang, JS. (2008). Bilevel Optimization and Machine Learning. In: Zurada, J.M., Yen, G.G., Wang, J. (eds) Computational Intelligence: Research Frontiers. WCCI 2008. Lecture Notes in Computer Science, vol 5050. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68860-0_2
Download citation
DOI: https://doi.org/10.1007/978-3-540-68860-0_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-68858-7
Online ISBN: 978-3-540-68860-0
eBook Packages: Computer ScienceComputer Science (R0)