Skip to main content
SpringerLink
Log in

Nonsmooth optimization via quasi-Newton methods

  • Full Length Paper
  • Series A
  • Published:
Mathematical Programming Submit manuscript

Abstract

We investigate the behavior of quasi-Newton algorithms applied to minimize a nonsmooth function f, not necessarily convex. We introduce an inexact line search that generates a sequence of nested intervals containing a set of points of nonzero measure that satisfy the Armijo and Wolfe conditions if f is absolutely continuous along the line. Furthermore, the line search is guaranteed to terminate if f is semi-algebraic. It seems quite difficult to establish a convergence theorem for quasi-Newton methods applied to such general classes of functions, so we give a careful analysis of a special but illuminating case, the Euclidean norm, in one variable using the inexact line search and in two variables assuming that the line search is exact. In practice, we find that when f is locally Lipschitz and semi-algebraic with bounded sublevel sets, the BFGS (Broyden–Fletcher–Goldfarb–Shanno) method with the inexact line search almost always generates sequences whose cluster points are Clarke stationary and with function values converging R-linearly to a Clarke stationary value. We give references documenting the successful use of BFGS in a variety of nonsmooth applications, particularly the design of low-order controllers for linear dynamical systems. We conclude with a challenging open question.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Arzelier, D., Gryazina, E.N., Peaucelle, D., Polyak, B.T.: Mixed LMI/randomized methods for static output feedback control design. Technical report 09535, LAAS-CNRS, Toulouse, Sept 2009

  2. Anstreicher K., Lee J.: A masked spectral bound for maximum-entropy sampling. In: di Bucchianico, A., Läuter, H., Wynn eds, H.P. (eds) MODA 7—Advances in Model-Oriented Design and Analysis, pp. 1–10. Springer, Berlin (2004)

    Chapter  Google Scholar 

  3. Boito P., Dedieu J.-P.: The condition metric in the space of rectangular full rank matrices. SIAM J. Matrix Anal. Appl. 31, 2580–2602 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bonnans J., Gilbert J., Lemaréchal C., Sagastizábal C.: A family of variable metric proximal methods. Math. Program. 68, 15–48 (1995)

    MATH  Google Scholar 

  5. Burke, J.V., Henrion, D., Lewis, A.S., Overton, M.L.: HIFOO—a MATLAB package for fixed-order controller design and H ∞ optimization. In: Proceedings of Fifth IFAC Symposium on Robust Control Design, Toulouse (2006)

  6. Borwein J.M., Lewis A.S.: Convex Analysis and Nonlinear Optimization: Theory and Examples. 2nd edn. Springer, New York (2005)

    Google Scholar 

  7. Burke J.V., Lewis A.S., Overton M.L.: A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. 15, 751–779 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  8. Byrd R.H., Nocedal J., Yuan Y.: Global convergence of a class of quasi-Newton methods on convex problems. SIAM J. Numer. Anal. 24, 1171–1190 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  9. Clarke, F.H.: Optimization and Nonsmooth Analysis. John Wiley, New York, 1983. Reprinted by SIAM, Philadelphia (1990)

  10. Dai Y.-H.: Convergence properties of the BFGS algorithm. SIAM J. Optim. 13, 693–701 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  11. Delwiche, T.: Contribution to the design of control laws for bilateral teleoperation with a view to applications in minimally invasive surgery. Ph.D. thesis, Free University of Brussels (2009)

  12. Dotta, D., e Silva, A.S., Decker, I.C.: Design of power systems controllers by nonsmooth, nonconvex optimization. In: IEEE Power and Energy Society General Meeting, Calgary (2009)

  13. Dixon L.C.W.: Quasi-Newton techniques generate identical points. II. The proofs of four new theorems. Math. Program. 3, 345–358 (1972)

    Article  MATH  Google Scholar 

  14. Gumussoy, S., Henrion, D., Millstone, M., Overton, M.L.: Multiobjective robust control with HIFOO 2.0. In: Proceedings of the Sixth IFAC Symposium on Robust Control Design, Haifa (2009)

  15. Gumussoy, S., Millstone, M., Overton, M.L.: H-infinity strong stabilization via HIFOO, a package for fixed-order controller design. In: Proceedings of the 47th IEEE Conference on Decision and Control, Cancun (2008)

  16. Gumussoy, S., Overton, M.L.: Fixed-order H-infinity controller design via HIFOO, a specialized nonsmooth optimization package. In: Proceedings of 2008 American Control Conference, Seattle (2008)

  17. Gürbüzbalaban M., Overton M.L.: On Nesterov’s nonsmooth Chebyshev–Rosenbrock functions. Nonlinear Anal. Theory Methods Appl. 75, 1282–1289 (2012)

    Article  MATH  Google Scholar 

  18. Haarala, M.: Large-scale nonsmooth optimization: variable metric bundle method with limited memory. Ph.D. thesis, University of Jyväskylä, Finland (2004)

  19. Hiriart-Urruty J.B., Lemaréchal C.: Convex Analysis and Minimization Algorithms, two volumes. Springer, New York (1993)

    Google Scholar 

  20. Kaku, A.: Implementation of high precision arithmetic in the BFGS method for nonsmooth optimization. Master’s thesis, New York University, Jan 2011. http://www.cs.nyu.edu/overton/mstheses/kaku/msthesis.pdf

  21. Knittel, D., Henrion, D., Millstone, M., Vedrines, M.: Fixed-order and structure H-infinity control with model based feedforward for elastic web winding systems. In: Proceedings of the IFAC/ IFORS/IMACS/IFIP Symposium on Large Scale Systems, Gdansk, Poland (2007)

  22. Kiwiel, K.C.: Methods of descent for nondifferentiable optimization. In: Lecture Notes in Mathematics, vol. 1133. Springer, Berlin (1985)

  23. Kiwiel K.C.: Convergence of the gradient sampling algorithm for nonsmooth nonconvex optimization. SIAM J. Optim. 18, 379–388 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  24. Lax P.D.: Linear Algebra. Wiley, New York (1997)

    MATH  Google Scholar 

  25. Lee J.: Constrained maximum-entropy sampling. Oper. Res. 46, 655–664 (1998)

    Article  MATH  Google Scholar 

  26. Lemaréchal, C.: A view of line searches. In: Optimization and optimal control (Proceedings of Conference at the Mathematical Research Institute, Oberwolfach, 1980), pp. 59–78. Springer, Berlin/New York, 1981. Lecture Notes in Control and Information Sciences, vol. 30

  27. Lemaréchal, C.: Numerical experiments in nonsmooth optimization. In: Nurminski, E.A. (ed.) Progress in Nondifferentiable Optimization, pp. 61–84. International Institute for Applied Systems Analysis (IIASA), Laxenburg, Austria (1982)

  28. Lewis A.S.: Active sets, nonsmoothness and sensitivity. SIAM J. Optim. 13, 702–725 (2003)

    Article  Google Scholar 

  29. Li D.-H., Fukushima M.: On the global convergence of the BFGS method for nonconvex unconstrained optimization problems. SIAM J. Optim. 11, 1054–1064 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  30. Lewis, A.S., Overton, M.L.: Behavior of BFGS with an exact line search on nonsmooth examples. http://www.cs.nyu.edu/overton/papers/pdffiles/bfgs_exactLS.pdf (2008)

  31. Lewis, A.S., Overton, M.L.: Nonsmooth optimization via BFGS. http://www.cs.nyu.edu/overton/papers/pdffiles/bfgs_inexactLS.pdf (2008)

  32. Lemaréchal C., Oustry F., Sagastizábal C.: The U-Lagrangian of a convex function. Trans. Am. Math. Soc. 352, 711–729 (2000)

    Article  MATH  Google Scholar 

  33. Lemaréchal, C., Sagastizábal, C.: An approach to variable metric bundle methods. In: IFIP Proceedings, Systems Modeling and Optimization (1994)

  34. Lukšan L., Vlček J.: Globally convergent variable metric method for convex nonsmooth unconstrained minimization. J. Optim. Theory Appl. 102, 593–613 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  35. Lukšan, L., Vlček, J.: Variable metric methods for nonsmooth optimization. Technical report 837, Academy of Sciences of the Czech Republic, May 2001

  36. Mascarenhas W.F.: The BFGS method with exact line searches fails for non-convex objective functions. Math. Program. 99, 49–61 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  37. Mifflin R.: An algorithm for constrained optimization with semismooth functions. Math. Oper. Res. 2, 191–207 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  38. Mifflin R., Sun D., Qi L.: Quasi-Newton bundle-type methods for nondifferentiable convex optimization. SIAM J. Optim. 8, 583–603 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  39. Nocedal J., Wright S.J.: Nonlinear Optimization. Springer, New York (2006)

    Google Scholar 

  40. Osting B.: Optimization of spectral functions of Dirichlet-Laplacian eigenvalues. J. Comput. Phys. 229, 8578–8590 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  41. Pouly, G., Lauffenburger, J.-P., Basset, M.: Reduced order H-infinity control design of a nose landing gear steering system. In: Proceedings of 12th IFAC Symposium on Control in Transportation Systems (2010)

  42. Powell, M.J.D.: Some global convergence properties of a variable metric algorithm for minimization without exact line searches. In: Nonlinear Programming, pp. 53–72. American Mathematical Society, Providence. SIAM-AMS Proceedings, vol. IX (1976)

  43. Rauf A.I., Fukushima M.: Globally convergent BFGS method for nonsmooth convex optimization. J. Optim. Theory Appl. 104, 539–558 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  44. Royden H.L.: Real Analysis. Macmillan, New York (1963)

    MATH  Google Scholar 

  45. Rockafellar R.T., Wets R.J.B.: Variational Analysis. Springer, New York (1998)

    Book  MATH  Google Scholar 

  46. Sagastizábal, C.: Composite proximal bundle method. Technical report. http://www.optimization-online.org/DB_HTML/2009/07/2356.html

  47. Skajaa, A.: Limited memory BFGS for nonsmooth optimization. Master’s thesis, New York University, Jan 2010. http://www.cs.nyu.edu/overton/mstheses/skajaa/msthesis.pdf

  48. Vlček J., Lukšan L.: Globally convergent variable metric method for nonconvex nondifferentiable unconstrained minimization. J. Optim. Theory Appl. 111, 407–430 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  49. Wang F.-C., Chen H.-T.: Design and implementation of fixed-order robust controllers for a proton exchange membrane fuel cell system. Int. J. Hydrogen Energy 34, 2705–2717 (2009)

    Article  Google Scholar 

  50. Wildschek, A., Maier, R., Hromcik, M., Hanis, T., Schirrer, A., Kozek, M., Westermayer, C., Hemedi M.: Hybrid controller for gust load alleviation and ride comfort improvement using direct lift control flaps. In: Proceedings of Third European Conference for Aerospace Sciences (EUCASS) (2009)

  51. Wolfe, P.: A method of conjugate subgradients for minimizing nondifferentiable functions. Math. Program. Stud. 3, 145–173, (1975). In: Balinski, M.L., Wolfe, P. (eds.) Nondifferentiable Optimization

  52. Yu, J., Vishwanathan, S.V.N., Günther, S., Schraudolph, N.: A quasi-Newton approach to non-smooth convex optimization. In: Proceedings of the 25th International Conference on Machine Learning (2008)

  53. Zhang, S.S.: Cornell University, Private Communication (2010)

  54. Zhang S., Zou X., Ahlquist J., Navon I.M., Sela J.G.: Use of differentiable and nondifferentiable optimization algorithms for variational data assimilation with discontinuous cost functions. Mon. Weather Rev. 128, 4031–4044 (2000)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Operations Research and Information Engineering, Cornell University, Ithaca, NY, 14853, USA

    Adrian S. Lewis

  2. Courant Institute of Mathematical Sciences, New York University, New York, NY, 10012, USA

    Michael L. Overton

Authors
  1. Adrian S. Lewis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael L. Overton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael L. Overton.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lewis, A.S., Overton, M.L. Nonsmooth optimization via quasi-Newton methods. Math. Program. 141, 135–163 (2013). https://doi.org/10.1007/s10107-012-0514-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-012-0514-2

Keywords

Mathematics Subject Classification (2000)

Search

Navigation