Mathematical Programming

, Volume 141, Issue 1–2, pp 135–163 | Cite as

Nonsmooth optimization via quasi-Newton methods

Full Length Paper Series A

Abstract

We investigate the behavior of quasi-Newton algorithms applied to minimize a nonsmooth function f, not necessarily convex. We introduce an inexact line search that generates a sequence of nested intervals containing a set of points of nonzero measure that satisfy the Armijo and Wolfe conditions if f is absolutely continuous along the line. Furthermore, the line search is guaranteed to terminate if f is semi-algebraic. It seems quite difficult to establish a convergence theorem for quasi-Newton methods applied to such general classes of functions, so we give a careful analysis of a special but illuminating case, the Euclidean norm, in one variable using the inexact line search and in two variables assuming that the line search is exact. In practice, we find that when f is locally Lipschitz and semi-algebraic with bounded sublevel sets, the BFGS (Broyden–Fletcher–Goldfarb–Shanno) method with the inexact line search almost always generates sequences whose cluster points are Clarke stationary and with function values converging R-linearly to a Clarke stationary value. We give references documenting the successful use of BFGS in a variety of nonsmooth applications, particularly the design of low-order controllers for linear dynamical systems. We conclude with a challenging open question.

Keywords

BFGS Nonconvex Line search R-linear convergence Clarke stationary Partly smooth 

Mathematics Subject Classification (2000)

90C30 65K05 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Arzelier, D., Gryazina, E.N., Peaucelle, D., Polyak, B.T.: Mixed LMI/randomized methods for static output feedback control design. Technical report 09535, LAAS-CNRS, Toulouse, Sept 2009Google Scholar
  2. 2.
    Anstreicher K., Lee J.: A masked spectral bound for maximum-entropy sampling. In: di Bucchianico, A., Läuter, H., Wynn eds, H.P. (eds) MODA 7—Advances in Model-Oriented Design and Analysis, pp. 1–10. Springer, Berlin (2004)CrossRefGoogle Scholar
  3. 3.
    Boito P., Dedieu J.-P.: The condition metric in the space of rectangular full rank matrices. SIAM J. Matrix Anal. Appl. 31, 2580–2602 (2010)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Bonnans J., Gilbert J., Lemaréchal C., Sagastizábal C.: A family of variable metric proximal methods. Math. Program. 68, 15–48 (1995)MATHGoogle Scholar
  5. 5.
    Burke, J.V., Henrion, D., Lewis, A.S., Overton, M.L.: HIFOO—a MATLAB package for fixed-order controller design and H optimization. In: Proceedings of Fifth IFAC Symposium on Robust Control Design, Toulouse (2006)Google Scholar
  6. 6.
    Borwein J.M., Lewis A.S.: Convex Analysis and Nonlinear Optimization: Theory and Examples. 2nd edn. Springer, New York (2005)Google Scholar
  7. 7.
    Burke J.V., Lewis A.S., Overton M.L.: A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. 15, 751–779 (2005)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Byrd R.H., Nocedal J., Yuan Y.: Global convergence of a class of quasi-Newton methods on convex problems. SIAM J. Numer. Anal. 24, 1171–1190 (1987)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Clarke, F.H.: Optimization and Nonsmooth Analysis. John Wiley, New York, 1983. Reprinted by SIAM, Philadelphia (1990)Google Scholar
  10. 10.
    Dai Y.-H.: Convergence properties of the BFGS algorithm. SIAM J. Optim. 13, 693–701 (2002)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Delwiche, T.: Contribution to the design of control laws for bilateral teleoperation with a view to applications in minimally invasive surgery. Ph.D. thesis, Free University of Brussels (2009)Google Scholar
  12. 12.
    Dotta, D., e Silva, A.S., Decker, I.C.: Design of power systems controllers by nonsmooth, nonconvex optimization. In: IEEE Power and Energy Society General Meeting, Calgary (2009)Google Scholar
  13. 13.
    Dixon L.C.W.: Quasi-Newton techniques generate identical points. II. The proofs of four new theorems. Math. Program. 3, 345–358 (1972)CrossRefMATHGoogle Scholar
  14. 14.
    Gumussoy, S., Henrion, D., Millstone, M., Overton, M.L.: Multiobjective robust control with HIFOO 2.0. In: Proceedings of the Sixth IFAC Symposium on Robust Control Design, Haifa (2009)Google Scholar
  15. 15.
    Gumussoy, S., Millstone, M., Overton, M.L.: H-infinity strong stabilization via HIFOO, a package for fixed-order controller design. In: Proceedings of the 47th IEEE Conference on Decision and Control, Cancun (2008)Google Scholar
  16. 16.
    Gumussoy, S., Overton, M.L.: Fixed-order H-infinity controller design via HIFOO, a specialized nonsmooth optimization package. In: Proceedings of 2008 American Control Conference, Seattle (2008)Google Scholar
  17. 17.
    Gürbüzbalaban M., Overton M.L.: On Nesterov’s nonsmooth Chebyshev–Rosenbrock functions. Nonlinear Anal. Theory Methods Appl. 75, 1282–1289 (2012)CrossRefMATHGoogle Scholar
  18. 18.
    Haarala, M.: Large-scale nonsmooth optimization: variable metric bundle method with limited memory. Ph.D. thesis, University of Jyväskylä, Finland (2004)Google Scholar
  19. 19.
    Hiriart-Urruty J.B., Lemaréchal C.: Convex Analysis and Minimization Algorithms, two volumes. Springer, New York (1993)Google Scholar
  20. 20.
    Kaku, A.: Implementation of high precision arithmetic in the BFGS method for nonsmooth optimization. Master’s thesis, New York University, Jan 2011. http://www.cs.nyu.edu/overton/mstheses/kaku/msthesis.pdf
  21. 21.
    Knittel, D., Henrion, D., Millstone, M., Vedrines, M.: Fixed-order and structure H-infinity control with model based feedforward for elastic web winding systems. In: Proceedings of the IFAC/ IFORS/IMACS/IFIP Symposium on Large Scale Systems, Gdansk, Poland (2007)Google Scholar
  22. 22.
    Kiwiel, K.C.: Methods of descent for nondifferentiable optimization. In: Lecture Notes in Mathematics, vol. 1133. Springer, Berlin (1985)Google Scholar
  23. 23.
    Kiwiel K.C.: Convergence of the gradient sampling algorithm for nonsmooth nonconvex optimization. SIAM J. Optim. 18, 379–388 (2007)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Lax P.D.: Linear Algebra. Wiley, New York (1997)MATHGoogle Scholar
  25. 25.
    Lee J.: Constrained maximum-entropy sampling. Oper. Res. 46, 655–664 (1998)CrossRefMATHGoogle Scholar
  26. 26.
    Lemaréchal, C.: A view of line searches. In: Optimization and optimal control (Proceedings of Conference at the Mathematical Research Institute, Oberwolfach, 1980), pp. 59–78. Springer, Berlin/New York, 1981. Lecture Notes in Control and Information Sciences, vol. 30Google Scholar
  27. 27.
    Lemaréchal, C.: Numerical experiments in nonsmooth optimization. In: Nurminski, E.A. (ed.) Progress in Nondifferentiable Optimization, pp. 61–84. International Institute for Applied Systems Analysis (IIASA), Laxenburg, Austria (1982)Google Scholar
  28. 28.
    Lewis A.S.: Active sets, nonsmoothness and sensitivity. SIAM J. Optim. 13, 702–725 (2003)CrossRefGoogle Scholar
  29. 29.
    Li D.-H., Fukushima M.: On the global convergence of the BFGS method for nonconvex unconstrained optimization problems. SIAM J. Optim. 11, 1054–1064 (2001)MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Lewis, A.S., Overton, M.L.: Behavior of BFGS with an exact line search on nonsmooth examples. http://www.cs.nyu.edu/overton/papers/pdffiles/bfgs_exactLS.pdf (2008)
  31. 31.
    Lewis, A.S., Overton, M.L.: Nonsmooth optimization via BFGS. http://www.cs.nyu.edu/overton/papers/pdffiles/bfgs_inexactLS.pdf (2008)
  32. 32.
    Lemaréchal C., Oustry F., Sagastizábal C.: The U-Lagrangian of a convex function. Trans. Am. Math. Soc. 352, 711–729 (2000)CrossRefMATHGoogle Scholar
  33. 33.
    Lemaréchal, C., Sagastizábal, C.: An approach to variable metric bundle methods. In: IFIP Proceedings, Systems Modeling and Optimization (1994)Google Scholar
  34. 34.
    Lukšan L., Vlček J.: Globally convergent variable metric method for convex nonsmooth unconstrained minimization. J. Optim. Theory Appl. 102, 593–613 (1999)MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Lukšan, L., Vlček, J.: Variable metric methods for nonsmooth optimization. Technical report 837, Academy of Sciences of the Czech Republic, May 2001Google Scholar
  36. 36.
    Mascarenhas W.F.: The BFGS method with exact line searches fails for non-convex objective functions. Math. Program. 99, 49–61 (2004)MathSciNetCrossRefMATHGoogle Scholar
  37. 37.
    Mifflin R.: An algorithm for constrained optimization with semismooth functions. Math. Oper. Res. 2, 191–207 (1977)MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Mifflin R., Sun D., Qi L.: Quasi-Newton bundle-type methods for nondifferentiable convex optimization. SIAM J. Optim. 8, 583–603 (1998)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Nocedal J., Wright S.J.: Nonlinear Optimization. Springer, New York (2006)Google Scholar
  40. 40.
    Osting B.: Optimization of spectral functions of Dirichlet-Laplacian eigenvalues. J. Comput. Phys. 229, 8578–8590 (2010)MathSciNetCrossRefMATHGoogle Scholar
  41. 41.
    Pouly, G., Lauffenburger, J.-P., Basset, M.: Reduced order H-infinity control design of a nose landing gear steering system. In: Proceedings of 12th IFAC Symposium on Control in Transportation Systems (2010)Google Scholar
  42. 42.
    Powell, M.J.D.: Some global convergence properties of a variable metric algorithm for minimization without exact line searches. In: Nonlinear Programming, pp. 53–72. American Mathematical Society, Providence. SIAM-AMS Proceedings, vol. IX (1976)Google Scholar
  43. 43.
    Rauf A.I., Fukushima M.: Globally convergent BFGS method for nonsmooth convex optimization. J. Optim. Theory Appl. 104, 539–558 (2000)MathSciNetCrossRefMATHGoogle Scholar
  44. 44.
    Royden H.L.: Real Analysis. Macmillan, New York (1963)MATHGoogle Scholar
  45. 45.
    Rockafellar R.T., Wets R.J.B.: Variational Analysis. Springer, New York (1998)CrossRefMATHGoogle Scholar
  46. 46.
    Sagastizábal, C.: Composite proximal bundle method. Technical report. http://www.optimization-online.org/DB_HTML/2009/07/2356.html
  47. 47.
    Skajaa, A.: Limited memory BFGS for nonsmooth optimization. Master’s thesis, New York University, Jan 2010. http://www.cs.nyu.edu/overton/mstheses/skajaa/msthesis.pdf
  48. 48.
    Vlček J., Lukšan L.: Globally convergent variable metric method for nonconvex nondifferentiable unconstrained minimization. J. Optim. Theory Appl. 111, 407–430 (2001)MathSciNetCrossRefMATHGoogle Scholar
  49. 49.
    Wang F.-C., Chen H.-T.: Design and implementation of fixed-order robust controllers for a proton exchange membrane fuel cell system. Int. J. Hydrogen Energy 34, 2705–2717 (2009)CrossRefGoogle Scholar
  50. 50.
    Wildschek, A., Maier, R., Hromcik, M., Hanis, T., Schirrer, A., Kozek, M., Westermayer, C., Hemedi M.: Hybrid controller for gust load alleviation and ride comfort improvement using direct lift control flaps. In: Proceedings of Third European Conference for Aerospace Sciences (EUCASS) (2009)Google Scholar
  51. 51.
    Wolfe, P.: A method of conjugate subgradients for minimizing nondifferentiable functions. Math. Program. Stud. 3, 145–173, (1975). In: Balinski, M.L., Wolfe, P. (eds.) Nondifferentiable OptimizationGoogle Scholar
  52. 52.
    Yu, J., Vishwanathan, S.V.N., Günther, S., Schraudolph, N.: A quasi-Newton approach to non-smooth convex optimization. In: Proceedings of the 25th International Conference on Machine Learning (2008)Google Scholar
  53. 53.
    Zhang, S.S.: Cornell University, Private Communication (2010)Google Scholar
  54. 54.
    Zhang S., Zou X., Ahlquist J., Navon I.M., Sela J.G.: Use of differentiable and nondifferentiable optimization algorithms for variational data assimilation with discontinuous cost functions. Mon. Weather Rev. 128, 4031–4044 (2000)CrossRefGoogle Scholar

Copyright information

© Springer and Mathematical Optimization Society 2012

Authors and Affiliations

  1. 1.School of Operations Research and Information EngineeringCornell UniversityIthacaUSA
  2. 2.Courant Institute of Mathematical SciencesNew York UniversityNew YorkUSA

Personalised recommendations