Abstract
Proximal bundle methods are a class of optimisation algorithms that leverage the proximal operator to address nonsmoothness in the objective function efficiently. This study focuses on a derivative-free (DFO) proximal bundle method and one of its applications called the DFO \(\mathcal{VU}\)-algorithm. These algorithms incorporate approximate proximal points as subprocedures in order to optimise convex nonsmooth functions based on approximated subdifferential information. Interestingly, the classical \(\mathcal{VU}\)-algorithm, which operates on true subgradient values, achieves superlinear convergence. At each iteration, the algorithm divides the whole space into two: the smooth \(\mathcal{U}\)-space and the nonsmooth \(\mathcal{V}\)-space. It takes a Newton-like step on the \(\mathcal{U}\)-space and a proximal-point step on the \(\mathcal{V}\)-space, enabling it to handle both smooth and nonsmooth parts effectively and converge faster. In this work, we reveal the worst possible convergence rate for the DFO \(\mathcal{VU}\)-method by showing the linear convergence of the DFO proximal bundle method. This will be done by presenting a suitable framework and using the subdifferential-based error bound on the distance to critical points.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Data Availability
Data sets generated during the current study are available from the corresponding author on reasonable request.
References
Absil, P.-A., Hosseini, S.: A collection of nonsmooth Riemannian optimization problems. In: Nonsmooth Optimization and Its Applications. International Series of Numerical Mathematics, vol. 170, pp. 1–15. Springer, Cham (2019)
Artacho, F.J.A., Geoffroy, M.H.: Characterization of metric regularity of subdifferentials. J. Convex Anal. 15, 365–380 (2008)
Atenas, F., Sagastizábal, C., Silva, P.J.S., Solodov, M.: A unified analysis of descent sequences in weakly convex optimization, including convergence rates for bundle methods. SIAM J. Optim. 33(1), 89–115 (2023)
Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137(1–2), 91–129 (2013)
Bagirov, A., Lai, D.T., Palaniswami, M.: A nonsmooth optimization approach to sensor network localization. In: 2007 3rd International Conference on Intelligent Sensors, Sensor Networks and Information, pp. 727–732. IEEE, Melbourne (2007)
Bertails-Descoubes, F., Cadoux, F., Daviet, G., Acary, V.: A nonsmooth Newton solver for capturing exact Coulomb friction in fiber assemblies. ACM Trans. Graph. 30(1), 1–14 (2011)
Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim. 18(2), 556–572 (2007)
Booker, A.J., Dennis, J.E., Frank, P.D., Serafini, D.B., Torczon, V.: Optimization using surrogate objectives on a helicopter test example. In: Computational Methods for Optimal Design and Control, pp. 49–58. Birkhäuser, Boston (1998)
Boulanger, J., Pustelnik, N., Condat, L., Sengmanivong, L., Piolot, T.: Nonsmooth convex optimization for structured illumination microscopy image reconstruction. Inverse Probl. 34(9), 095004 (2018)
Brøndsted, A., Rockafellar, R.T.: On the subdifferentiability of convex functions. Proc. Am. Math. Soc. 16, 605–611 (1965)
Chen, Y., Liu, H., Ye, X., Zhang, Q.: Learnable descent algorithm for nonsmooth nonconvex image reconstruction. SIAM J. Imaging Sci. 14(4), 1532–1564 (2021)
Chiou, S.-W.: A non-smooth model for signalized road network design problems. Appl. Math. Model. 32(7), 1179–1190 (2008)
Clarke, F.H.: Generalized gradients and applications. Trans. Am. Math. Soc. 205, 247–262 (1975)
de Oliveira, W., Sagastizábal, C., Lemaréchal, C.: Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math. Program. 148(1–2), 241–277 (2014)
Drusvyatskiy, D., Lewis, A.S.: Error bounds, quadratic growth, and linear convergence of proximal methods. Math. Oper. Res. 43(3), 919–948 (2018)
Drusvyatskiy, D., Mordukhovich, B.S., Nghia, T.T.A.: Second-order growth, tilt stability, and metric regularity of the subdifferential. J. Convex Anal. 21(4), 1165–1192 (2013)
Dubois, F., Acary, V., Jean, M.: The contact dynamics method: a nonsmooth story. C. R., Méc. 346(3), 247–262 (2018)
Duvigneau, R., Visonneau, M.: Hydrodynamic design using a derivative-free method. Struct. Multidiscip. Optim. 28(2–3), 195–205 (2004)
Hare, W.: Using derivative free optimization for constrained parameter selection in a home and community care forecasting model. In: International Perspectives on Operations Research and Health Care, Proceedings of the 34th Meeting of the EURO Working Group on Operational Research Applied to Health Sciences, pp. 61–73 (2010)
Hare, W.: Numerical analysis of \(\mathcal{VU}\)-decomposition, \(\mathcal{U}\)-gradient, and \(\mathcal{U}\)-Hessian approximations. SIAM J. Optim. 24(4), 1890–1913 (2014)
Hare, W., Nutini, J.: A derivative-free approximate gradient sampling algorithm for finite minimax problems. Comput. Optim. Appl. 56(1), 1–38 (2013)
Hare, W., Planiden, C.: Computing proximal points of convex functions with inexact subgradients. Set-Valued Var. Anal. 26(3), 469–492 (2018)
Hare, W., Sagastizábal, C.: A redistributed proximal bundle method for nonconvex optimization. SIAM J. Optim. 20(5), 2442–2473 (2010)
Hare, W., Planiden, C., Sagastizábal, C.: A derivative-free \(\mathcal{VU}\)-algorithm for convex finite-max problems. Optim. Methods Softw. 35, 521–559 (2019)
Hare, W., Jarry–Bolduc, G., Planiden, C.: Error bounds for overdetermined and underdetermined generalized centred simplex gradients. IMA J. Numer. Anal. 42(1), 744–770 (2022)
Haslinger, J., Miettinen, M., Panagiotopoulos, P.D.: Finite Element Method for Hemivariational Inequalities. Nonconvex Optimization and Its Applications, vol. 35. Springer, Boston (1999)
Helou, E.S., Zibetti, M.V.W., Herman, G.T.: Fast proximal gradient methods for nonsmooth convex optimization for tomographic image reconstruction. Sens. Imag. 21(1), 45 (2020)
Hiriart-Urruty, J.-B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms II: Advanced Theory and Bundle Methods. Grundlehren der Mathematischen Wissenschaften, vol. 306. Springer, Berlin (1993)
Hosseini, S., Pouryayevali, M.R.: Nonsmooth optimization techniques on Riemannian manifolds. J. Optim. Theory Appl. 158(2), 328–342 (2013)
Kiwiel, K.C.: Proximity control in bundle methods for convex nondifferentiable minimization. Math. Program. 46(1), 105–122 (1990)
Kiwiel, K.C.: Approximations in proximal bundle methods and decomposition of convex programs. J. Optim. Theory Appl. 84(3), 529–548 (1995)
Larson, J., Menickelly, M., Wild, S.: Derivative-free optimization methods. Acta Numer. 28, 287–404 (2019)
Lemaréchal, C.: Bundle methods in nonsmooth optimization. In: Nonsmooth Optimization, IIASA Proceedings Series, vol. 3, pp. 79–102. Pergamon Press, London (1978)
Lemaréchal, C., Sagastizábal, C.: Variable metric bundle methods: from conceptual to implementable forms. Math. Program. 76(3), 393–410 (1997)
Lemaréchal, C., Oustry, F., Sagastizábal, C.: The \(\mathcal{U}\)-Lagrangian of a convex function. Trans. Am. Math. Soc. 352, 711–729 (1999)
Linear, S.M.R.: Convergence of epsilon-subgradient descent methods for a class of convex functions. Math. Program. 86, 41–50 (1999)
Liu, H.: Nonsmooth optimization with applications in network design and machine learning. PhD thesis, Pennsylvania State University (2015)
Liu, Q., Wang, J.: A one-layer projection neural network for nonsmooth optimization subject to linear equalities and bound constraints. IEEE Trans. Neural Netw. Learn. Syst. 24(5), 812–824 (2013)
Liu, S., Sagastizábal, C., Solodov, M.: Subdifferential enlargements and continuity properties of the \(\mathcal{VU}\)-decomposition in convex optimization. Nonsmooth Optim. Appl. 170, 55–87 (2019)
Macklin, M., Erleben, K., Müller, M., Chentanez, N., Jeschke, S., Makoviychuk, V.: Non-smooth Newton methods for deformable multi-body dynamics. ACM Trans. Graph. 38(5), 1–20 (2019)
Mäkelä, M.: Survey of bundle methods for nonsmooth optimization. Optim. Methods Softw. 17(1), 1–29 (2002)
Mäkelä, M.M., Karmitsa, N., Bagirov, A.: Subgradient and bundle methods for nonsmooth optimization. In: Numerical Methods for Differential Equations, Optimization, and Technological Problems. Computational Methods in Applied Sciences, vol. 27, pp. 275–304. Springer, Netherlands (2013)
Marsden, A., Feinstein, J., Taylor, C.: A computational framework for derivative-free optimization of cardiovascular geometries. Comput. Methods Appl. Mech. Eng. 197, 1890–1905 (2008)
Martinet, B.: Brève communication. Régularisation d’inéquations variationnelles par approximations successives. Rev. Fr. Inform. Rech. Opér. 4(R3), 154–158 (1970)
Mifflin, R., Sagastizábal, C.: A \(\mathcal{VU}\)-algorithm for convex minimization. Math. Program. 104(2–3), 583–608 (2005)
Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research. Springer, New York (1999)
Planiden, C., Jarry–Bolduc, G., Hare, W.: Hessian approximations (2020). ArXiv preprint. arXiv:2011.02584
Regis, R.G.: The calculus of simplex gradients. Optim. Lett. 9(5), 845–865 (2015)
Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56(3), 1247–1293 (2013)
Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14(5), 877–898 (1976)
Rockafellar, R.T., Wets, R.J.B.: Variational Analysis. Grundlehren der Mathematischen Wissenschaften, vol. 317. Springer, Berlin (1998)
Sagastizábal, C.: A \(\mathcal{VU}\)-point of view of nonsmooth optimization. In: Proceedings of the International Congress of Mathematicians (ICM 2018), Rio de Janeiro, Brazil, pp. 3797–3818 (2019)
Funding
Open Access funding enabled and organized by CAUL and its Member Institutions. No funding was received to assist with the preparation of this manuscript.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing Interests
The authors declare no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Communicated by Claudia Sagastizabal
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Planiden, C., Rajapaksha, T. Linear Convergence of the Derivative-Free Proximal Bundle Method on Convex Nonsmooth Functions, with Application to the Derivative-Free \(\mathcal{VU}\)-Algorithm. Set-Valued Var. Anal 32, 15 (2024). https://doi.org/10.1007/s11228-024-00718-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11228-024-00718-2
Keywords
- Convex optimisation
- Derivative-free optimisation
- Nonsmooth optimisation
- Proximal point
- Bundle methods
- Inexact subgradient
- \(\mathcal{VU}\)-Algorithm
- Error bound
- Linear convergence