Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Diagonal Bundle Method for Nonsmooth Sparse Optimization

  • 288 Accesses

  • 7 Citations

Abstract

We propose an efficient diagonal bundle method for sparse nonsmooth, possibly nonconvex optimization. The convergence of the proposed method is proved for locally Lipschitz continuous functions, which are not necessary differentiable or convex. The numerical experiments have been made using problems with up to million variables. The results to be presented confirm the usability of the diagonal bundle method especially for extremely large-scale problems.

This is a preview of subscription content, log in to check access.

References

  1. 1.

    Outrata, J., Kočvara, M., Zowe, J.: Nonsmooth Approach to Optimization Problems with Equilibrium Constraints. Theory, Applications and Numerical Results. Kluwert Academic Publisher, Dordrecht (1998)

  2. 2.

    Moreau, J.J., Panagiotopoulos, P.D., Strang, G. (eds.): Topics in Nonsmooth Mechanics. Birkhäuser Verlag, Basel (1988)

  3. 3.

    Mistakidis, E.S., Stavroulakis, G.E.: Nonconvex Optimization in Mechanics. Smooth and Nonsmooth Algorithms, Heuristics and Engineering Applications by the F.E.M. Kluwert Academic Publishers, Dordrecht (1998)

  4. 4.

    Clarke, F.H., Ledyaev, Y.S., Stern, R.J., Wolenski, P.R.: Nonsmooth Analysis and Control Theory. Springer, New York (1998)

  5. 5.

    Haslinger, J., Neittaanmäki, P.: Finite Element Approximation for Optimal Shape, Material and Topology Design, 2nd edn. John Wiley & Sons, Chichester (1996)

  6. 6.

    Kärkkäinen, T., Heikkola, E.: Robust formulations for training multilayer perceptrons. Neural. Comput. 16, 837–862 (2004)

  7. 7.

    Äyrämö, S.: Knowledge mining using robust clustering. Ph.D. thesis, University of Jyväskylä, Department of Mathematical Information Technology (2006).

  8. 8.

    Bradley, P.S., Fayyad, U.M., Mangasarian, O.L.: Mathematical programming for data mining: formulations and challenges. INFORMS J. Comput. 11, 217–238 (1999)

  9. 9.

    Demyanov, V.F., Bagirov, A.M., Rubinov, A.M.: A method of truncated codifferential with application to some problems of cluster analysis. J. Glob. Optim. 23(1), 63–80 (2002)

  10. 10.

    Astorino, A., Fuduli, A.: Nonsmooth optimization techniques for semi-supervised classification. IEEE Trans. Pattern. Anal. Mach. Intell. 29(12), 2135–2142 (2007)

  11. 11.

    Astorino, A., Fuduli, A., Gorgone, E.: Nonsmoothness in classification problems. Optim. Methods Softw. 23(5), 675–688 (2008)

  12. 12.

    Bergeron, C., Moore, G., Zaretzki, J., Breneman, C., Bennett, K.: Fast bundle algorithm for multiple instance learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(6), 1068–1079 (2012).

  13. 13.

    Carrizosa, E., Morales, D.R.: Supervised classification and mathematical optimization. Comput. Oper. Res. 40(1), 150–165 (2013)

  14. 14.

    Lemaréchal, C.: Nondifferentiable optimization. In: G.L. Nemhauser, A.H.G. Rinnooy Kan, M.J. Todd (eds.) Optimization, pp. 529–572. Elsevier North-Holland Inc, New York (1989).

  15. 15.

    Beck, A., Teboulle, M.: Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. 31(3), 167–175 (2003)

  16. 16.

    Ben-Tal, A., Nemirovski, A.: Non-Euclidean restricted memory level method for large-scale convex optimization. Math. Program. 102(3), 407–456 (2005)

  17. 17.

    Shor, N.Z.: Minimization Methods for Non-Differentiable Functions. Springer-Verlag, Berlin (1985)

  18. 18.

    Uryasev, S.P.: Algorithms for nondifferentiable optimization problems. J. Optim. Theory Appl. 71, 359–388 (1991)

  19. 19.

    Gaudioso, M., Monaco, M.F.: Variants to the cutting plane approach for convex nondifferentiable optimization. Optimization 25, 65–75 (1992)

  20. 20.

    Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms II. Springer-Verlag, Berlin (1993)

  21. 21.

    Kiwiel, K.C.: Methods of Descent for Nondifferentiable Optimization. Lecture Notes in Mathematics 1133. Springer-Verlag, Berlin (1985).

  22. 22.

    Lukšan, L., Vlček, J.: Globally convergent variable metric method for convex nonsmooth unconstrained minimization. J. Optim. Theory Appl. 102(3), 593–613 (1999)

  23. 23.

    Mäkelä, M.M.: Survey of bundle methods for nonsmooth optimization. Optim. Methods Softw. 17(1), 1–29 (2002)

  24. 24.

    Mäkelä, M.M., Neittaanmäki, P.: Nonsmooth Optimization: Analysis and Algorithms with Applications to Optimal Control. World Scientific Publishing Co., Singapore (1992)

  25. 25.

    Sagastizábal, C., Solodov, M.: An infeasible bundle method for nonsmooth convex constrained optimization without a penalty function or a filter. SIAM J. Optim. 16(1), 146–169 (2005)

  26. 26.

    Schramm, H., Zowe, J.: A version of the bundle idea for minimizing a nonsmooth function: conceptual idea, convergence analysis, numerical results. SIAM J. Optim. 2(1), 121–152 (1992)

  27. 27.

    Polak, E., Royset, J.O.: Algorithms for finite and semi-finite min-max-min problems using adaptive smoothing techniques. J. Optim. Theory Appl. 119, 421–457 (2003)

  28. 28.

    Burke, J.V., Lewis, A.S., Overton, M.L.: A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. 15, 751–779 (2005)

  29. 29.

    Apkarian, P., Noll, D., Prot, O.: A trust region spectral bundle method for non-convex eigenvalue optimization. SIAM J. Optim. 19(1), 281–306 (2008)

  30. 30.

    Fuduli, A., Gaudioso, M., Nurminski, E.A.: A splitting bundle approach for non-smooth non-convex minimization. Optimization, (2013, in press ), doi:10.1080/02331934.2013.840625.

  31. 31.

    Fuduli, A., Gaudioso, M., Giallombardo, G.: Minimizing nonconvex nonsmooth functions via cutting planes and proximity control. SIAM J. Optim. 14(3), 743–756 (2004)

  32. 32.

    Hare, W., Sagastizábal, C.: A redistributed proximal bundle method for nonconvex optimization. SIAM J. Optim. 20(5), 2442–2473 (2010)

  33. 33.

    Karmitsa, N., Tanaka Filho, M., Herskovits, J.: Globally convergent cutting plane method for nonconvex nonsmooth minimization. J. Optim. Theory Appl. 148(3), 528–549 (2011)

  34. 34.

    Noll, D., Prot, O., Rondepierre, A.: A proximity control algorithm to minimize nonsmooth and nonconvex functions. Pac. J. Optim. 4(3), 571–604 (2008)

  35. 35.

    Haarala, M.: Large-scale nonsmooth optimization: Variable metric bundle method with limited memory. Ph.D. thesis, University of Jyväskylä, Department of Mathematical Information Technology (2004).

  36. 36.

    Haarala, M., Miettinen, K., Mäkelä, M.M.: New limited memory bundle method for large-scale nonsmooth optimization. Optim. Methods Softw. 19(6), 673–692 (2004)

  37. 37.

    Haarala, N., Miettinen, K., Mäkelä, M.M.: Globally convergent limited memory bundle method for large-scale nonsmooth optimization. Math. Program. 109(1), 181–205 (2007)

  38. 38.

    Bagirov, A.M., Karmitsa, N., Mäkelä, M.M.: Introduction to Nonsmooth Optimization: Theory, Practice and Software. Springer International Publishing (2014).

  39. 39.

    Karmitsa, N., Bagirov, A., Mäkelä, M.M.: Comparing different nonsmooth optimization methods and software. Optim. Methods Softw. 27(1), 131–153 (2012)

  40. 40.

    Herskovits, J., Goulart, E.: Sparse quasi-Newton matrices for large scale nonlinear optimization. In: Proceedings of the 6th Word Congress on Structurla and Multidisciplinary Optimization (2005).

  41. 41.

    Clarke, F.H.: Optimization and Nonsmooth Analysis. Wiley-Interscience, New York (1983)

  42. 42.

    Karmitsa, N., Mäkelä, M.M., Ali, M.M.: Limited memory interior point bundle method for large inequality constrained nonsmooth minimization. Appl. Math. Comput. 198(1), 382–400 (2008)

  43. 43.

    Vlček, J., Lukšan, L.: Globally convergent variable metric method for nonconvex nondifferentiable unconstrained minimization. J. Optim. Theory Appl. 111(2), 407–430 (2001)

  44. 44.

    Byrd, R.H., Nocedal, J., Schnabel, R.B.: Representations of quasi-Newton matrices and their use in limited memory methods. Math. Program. 63, 129–156 (1994)

  45. 45.

    Lemaréchal, C., Strodiot, J.J., Bihain, A.: On a bundle algorithm for nonsmooth optimization. In: Mangasarian, O.L., Mayer, R.R., Robinson, S.M. (eds.) Nonlinear Programming, pp. 245–281. Academic Press, New York (1981)

  46. 46.

    Mifflin, R.: A modification and an extension of Lemaréchal’s algorithm for nonsmooth minimization. Mat. Program. Study 17, 77–90 (1982)

  47. 47.

    Bihain, A.: Optimization of upper semidifferentiable functions. J. Optim. Theory Appl. 4, 545–568 (1984)

  48. 48.

    Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, Chichester (1987)

  49. 49.

    Nocedal, J.: Updating Quasi-Newton matrices with limited storage. Math. Comput. 35(151), 773–782 (1980)

  50. 50.

    Fletcher, R.: An optimal positive definite update for sparse Hessian matrices. SIAM J. Optim. 5(1), 192–218 (1995)

  51. 51.

    Fletcher, R., Grothey, A., Leyffer, S.: Computing sparse Hessian and Jacobian approximations with optimal hereditary properties. University of Dundee Numerical Analysis Report NA/164 (1995).

  52. 52.

    Toint, P.L.: On sparse and symmetric matrix updating subject to a linear equation. Math. Comput. 31(140), 954–961 (1977)

  53. 53.

    Mäkelä, M.M.: Multiobjective proximal bundle method for nonconvex nonsmooth optimization: Fortran subroutine MPBNGC 2.0. Reports of the Department of Mathematical Information Technology, Series B. Scientific Computing, B. 13/2003 University of Jyväskylä, Jyväskylä (2003).

  54. 54.

    Bagirov, A.M., Ganjehlou, A.N.: A secant method for nonsmooth optimization. Submitted (2009).

  55. 55.

    Bagirov, A.M., Ganjehlou, A.N.: A quasisecant method for minimizing nonsmooth functions. Optim. Methods Softw. 25(1), 3–18 (2010)

  56. 56.

    Lukšan, L.: Dual method for solving a special problem of quadratic programming as a subproblem at linearly constrained nonlinear minimax approximation. Kybernetika 20, 445–457 (1984)

  57. 57.

    Miettinen, K., Mäkelä, M.M.: Synchronous approach in interactive multiobjective optimization. Eur. J. Oper. Res. 170(3), 909–922 (2006)

Download references

Acknowledgments

The work was financially supported by the University of Turku (Finland) and Magnus Ehrnrooth foundation.

Author information

Correspondence to Napsu Karmitsa.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Karmitsa, N. Diagonal Bundle Method for Nonsmooth Sparse Optimization. J Optim Theory Appl 166, 889–905 (2015). https://doi.org/10.1007/s10957-014-0666-8

Download citation

Keywords

  • Nondifferentiable optimization
  • Sparse problems
  • Bundle methods
  • Diagonal variable metric methods

Mathematics Subject Classification

  • 65K05
  • 90C25