Journal of Optimization Theory and Applications

, Volume 166, Issue 3, pp 889–905 | Cite as

Diagonal Bundle Method for Nonsmooth Sparse Optimization

  • Napsu Karmitsa


We propose an efficient diagonal bundle method for sparse nonsmooth, possibly nonconvex optimization. The convergence of the proposed method is proved for locally Lipschitz continuous functions, which are not necessary differentiable or convex. The numerical experiments have been made using problems with up to million variables. The results to be presented confirm the usability of the diagonal bundle method especially for extremely large-scale problems.


Nondifferentiable optimization Sparse problems Bundle methods Diagonal variable metric methods 

Mathematics Subject Classification

65K05 90C25 



The work was financially supported by the University of Turku (Finland) and Magnus Ehrnrooth foundation.


  1. 1.
    Outrata, J., Kočvara, M., Zowe, J.: Nonsmooth Approach to Optimization Problems with Equilibrium Constraints. Theory, Applications and Numerical Results. Kluwert Academic Publisher, Dordrecht (1998)CrossRefzbMATHGoogle Scholar
  2. 2.
    Moreau, J.J., Panagiotopoulos, P.D., Strang, G. (eds.): Topics in Nonsmooth Mechanics. Birkhäuser Verlag, Basel (1988)zbMATHGoogle Scholar
  3. 3.
    Mistakidis, E.S., Stavroulakis, G.E.: Nonconvex Optimization in Mechanics. Smooth and Nonsmooth Algorithms, Heuristics and Engineering Applications by the F.E.M. Kluwert Academic Publishers, Dordrecht (1998)Google Scholar
  4. 4.
    Clarke, F.H., Ledyaev, Y.S., Stern, R.J., Wolenski, P.R.: Nonsmooth Analysis and Control Theory. Springer, New York (1998)zbMATHGoogle Scholar
  5. 5.
    Haslinger, J., Neittaanmäki, P.: Finite Element Approximation for Optimal Shape, Material and Topology Design, 2nd edn. John Wiley & Sons, Chichester (1996)zbMATHGoogle Scholar
  6. 6.
    Kärkkäinen, T., Heikkola, E.: Robust formulations for training multilayer perceptrons. Neural. Comput. 16, 837–862 (2004)CrossRefzbMATHGoogle Scholar
  7. 7.
    Äyrämö, S.: Knowledge mining using robust clustering. Ph.D. thesis, University of Jyväskylä, Department of Mathematical Information Technology (2006).Google Scholar
  8. 8.
    Bradley, P.S., Fayyad, U.M., Mangasarian, O.L.: Mathematical programming for data mining: formulations and challenges. INFORMS J. Comput. 11, 217–238 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Demyanov, V.F., Bagirov, A.M., Rubinov, A.M.: A method of truncated codifferential with application to some problems of cluster analysis. J. Glob. Optim. 23(1), 63–80 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Astorino, A., Fuduli, A.: Nonsmooth optimization techniques for semi-supervised classification. IEEE Trans. Pattern. Anal. Mach. Intell. 29(12), 2135–2142 (2007)CrossRefGoogle Scholar
  11. 11.
    Astorino, A., Fuduli, A., Gorgone, E.: Nonsmoothness in classification problems. Optim. Methods Softw. 23(5), 675–688 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Bergeron, C., Moore, G., Zaretzki, J., Breneman, C., Bennett, K.: Fast bundle algorithm for multiple instance learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(6), 1068–1079 (2012).Google Scholar
  13. 13.
    Carrizosa, E., Morales, D.R.: Supervised classification and mathematical optimization. Comput. Oper. Res. 40(1), 150–165 (2013)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Lemaréchal, C.: Nondifferentiable optimization. In: G.L. Nemhauser, A.H.G. Rinnooy Kan, M.J. Todd (eds.) Optimization, pp. 529–572. Elsevier North-Holland Inc, New York (1989).Google Scholar
  15. 15.
    Beck, A., Teboulle, M.: Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. 31(3), 167–175 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Ben-Tal, A., Nemirovski, A.: Non-Euclidean restricted memory level method for large-scale convex optimization. Math. Program. 102(3), 407–456 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Shor, N.Z.: Minimization Methods for Non-Differentiable Functions. Springer-Verlag, Berlin (1985)CrossRefzbMATHGoogle Scholar
  18. 18.
    Uryasev, S.P.: Algorithms for nondifferentiable optimization problems. J. Optim. Theory Appl. 71, 359–388 (1991)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Gaudioso, M., Monaco, M.F.: Variants to the cutting plane approach for convex nondifferentiable optimization. Optimization 25, 65–75 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms II. Springer-Verlag, Berlin (1993)zbMATHGoogle Scholar
  21. 21.
    Kiwiel, K.C.: Methods of Descent for Nondifferentiable Optimization. Lecture Notes in Mathematics 1133. Springer-Verlag, Berlin (1985).Google Scholar
  22. 22.
    Lukšan, L., Vlček, J.: Globally convergent variable metric method for convex nonsmooth unconstrained minimization. J. Optim. Theory Appl. 102(3), 593–613 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Mäkelä, M.M.: Survey of bundle methods for nonsmooth optimization. Optim. Methods Softw. 17(1), 1–29 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Mäkelä, M.M., Neittaanmäki, P.: Nonsmooth Optimization: Analysis and Algorithms with Applications to Optimal Control. World Scientific Publishing Co., Singapore (1992)CrossRefzbMATHGoogle Scholar
  25. 25.
    Sagastizábal, C., Solodov, M.: An infeasible bundle method for nonsmooth convex constrained optimization without a penalty function or a filter. SIAM J. Optim. 16(1), 146–169 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Schramm, H., Zowe, J.: A version of the bundle idea for minimizing a nonsmooth function: conceptual idea, convergence analysis, numerical results. SIAM J. Optim. 2(1), 121–152 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Polak, E., Royset, J.O.: Algorithms for finite and semi-finite min-max-min problems using adaptive smoothing techniques. J. Optim. Theory Appl. 119, 421–457 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Burke, J.V., Lewis, A.S., Overton, M.L.: A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. 15, 751–779 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Apkarian, P., Noll, D., Prot, O.: A trust region spectral bundle method for non-convex eigenvalue optimization. SIAM J. Optim. 19(1), 281–306 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Fuduli, A., Gaudioso, M., Nurminski, E.A.: A splitting bundle approach for non-smooth non-convex minimization. Optimization, (2013, in press ), doi: 10.1080/02331934.2013.840625.
  31. 31.
    Fuduli, A., Gaudioso, M., Giallombardo, G.: Minimizing nonconvex nonsmooth functions via cutting planes and proximity control. SIAM J. Optim. 14(3), 743–756 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Hare, W., Sagastizábal, C.: A redistributed proximal bundle method for nonconvex optimization. SIAM J. Optim. 20(5), 2442–2473 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Karmitsa, N., Tanaka Filho, M., Herskovits, J.: Globally convergent cutting plane method for nonconvex nonsmooth minimization. J. Optim. Theory Appl. 148(3), 528–549 (2011)Google Scholar
  34. 34.
    Noll, D., Prot, O., Rondepierre, A.: A proximity control algorithm to minimize nonsmooth and nonconvex functions. Pac. J. Optim. 4(3), 571–604 (2008)MathSciNetzbMATHGoogle Scholar
  35. 35.
    Haarala, M.: Large-scale nonsmooth optimization: Variable metric bundle method with limited memory. Ph.D. thesis, University of Jyväskylä, Department of Mathematical Information Technology (2004).Google Scholar
  36. 36.
    Haarala, M., Miettinen, K., Mäkelä, M.M.: New limited memory bundle method for large-scale nonsmooth optimization. Optim. Methods Softw. 19(6), 673–692 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Haarala, N., Miettinen, K., Mäkelä, M.M.: Globally convergent limited memory bundle method for large-scale nonsmooth optimization. Math. Program. 109(1), 181–205 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Bagirov, A.M., Karmitsa, N., Mäkelä, M.M.: Introduction to Nonsmooth Optimization: Theory, Practice and Software. Springer International Publishing (2014).Google Scholar
  39. 39.
    Karmitsa, N., Bagirov, A., Mäkelä, M.M.: Comparing different nonsmooth optimization methods and software. Optim. Methods Softw. 27(1), 131–153 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Herskovits, J., Goulart, E.: Sparse quasi-Newton matrices for large scale nonlinear optimization. In: Proceedings of the 6th Word Congress on Structurla and Multidisciplinary Optimization (2005).Google Scholar
  41. 41.
    Clarke, F.H.: Optimization and Nonsmooth Analysis. Wiley-Interscience, New York (1983)zbMATHGoogle Scholar
  42. 42.
    Karmitsa, N., Mäkelä, M.M., Ali, M.M.: Limited memory interior point bundle method for large inequality constrained nonsmooth minimization. Appl. Math. Comput. 198(1), 382–400 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Vlček, J., Lukšan, L.: Globally convergent variable metric method for nonconvex nondifferentiable unconstrained minimization. J. Optim. Theory Appl. 111(2), 407–430 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Byrd, R.H., Nocedal, J., Schnabel, R.B.: Representations of quasi-Newton matrices and their use in limited memory methods. Math. Program. 63, 129–156 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Lemaréchal, C., Strodiot, J.J., Bihain, A.: On a bundle algorithm for nonsmooth optimization. In: Mangasarian, O.L., Mayer, R.R., Robinson, S.M. (eds.) Nonlinear Programming, pp. 245–281. Academic Press, New York (1981)Google Scholar
  46. 46.
    Mifflin, R.: A modification and an extension of Lemaréchal’s algorithm for nonsmooth minimization. Mat. Program. Study 17, 77–90 (1982)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Bihain, A.: Optimization of upper semidifferentiable functions. J. Optim. Theory Appl. 4, 545–568 (1984)MathSciNetCrossRefGoogle Scholar
  48. 48.
    Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, Chichester (1987)zbMATHGoogle Scholar
  49. 49.
    Nocedal, J.: Updating Quasi-Newton matrices with limited storage. Math. Comput. 35(151), 773–782 (1980)MathSciNetCrossRefzbMATHGoogle Scholar
  50. 50.
    Fletcher, R.: An optimal positive definite update for sparse Hessian matrices. SIAM J. Optim. 5(1), 192–218 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  51. 51.
    Fletcher, R., Grothey, A., Leyffer, S.: Computing sparse Hessian and Jacobian approximations with optimal hereditary properties. University of Dundee Numerical Analysis Report NA/164 (1995).Google Scholar
  52. 52.
    Toint, P.L.: On sparse and symmetric matrix updating subject to a linear equation. Math. Comput. 31(140), 954–961 (1977)MathSciNetCrossRefzbMATHGoogle Scholar
  53. 53.
    Mäkelä, M.M.: Multiobjective proximal bundle method for nonconvex nonsmooth optimization: Fortran subroutine MPBNGC 2.0. Reports of the Department of Mathematical Information Technology, Series B. Scientific Computing, B. 13/2003 University of Jyväskylä, Jyväskylä (2003).Google Scholar
  54. 54.
    Bagirov, A.M., Ganjehlou, A.N.: A secant method for nonsmooth optimization. Submitted (2009).Google Scholar
  55. 55.
    Bagirov, A.M., Ganjehlou, A.N.: A quasisecant method for minimizing nonsmooth functions. Optim. Methods Softw. 25(1), 3–18 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  56. 56.
    Lukšan, L.: Dual method for solving a special problem of quadratic programming as a subproblem at linearly constrained nonlinear minimax approximation. Kybernetika 20, 445–457 (1984)MathSciNetzbMATHGoogle Scholar
  57. 57.
    Miettinen, K., Mäkelä, M.M.: Synchronous approach in interactive multiobjective optimization. Eur. J. Oper. Res. 170(3), 909–922 (2006)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of Mathematics and StatisticsUniversity of TurkuTurkuFinland

Personalised recommendations