Journal of Optimization Theory and Applications

, Volume 166, Issue 3, pp 889–905 | Cite as

Diagonal Bundle Method for Nonsmooth Sparse Optimization

Article
First Online:
Received:
Accepted:

Abstract

We propose an efficient diagonal bundle method for sparse nonsmooth, possibly nonconvex optimization. The convergence of the proposed method is proved for locally Lipschitz continuous functions, which are not necessary differentiable or convex. The numerical experiments have been made using problems with up to million variables. The results to be presented confirm the usability of the diagonal bundle method especially for extremely large-scale problems.

Keywords

Nondifferentiable optimization Sparse problems Bundle methods Diagonal variable metric methods 

Mathematics Subject Classification

65K05 90C25 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

The work was financially supported by the University of Turku (Finland) and Magnus Ehrnrooth foundation.

References

  1. 1.
    Outrata, J., Kočvara, M., Zowe, J.: Nonsmooth Approach to Optimization Problems with Equilibrium Constraints. Theory, Applications and Numerical Results. Kluwert Academic Publisher, Dordrecht (1998)CrossRefMATHGoogle Scholar
  2. 2.
    Moreau, J.J., Panagiotopoulos, P.D., Strang, G. (eds.): Topics in Nonsmooth Mechanics. Birkhäuser Verlag, Basel (1988)MATHGoogle Scholar
  3. 3.
    Mistakidis, E.S., Stavroulakis, G.E.: Nonconvex Optimization in Mechanics. Smooth and Nonsmooth Algorithms, Heuristics and Engineering Applications by the F.E.M. Kluwert Academic Publishers, Dordrecht (1998)Google Scholar
  4. 4.
    Clarke, F.H., Ledyaev, Y.S., Stern, R.J., Wolenski, P.R.: Nonsmooth Analysis and Control Theory. Springer, New York (1998)MATHGoogle Scholar
  5. 5.
    Haslinger, J., Neittaanmäki, P.: Finite Element Approximation for Optimal Shape, Material and Topology Design, 2nd edn. John Wiley & Sons, Chichester (1996)MATHGoogle Scholar
  6. 6.
    Kärkkäinen, T., Heikkola, E.: Robust formulations for training multilayer perceptrons. Neural. Comput. 16, 837–862 (2004)CrossRefMATHGoogle Scholar
  7. 7.
    Äyrämö, S.: Knowledge mining using robust clustering. Ph.D. thesis, University of Jyväskylä, Department of Mathematical Information Technology (2006).Google Scholar
  8. 8.
    Bradley, P.S., Fayyad, U.M., Mangasarian, O.L.: Mathematical programming for data mining: formulations and challenges. INFORMS J. Comput. 11, 217–238 (1999)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Demyanov, V.F., Bagirov, A.M., Rubinov, A.M.: A method of truncated codifferential with application to some problems of cluster analysis. J. Glob. Optim. 23(1), 63–80 (2002)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Astorino, A., Fuduli, A.: Nonsmooth optimization techniques for semi-supervised classification. IEEE Trans. Pattern. Anal. Mach. Intell. 29(12), 2135–2142 (2007)CrossRefGoogle Scholar
  11. 11.
    Astorino, A., Fuduli, A., Gorgone, E.: Nonsmoothness in classification problems. Optim. Methods Softw. 23(5), 675–688 (2008)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Bergeron, C., Moore, G., Zaretzki, J., Breneman, C., Bennett, K.: Fast bundle algorithm for multiple instance learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(6), 1068–1079 (2012).Google Scholar
  13. 13.
    Carrizosa, E., Morales, D.R.: Supervised classification and mathematical optimization. Comput. Oper. Res. 40(1), 150–165 (2013)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Lemaréchal, C.: Nondifferentiable optimization. In: G.L. Nemhauser, A.H.G. Rinnooy Kan, M.J. Todd (eds.) Optimization, pp. 529–572. Elsevier North-Holland Inc, New York (1989).Google Scholar
  15. 15.
    Beck, A., Teboulle, M.: Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. 31(3), 167–175 (2003)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Ben-Tal, A., Nemirovski, A.: Non-Euclidean restricted memory level method for large-scale convex optimization. Math. Program. 102(3), 407–456 (2005)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Shor, N.Z.: Minimization Methods for Non-Differentiable Functions. Springer-Verlag, Berlin (1985)CrossRefMATHGoogle Scholar
  18. 18.
    Uryasev, S.P.: Algorithms for nondifferentiable optimization problems. J. Optim. Theory Appl. 71, 359–388 (1991)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Gaudioso, M., Monaco, M.F.: Variants to the cutting plane approach for convex nondifferentiable optimization. Optimization 25, 65–75 (1992)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms II. Springer-Verlag, Berlin (1993)MATHGoogle Scholar
  21. 21.
    Kiwiel, K.C.: Methods of Descent for Nondifferentiable Optimization. Lecture Notes in Mathematics 1133. Springer-Verlag, Berlin (1985).Google Scholar
  22. 22.
    Lukšan, L., Vlček, J.: Globally convergent variable metric method for convex nonsmooth unconstrained minimization. J. Optim. Theory Appl. 102(3), 593–613 (1999)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Mäkelä, M.M.: Survey of bundle methods for nonsmooth optimization. Optim. Methods Softw. 17(1), 1–29 (2002)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Mäkelä, M.M., Neittaanmäki, P.: Nonsmooth Optimization: Analysis and Algorithms with Applications to Optimal Control. World Scientific Publishing Co., Singapore (1992)CrossRefMATHGoogle Scholar
  25. 25.
    Sagastizábal, C., Solodov, M.: An infeasible bundle method for nonsmooth convex constrained optimization without a penalty function or a filter. SIAM J. Optim. 16(1), 146–169 (2005)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Schramm, H., Zowe, J.: A version of the bundle idea for minimizing a nonsmooth function: conceptual idea, convergence analysis, numerical results. SIAM J. Optim. 2(1), 121–152 (1992)MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Polak, E., Royset, J.O.: Algorithms for finite and semi-finite min-max-min problems using adaptive smoothing techniques. J. Optim. Theory Appl. 119, 421–457 (2003)MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Burke, J.V., Lewis, A.S., Overton, M.L.: A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. 15, 751–779 (2005)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Apkarian, P., Noll, D., Prot, O.: A trust region spectral bundle method for non-convex eigenvalue optimization. SIAM J. Optim. 19(1), 281–306 (2008)MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Fuduli, A., Gaudioso, M., Nurminski, E.A.: A splitting bundle approach for non-smooth non-convex minimization. Optimization, (2013, in press ), doi: 10.1080/02331934.2013.840625.
  31. 31.
    Fuduli, A., Gaudioso, M., Giallombardo, G.: Minimizing nonconvex nonsmooth functions via cutting planes and proximity control. SIAM J. Optim. 14(3), 743–756 (2004)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Hare, W., Sagastizábal, C.: A redistributed proximal bundle method for nonconvex optimization. SIAM J. Optim. 20(5), 2442–2473 (2010)MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Karmitsa, N., Tanaka Filho, M., Herskovits, J.: Globally convergent cutting plane method for nonconvex nonsmooth minimization. J. Optim. Theory Appl. 148(3), 528–549 (2011)Google Scholar
  34. 34.
    Noll, D., Prot, O., Rondepierre, A.: A proximity control algorithm to minimize nonsmooth and nonconvex functions. Pac. J. Optim. 4(3), 571–604 (2008)MathSciNetMATHGoogle Scholar
  35. 35.
    Haarala, M.: Large-scale nonsmooth optimization: Variable metric bundle method with limited memory. Ph.D. thesis, University of Jyväskylä, Department of Mathematical Information Technology (2004).Google Scholar
  36. 36.
    Haarala, M., Miettinen, K., Mäkelä, M.M.: New limited memory bundle method for large-scale nonsmooth optimization. Optim. Methods Softw. 19(6), 673–692 (2004)MathSciNetCrossRefMATHGoogle Scholar
  37. 37.
    Haarala, N., Miettinen, K., Mäkelä, M.M.: Globally convergent limited memory bundle method for large-scale nonsmooth optimization. Math. Program. 109(1), 181–205 (2007)MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Bagirov, A.M., Karmitsa, N., Mäkelä, M.M.: Introduction to Nonsmooth Optimization: Theory, Practice and Software. Springer International Publishing (2014).Google Scholar
  39. 39.
    Karmitsa, N., Bagirov, A., Mäkelä, M.M.: Comparing different nonsmooth optimization methods and software. Optim. Methods Softw. 27(1), 131–153 (2012)MathSciNetCrossRefMATHGoogle Scholar
  40. 40.
    Herskovits, J., Goulart, E.: Sparse quasi-Newton matrices for large scale nonlinear optimization. In: Proceedings of the 6th Word Congress on Structurla and Multidisciplinary Optimization (2005).Google Scholar
  41. 41.
    Clarke, F.H.: Optimization and Nonsmooth Analysis. Wiley-Interscience, New York (1983)MATHGoogle Scholar
  42. 42.
    Karmitsa, N., Mäkelä, M.M., Ali, M.M.: Limited memory interior point bundle method for large inequality constrained nonsmooth minimization. Appl. Math. Comput. 198(1), 382–400 (2008)MathSciNetCrossRefMATHGoogle Scholar
  43. 43.
    Vlček, J., Lukšan, L.: Globally convergent variable metric method for nonconvex nondifferentiable unconstrained minimization. J. Optim. Theory Appl. 111(2), 407–430 (2001)MathSciNetCrossRefMATHGoogle Scholar
  44. 44.
    Byrd, R.H., Nocedal, J., Schnabel, R.B.: Representations of quasi-Newton matrices and their use in limited memory methods. Math. Program. 63, 129–156 (1994)MathSciNetCrossRefMATHGoogle Scholar
  45. 45.
    Lemaréchal, C., Strodiot, J.J., Bihain, A.: On a bundle algorithm for nonsmooth optimization. In: Mangasarian, O.L., Mayer, R.R., Robinson, S.M. (eds.) Nonlinear Programming, pp. 245–281. Academic Press, New York (1981)Google Scholar
  46. 46.
    Mifflin, R.: A modification and an extension of Lemaréchal’s algorithm for nonsmooth minimization. Mat. Program. Study 17, 77–90 (1982)MathSciNetCrossRefMATHGoogle Scholar
  47. 47.
    Bihain, A.: Optimization of upper semidifferentiable functions. J. Optim. Theory Appl. 4, 545–568 (1984)MathSciNetCrossRefGoogle Scholar
  48. 48.
    Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, Chichester (1987)MATHGoogle Scholar
  49. 49.
    Nocedal, J.: Updating Quasi-Newton matrices with limited storage. Math. Comput. 35(151), 773–782 (1980)MathSciNetCrossRefMATHGoogle Scholar
  50. 50.
    Fletcher, R.: An optimal positive definite update for sparse Hessian matrices. SIAM J. Optim. 5(1), 192–218 (1995)MathSciNetCrossRefMATHGoogle Scholar
  51. 51.
    Fletcher, R., Grothey, A., Leyffer, S.: Computing sparse Hessian and Jacobian approximations with optimal hereditary properties. University of Dundee Numerical Analysis Report NA/164 (1995).Google Scholar
  52. 52.
    Toint, P.L.: On sparse and symmetric matrix updating subject to a linear equation. Math. Comput. 31(140), 954–961 (1977)MathSciNetCrossRefMATHGoogle Scholar
  53. 53.
    Mäkelä, M.M.: Multiobjective proximal bundle method for nonconvex nonsmooth optimization: Fortran subroutine MPBNGC 2.0. Reports of the Department of Mathematical Information Technology, Series B. Scientific Computing, B. 13/2003 University of Jyväskylä, Jyväskylä (2003).Google Scholar
  54. 54.
    Bagirov, A.M., Ganjehlou, A.N.: A secant method for nonsmooth optimization. Submitted (2009).Google Scholar
  55. 55.
    Bagirov, A.M., Ganjehlou, A.N.: A quasisecant method for minimizing nonsmooth functions. Optim. Methods Softw. 25(1), 3–18 (2010)MathSciNetCrossRefMATHGoogle Scholar
  56. 56.
    Lukšan, L.: Dual method for solving a special problem of quadratic programming as a subproblem at linearly constrained nonlinear minimax approximation. Kybernetika 20, 445–457 (1984)MathSciNetMATHGoogle Scholar
  57. 57.
    Miettinen, K., Mäkelä, M.M.: Synchronous approach in interactive multiobjective optimization. Eur. J. Oper. Res. 170(3), 909–922 (2006)CrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of Mathematics and StatisticsUniversity of TurkuTurkuFinland

Personalised recommendations