Subgradient and Bundle Methods for Nonsmooth Optimization

Part of the Computational Methods in Applied Sciences book series (COMPUTMETHODS, volume 27)


The nonsmooth optimization methods can mainly be divided into two groups: subgradient and bundle methods. Usually, when developing new algorithms and testing them, the comparison is made between similar kinds of methods. The goal of this work is to test and compare different bundle and subgradient methods as well as some hybrids of these two and/or some others. The test set included a large amount of different unconstrained nonsmooth minimization problems, e.g., convex and nonconvex problems, piecewise linear and quadratic problems, and problems with different sizes. Rather than foreground some method over the others, our aim is to get some insight on which method is suitable for certain types of problems.


Descent Direction Quadratic Problem Bundle Method Subgradient Method Nonconvex Problem 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We would like to acknowledge professors A. Kuntsevich and F. Kappel for providing Shor’s r-algorithm in their web-page as well as professors L. Lukšan and J. Vlček for providing the bundle-Newton algorithm. The work was financially supported by the University of Turku (Finland) and the University of Ballarat (Australia) and the Australian Research Council.


  1. 1.
    Äyrämö S (2006) Knowledge mining using robust clustering. PhD thesis, University of Jyväskylä Google Scholar
  2. 2.
    Bagirov AM, Ganjehlou AN (2010) A quasisecant method for minimizing nonsmooth functions. Optim Methods Softw 25(1):3–18 MathSciNetMATHCrossRefGoogle Scholar
  3. 3.
    Bagirov AM, Karasözen B, Sezer M (2008) Discrete gradient method: Derivative-free method for nonsmooth optimization. J Optim Theory Appl 137(2):317–334 MathSciNetMATHCrossRefGoogle Scholar
  4. 4.
    Beck A, Teboulle M (2003) Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper Res Lett 31(3):167–175 MathSciNetMATHCrossRefGoogle Scholar
  5. 5.
    Ben-Tal A, Nemirovski A (2005) Non-Euclidean restricted memory level method for large-scale convex optimization. Math Program 102(3):407–456 MathSciNetMATHCrossRefGoogle Scholar
  6. 6.
    Bihain A (1984) Optimization of upper semidifferentiable functions. J Optim Theory Appl 44(4):545–568 MathSciNetMATHCrossRefGoogle Scholar
  7. 7.
    Bradley PS, Fayyad UM, Mangasarian OL (1999) Mathematical programming for data mining: Formulations and challenges. INFORMS J Comput 11(3):217–238 MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    Burke JV, Lewis AS, Overton ML (2005) A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J Optim 15(3):751–779 MathSciNetMATHCrossRefGoogle Scholar
  9. 9.
    Byrd RH, Nocedal J, Schnabel RB (1994) Representations of quasi-Newton matrices and their use in limited memory methods. Math Program 63(2):129–156 MathSciNetMATHCrossRefGoogle Scholar
  10. 10.
    Clarke FH (1983) Optimization and nonsmooth analysis. Wiley, New York MATHGoogle Scholar
  11. 11.
    Clarke FH, Ledyaev YS, Stern RJ, Wolenski PR (1998) Nonsmooth analysis and control theory. Springer, New York MATHGoogle Scholar
  12. 12.
    Demyanov VF, Bagirov AM, Rubinov AM (2002) A method of truncated codifferential with application to some problems of cluster analysis. J Glob Optim 23(1):63–80 MathSciNetMATHCrossRefGoogle Scholar
  13. 13.
    Dolan ED, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math Program 91(2):201–213 MathSciNetMATHCrossRefGoogle Scholar
  14. 14.
    Gaudioso M, Monaco MF (1992) Variants to the cutting plane approach for convex nondifferentiable optimization. Optimization 25(1):65–75 MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Haarala M, Miettinen K, Mäkelä MM (2004) New limited memory bundle method for large-scale nonsmooth optimization. Optim Methods Softw 19(6):673–692 MathSciNetMATHCrossRefGoogle Scholar
  16. 16.
    Haarala N, Miettinen K, Mäkelä MM (2007) Globally convergent limited memory bundle method for large-scale nonsmooth optimization. Math Program 109(1):181–205 MathSciNetMATHCrossRefGoogle Scholar
  17. 17.
    Haslinger J, Neittaanmäki P (1996) Finite element approximation for optimal shape, material and topology design, 2nd edn. Wiley, Chichester MATHGoogle Scholar
  18. 18.
    Hiriart-Urruty J-B, Lemaréchal C (1993) Convex analysis and minimization algorithms. II. Advanced theory and bundle methods. Springer, Berlin MATHGoogle Scholar
  19. 19.
    Kappel F, Kuntsevich AV (2000) An implementation of Shor’s r-algorithm. Comput Optim Appl 15(2):193–205 MathSciNetMATHCrossRefGoogle Scholar
  20. 20.
    Kärkkäinen T, Heikkola E (2004) Robust formulations for training multilayer perceptrons. Neural Comput 16(4):837–862 MATHCrossRefGoogle Scholar
  21. 21.
    Karmitsa N, Bagirov A, Mäkelä MM (2009) Empirical and theoretical comparisons of several nonsmooth minimization methods and software. TUCS Technical Report 959, Turku Centre for Computer Science, Turku. Available online
  22. 22.
    Karmitsa N, Bagirov A, Mäkelä MM (2012) Comparing different nonsmooth minimization methods and software. Optim Methods Softw 27(1):131–153. doi: 10.1080/10556788.2010.526116 MathSciNetMATHCrossRefGoogle Scholar
  23. 23.
    Kiwiel KC (1985) Methods of descent for nondifferentiable optimization. Lecture notes in mathematics, vol 1133. Springer, Berlin MATHGoogle Scholar
  24. 24.
    Kiwiel KC (1990) Proximity control in bundle methods for convex nondifferentiable minimization. Math Program 46(1):105–122 MathSciNetMATHCrossRefGoogle Scholar
  25. 25.
    Kuntsevich A, Kappel F (1997) SolvOpt – the solver for local nonlinear optimization problems. Graz.
  26. 26.
    Lemaréchal C (1989) Nondifferentiable optimization. In: Nemhauser GL, Rinnooy Kan AHG, Todd MJ (eds) Optimization. North-Holland, Amsterdam, pp 529–572 CrossRefGoogle Scholar
  27. 27.
    Lukšan L (1984) Dual method for solving a special problem of quadratic programming as a subproblem at linearly constrained nonlinear minimax approximation. Kybernetika 20(6):445–457 MathSciNetMATHGoogle Scholar
  28. 28.
    Lukšan L, Tu̇ma M, Šiška M, Vlček J, Ramešová N (2002) Ufo 2002: Interactive system for universal functional optimization. Technical report V-883, Academy of Sciences of the Czech Republic, Prague Google Scholar
  29. 29.
    Lukšan L, Vlček J (1998) A bundle-Newton method for nonsmooth unconstrained minimization. Math Program 83(3):373–391 MATHCrossRefGoogle Scholar
  30. 30.
    Lukšan L, Vlček J (2000) NDA: Algorithms for nondifferentiable optimization. Technical report V-797, Academy of Sciences of the Czech Republic, Prague Google Scholar
  31. 31.
    Lukšan L, Vlček J (2000) Test problems for nonsmooth unconstrained and linearly constrained optimization. Technical report V-798, Academy of Sciences of the Czech Republic, Prague Google Scholar
  32. 32.
    Mäkelä MM (2002) Survey of bundle methods for nonsmooth optimization. Optim Methods Softw 17(1):1–29 MathSciNetMATHCrossRefGoogle Scholar
  33. 33.
    Mäkelä MM (2003) Multiobjective proximal bundle method for nonconvex nonsmooth optimization: Fortran subroutine MPBNGC 2.0. Reports of the Department of Mathematical Information Technology, Series B, Scientific Computing B13/2003, University of Jyväskylä, Jyväskylä Google Scholar
  34. 34.
    Mäkelä MM, Miettinen M, Lukšan L, Vlček J (1999) Comparing nonsmooth nonconvex bundle methods in solving hemivariational inequalities. J Glob Optim 14(2):117–135 MATHCrossRefGoogle Scholar
  35. 35.
    Mäkelä MM, Neittaanmäki P (1992) Nonsmooth optimization: Analysis and algorithms with applications to optimal control. World Scientific, River Edge MATHGoogle Scholar
  36. 36.
    Mistakidis ES, Stavroulakis GE (1998) Nonconvex optimization in mechanics. Algorithms, heuristics and engineering applications by the FEM. Kluwer, Dordrecht Google Scholar
  37. 37.
    Moreau JJ, Panagiotopoulos PD, Strang G (eds) (1988) Topics in nonsmooth mechanics. Birkhäuser, Basel MATHGoogle Scholar
  38. 38.
    Outrata J, Kočvara M, Zowe J (1998) Nonsmooth approach to optimization problems with equilibrium constraints. Theory, applications and numerical results. Kluwer, Dordrecht MATHGoogle Scholar
  39. 39.
    Robinson SM (1999) Linear convergence of epsilon-subgradient descent methods for a class of convex functions. Math Program 86(1):41–50 MathSciNetMATHCrossRefGoogle Scholar
  40. 40.
    Sagastizábal C, Solodov M (2005) An infeasible bundle method for nonsmooth convex constrained optimization without a penalty function or a filter. SIAM J Optim 16(1):146–169 MathSciNetMATHCrossRefGoogle Scholar
  41. 41.
    Schramm H, Zowe J (1992) A version of the bundle idea for minimizing a nonsmooth function: Conceptual idea, convergence analysis, numerical results. SIAM J Optim 2(1):121–152 MathSciNetMATHCrossRefGoogle Scholar
  42. 42.
    Shor NZ (1985) Minimization methods for non-differentiable functions. Springer, Berlin MATHCrossRefGoogle Scholar
  43. 43.
    Uryasev SP (1991) New variable-metric algorithms for nondifferentiable optimization problems. J Optim Theory Appl 71(2):359–388 MathSciNetCrossRefGoogle Scholar
  44. 44.
    Vlček J, Lukšan L (2001) Globally convergent variable metric method for nonconvex nondifferentiable unconstrained minimization. J Optim Theory Appl 111(2):407–430 MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Marko M. Mäkelä
    • 1
  • Napsu Karmitsa
    • 1
  • Adil Bagirov
    • 2
  1. 1.Department of MathematicsUniversity of TurkuTurkuFinland
  2. 2.Centre for Informatics and Applied Optimization, School of Science, Information Technology and EngineeringUniversity of BallaratBallaratAustralia

Personalised recommendations