Advertisement

Subgradient and Bundle Methods for Nonsmooth Optimization

  • Marko M. Mäkelä
  • Napsu Karmitsa
  • Adil Bagirov
Part of the Computational Methods in Applied Sciences book series (COMPUTMETHODS, volume 27)

Abstract

The nonsmooth optimization methods can mainly be divided into two groups: subgradient and bundle methods. Usually, when developing new algorithms and testing them, the comparison is made between similar kinds of methods. The goal of this work is to test and compare different bundle and subgradient methods as well as some hybrids of these two and/or some others. The test set included a large amount of different unconstrained nonsmooth minimization problems, e.g., convex and nonconvex problems, piecewise linear and quadratic problems, and problems with different sizes. Rather than foreground some method over the others, our aim is to get some insight on which method is suitable for certain types of problems.

Keywords

Descent Direction Quadratic Problem Bundle Method Subgradient Method Nonconvex Problem 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgements

We would like to acknowledge professors A. Kuntsevich and F. Kappel for providing Shor’s r-algorithm in their web-page as well as professors L. Lukšan and J. Vlček for providing the bundle-Newton algorithm. The work was financially supported by the University of Turku (Finland) and the University of Ballarat (Australia) and the Australian Research Council.

References

  1. 1.
    Äyrämö S (2006) Knowledge mining using robust clustering. PhD thesis, University of Jyväskylä Google Scholar
  2. 2.
    Bagirov AM, Ganjehlou AN (2010) A quasisecant method for minimizing nonsmooth functions. Optim Methods Softw 25(1):3–18 MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Bagirov AM, Karasözen B, Sezer M (2008) Discrete gradient method: Derivative-free method for nonsmooth optimization. J Optim Theory Appl 137(2):317–334 MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Beck A, Teboulle M (2003) Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper Res Lett 31(3):167–175 MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Ben-Tal A, Nemirovski A (2005) Non-Euclidean restricted memory level method for large-scale convex optimization. Math Program 102(3):407–456 MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Bihain A (1984) Optimization of upper semidifferentiable functions. J Optim Theory Appl 44(4):545–568 MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Bradley PS, Fayyad UM, Mangasarian OL (1999) Mathematical programming for data mining: Formulations and challenges. INFORMS J Comput 11(3):217–238 MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Burke JV, Lewis AS, Overton ML (2005) A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J Optim 15(3):751–779 MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Byrd RH, Nocedal J, Schnabel RB (1994) Representations of quasi-Newton matrices and their use in limited memory methods. Math Program 63(2):129–156 MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Clarke FH (1983) Optimization and nonsmooth analysis. Wiley, New York zbMATHGoogle Scholar
  11. 11.
    Clarke FH, Ledyaev YS, Stern RJ, Wolenski PR (1998) Nonsmooth analysis and control theory. Springer, New York zbMATHGoogle Scholar
  12. 12.
    Demyanov VF, Bagirov AM, Rubinov AM (2002) A method of truncated codifferential with application to some problems of cluster analysis. J Glob Optim 23(1):63–80 MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Dolan ED, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math Program 91(2):201–213 MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Gaudioso M, Monaco MF (1992) Variants to the cutting plane approach for convex nondifferentiable optimization. Optimization 25(1):65–75 MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Haarala M, Miettinen K, Mäkelä MM (2004) New limited memory bundle method for large-scale nonsmooth optimization. Optim Methods Softw 19(6):673–692 MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Haarala N, Miettinen K, Mäkelä MM (2007) Globally convergent limited memory bundle method for large-scale nonsmooth optimization. Math Program 109(1):181–205 MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    Haslinger J, Neittaanmäki P (1996) Finite element approximation for optimal shape, material and topology design, 2nd edn. Wiley, Chichester zbMATHGoogle Scholar
  18. 18.
    Hiriart-Urruty J-B, Lemaréchal C (1993) Convex analysis and minimization algorithms. II. Advanced theory and bundle methods. Springer, Berlin zbMATHGoogle Scholar
  19. 19.
    Kappel F, Kuntsevich AV (2000) An implementation of Shor’s r-algorithm. Comput Optim Appl 15(2):193–205 MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Kärkkäinen T, Heikkola E (2004) Robust formulations for training multilayer perceptrons. Neural Comput 16(4):837–862 zbMATHCrossRefGoogle Scholar
  21. 21.
    Karmitsa N, Bagirov A, Mäkelä MM (2009) Empirical and theoretical comparisons of several nonsmooth minimization methods and software. TUCS Technical Report 959, Turku Centre for Computer Science, Turku. Available online http://tucs.fi/publications/insight.php?id=tKaBaMa09a
  22. 22.
    Karmitsa N, Bagirov A, Mäkelä MM (2012) Comparing different nonsmooth minimization methods and software. Optim Methods Softw 27(1):131–153. doi: 10.1080/10556788.2010.526116 MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Kiwiel KC (1985) Methods of descent for nondifferentiable optimization. Lecture notes in mathematics, vol 1133. Springer, Berlin zbMATHGoogle Scholar
  24. 24.
    Kiwiel KC (1990) Proximity control in bundle methods for convex nondifferentiable minimization. Math Program 46(1):105–122 MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Kuntsevich A, Kappel F (1997) SolvOpt – the solver for local nonlinear optimization problems. Graz. http://www.uni-graz.at/imawww/kuntsevich/solvopt/
  26. 26.
    Lemaréchal C (1989) Nondifferentiable optimization. In: Nemhauser GL, Rinnooy Kan AHG, Todd MJ (eds) Optimization. North-Holland, Amsterdam, pp 529–572 CrossRefGoogle Scholar
  27. 27.
    Lukšan L (1984) Dual method for solving a special problem of quadratic programming as a subproblem at linearly constrained nonlinear minimax approximation. Kybernetika 20(6):445–457 MathSciNetzbMATHGoogle Scholar
  28. 28.
    Lukšan L, Tu̇ma M, Šiška M, Vlček J, Ramešová N (2002) Ufo 2002: Interactive system for universal functional optimization. Technical report V-883, Academy of Sciences of the Czech Republic, Prague Google Scholar
  29. 29.
    Lukšan L, Vlček J (1998) A bundle-Newton method for nonsmooth unconstrained minimization. Math Program 83(3):373–391 zbMATHCrossRefGoogle Scholar
  30. 30.
    Lukšan L, Vlček J (2000) NDA: Algorithms for nondifferentiable optimization. Technical report V-797, Academy of Sciences of the Czech Republic, Prague Google Scholar
  31. 31.
    Lukšan L, Vlček J (2000) Test problems for nonsmooth unconstrained and linearly constrained optimization. Technical report V-798, Academy of Sciences of the Czech Republic, Prague Google Scholar
  32. 32.
    Mäkelä MM (2002) Survey of bundle methods for nonsmooth optimization. Optim Methods Softw 17(1):1–29 MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Mäkelä MM (2003) Multiobjective proximal bundle method for nonconvex nonsmooth optimization: Fortran subroutine MPBNGC 2.0. Reports of the Department of Mathematical Information Technology, Series B, Scientific Computing B13/2003, University of Jyväskylä, Jyväskylä Google Scholar
  34. 34.
    Mäkelä MM, Miettinen M, Lukšan L, Vlček J (1999) Comparing nonsmooth nonconvex bundle methods in solving hemivariational inequalities. J Glob Optim 14(2):117–135 zbMATHCrossRefGoogle Scholar
  35. 35.
    Mäkelä MM, Neittaanmäki P (1992) Nonsmooth optimization: Analysis and algorithms with applications to optimal control. World Scientific, River Edge zbMATHGoogle Scholar
  36. 36.
    Mistakidis ES, Stavroulakis GE (1998) Nonconvex optimization in mechanics. Algorithms, heuristics and engineering applications by the FEM. Kluwer, Dordrecht Google Scholar
  37. 37.
    Moreau JJ, Panagiotopoulos PD, Strang G (eds) (1988) Topics in nonsmooth mechanics. Birkhäuser, Basel zbMATHGoogle Scholar
  38. 38.
    Outrata J, Kočvara M, Zowe J (1998) Nonsmooth approach to optimization problems with equilibrium constraints. Theory, applications and numerical results. Kluwer, Dordrecht zbMATHGoogle Scholar
  39. 39.
    Robinson SM (1999) Linear convergence of epsilon-subgradient descent methods for a class of convex functions. Math Program 86(1):41–50 MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Sagastizábal C, Solodov M (2005) An infeasible bundle method for nonsmooth convex constrained optimization without a penalty function or a filter. SIAM J Optim 16(1):146–169 MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Schramm H, Zowe J (1992) A version of the bundle idea for minimizing a nonsmooth function: Conceptual idea, convergence analysis, numerical results. SIAM J Optim 2(1):121–152 MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    Shor NZ (1985) Minimization methods for non-differentiable functions. Springer, Berlin zbMATHCrossRefGoogle Scholar
  43. 43.
    Uryasev SP (1991) New variable-metric algorithms for nondifferentiable optimization problems. J Optim Theory Appl 71(2):359–388 MathSciNetCrossRefGoogle Scholar
  44. 44.
    Vlček J, Lukšan L (2001) Globally convergent variable metric method for nonconvex nondifferentiable unconstrained minimization. J Optim Theory Appl 111(2):407–430 MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Marko M. Mäkelä
    • 1
  • Napsu Karmitsa
    • 1
  • Adil Bagirov
    • 2
  1. 1.Department of MathematicsUniversity of TurkuTurkuFinland
  2. 2.Centre for Informatics and Applied Optimization, School of Science, Information Technology and EngineeringUniversity of BallaratBallaratAustralia

Personalised recommendations