Skip to main content
SpringerLink
Log in

Nonlinear Optimization: A Brief Overview

  • Chapter
  • First Online:
Numerical Infinities and Infinitesimals in Optimization

Part of the book series: Emergence, Complexity and Computation ((ECC,volume 43))

Abstract

In this chapter some of the most important results for unconstrained and constrained optimization problems are discussed. This chapter does not claim to cover all the aspects in nonlinear optimization that will require more than one complete book. We decided, instead, to concentrate our attention on few fundamental topics that are also at the basis of the new results in nonlinear optimization using grossone introduced in the successive chapters.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Here \(\left\| x \right\| ^2_M :={x}^T{Mx}\).

  2. 2.

    For a matrix \(A \in \mathrm{I}\!\mathrm{R}^{m \times n}\), the Frobenius norm \(\left\| A \right\| _{\text {F}}\) of A is defined as [48]

    $$\begin{aligned} \left\| A \right\| _{\text {F}}:= & {} \sqrt{\sum _{i=1}^m \sum _{j=1}^n A_{ij}^2} = \sqrt{\sum _{i=1}^m \left\| A_{i.} \right\| ^2_2} = \sqrt{\sum _{j=1}^n \left\| A_{.j} \right\| ^2_2} \\= & {} \sqrt{\mathop {\text{ trace }}{\left( A^TA\right) }} = \sqrt{\mathop {\text{ trace }}{\left( AA^T\right) }}. \end{aligned}$$
  3. 3.

    Give a cone K, the dual cone of K is the set

    $$\begin{aligned} K^* :=\left\{ d \in \mathrm{I}\!\mathrm{R}^n: {d}^T{x} \ge 0, \forall \; x \in K\right\} \end{aligned}$$
  4. 4.

    For a set S, cl (S) indicates its closure.

  5. 5.

    A generic function g is quasi–concave if and only if \(-g\) is quasi–convex.

  6. 6.

    One of the most important technique for solving convex quadratic programming problems with equality and inequality constraints is based on Active Set strategy where, at each iteration, some of the inequality constraints, and all the equality constraints, are imposed as equalities (the “Working Set”) and a simpler quadratic problem with only equality constraints is solved. Then the Working Set is update and a new iteration is performed. For further details refer to [25, 10.3] and [49, 16.5].

References

  1. Abadie, J.: On the Kuhn-Tucker Theorem. Operations Research Center University of Calif Berkeley, Technical report (1966)

    Google Scholar 

  2. Al-Baali, M.: Descent property and global convergence of the Fletcher’ Reeves method with inexact line search. IMA J. Numer. Anal. 5(1), 121–124 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  3. Apostol, T.M.: Calculus, 2nd edn. Wiley (1967)

    Google Scholar 

  4. Armijo, L.: Minimization of functions having Lipschitz continuous first partial derivatives. Pac. J. Math. 16(1), 1–3 (1966)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bagirov, A.M., Gaudioso, M., Karmitsa, N., Mäkelä, M.M., Taheri, S.: Numerical Nonsmooth Optimization: State of the Art Algorithms. Springer Nature (2020)

    Google Scholar 

  6. Bazaraa, M.S., Sherali, H.D., Shetty, C.M.: Nonlinear Programming: Theory and Algorithms, 3rd edn. Wiley, Newark, NJ (2006)

    Book  MATH  Google Scholar 

  7. Bertsekas, D.P.: Nonlinear Programming. Athena Scientific (1999)

    Google Scholar 

  8. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, New York, NY, USA (2004)

    Book  MATH  Google Scholar 

  9. Broyden, C.G.: A class of methods for solving nonlinear simultaneous equations. Math. Comput. 19, 577–593 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  10. Broyden, C.G.: Quasi-Newton methods and their application to function minimization. Math. Comput. 21, 368–381 (1967)

    Article  MATH  Google Scholar 

  11. Broyden, C.G.: The convergence of single-rank quasi-Newton methods. Math. Comput. 24, 365–382 (1970)

    Article  MathSciNet  MATH  Google Scholar 

  12. Broyden, C.G., Dennis, J.E., Moré, J.J.: On the local and superlinear convergence of quasi-Newton methods. IMA J. Appl. Math. 12(3), 223–245 (1973)

    Article  MathSciNet  MATH  Google Scholar 

  13. Conn, A.R., Gould, N.I.M., Toint, P.L.: Convergence of Quasi-Newton matrices generated by the symmetric rank one update. Math. Program. 50(2), 177–195 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  14. Davidon, W.C.: Variable metric method for minimization. AEC Research and Development Report ANL-5990, Argonne National Laboratory (1959)

    Google Scholar 

  15. De Leone, R., Gaudioso, M., Grippo, L.: Stopping criteria for linesearch methods without derivatives. Math. Program. 30(3), 285–300 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  16. Deb, K., Deb, K.: Multi-objective optimization. In: Burke, E., Kendall, G. (eds.) Search Methodologies: Introductory Tutorials in Optimization and Decision Support Techniques, pp. 403–449. Springer, US, Boston, MA (2014)

    Chapter  Google Scholar 

  17. Dennis, J.E., Moré, J.J.: Quasi-Newton methods, motivations and theory. SIAM Rev. 19, 46–89 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  18. Dennis, J.E., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice-Hall, Englewook Cliffs, New Jersey (1983)

    MATH  Google Scholar 

  19. Dennis, J.E., Schnabel, R.B.: Chapter I A view of unconstrained optimization. In: Optimization. Handbooks in Operations Research and Management Science, vol. 1, pp. 1–72. Elsevier (1989)

    Google Scholar 

  20. Di Pillo, G., Grippo, L.: Exact penalty functions in constrained optimization. SIAM J. Control Optim. 27(6), 1333–1360 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  21. Di Pillo, G., Liuzzi, G., Lucidi, S., Palagi, L.: An exact augmented Lagrangian function for nonlinear programming with two-sided constraints. Comput. Optim. Appl. 25(1), 57–83 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  22. Dixon, L.C.W.: Variable metric algorithms: necessary and sufficient conditions for identical behavior of nonquadratic functions. J. Optim. Theory Appl. 10, 34–40 (1972)

    Article  MathSciNet  MATH  Google Scholar 

  23. Ehrgott, M.: Multicriteria Optimization, vol. 491. Springer Science & Business Media (2005)

    Google Scholar 

  24. Fletcher, R.: A new approach to variable metric algorithms. Comput. J. 13, 317–322 (1970)

    Article  MATH  Google Scholar 

  25. Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley (1990)

    Google Scholar 

  26. Fletcher, R., Powell, M.J.D.: A rapidly convergent descent method for minimization. Comput. J. 6(2), 163–168 (1963)

    Article  MathSciNet  MATH  Google Scholar 

  27. Fletcher, R., Reeves, C.M.: Function minimization by conjugate gradients. Comput. J. 7(2), 149–154 (1964)

    Article  MathSciNet  MATH  Google Scholar 

  28. Gaudioso, M., Giallombardo, G., Miglionico, G.: Essentials of numerical nonsmooth optimization. 4OR 18(1), 1–47 (2020)

    Google Scholar 

  29. Gilbert, J.C., Nocedal, J.: Global convergence properties of conjugate gradient methods for optimization. SIAM J. Optim. 2(1), 21–42 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  30. Goldfarb, D.: A family of variable metric updates derived by variational means. Math. Comput. 24, 23–26 (1970)

    Article  MATH  Google Scholar 

  31. Goldstein, A.A.: Constructive Real Analysis. Harper and Row, London (1967)

    MATH  Google Scholar 

  32. Golub, G.H., Van Loan, C.F.: Matrix Computations. Johns Hopkins University Press, Baltimore, Maryland (1983)

    MATH  Google Scholar 

  33. Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer. Anal. 23(4), 707–716 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  34. Grippo, L., Sciandrone, M.: Nonmonotone globalization techniques for the Barzilai-Borwein gradient method. Comput. Optim. Appl. 23(2), 143–169 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  35. Grippo, L., Sciandrone, M.: Methods of unconstrained optimization. (Metodi di ottimizzazione non vincolata.). Unitext 53. La Matematica per il 3+2. Springer, Italia (2011)

    Google Scholar 

  36. Guignard, M.: Generalized Kuhn-Tucker conditions for mathematical programming problems in a Banach space. SIAM J. Control 7(2), 232–241 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  37. Hager, W.W.: Updating the inverse of a matrix. SIAM Rev. 31(2), 221–239 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  38. Hager, W.W., Zhang, H.C.: A survey of nonlinear conjugate gradient methods. Pac. J. Optim. 2(1), 35–58 (2006)

    MathSciNet  MATH  Google Scholar 

  39. Hestenes, M.R., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bur. Stand. 49, 409–436 (1952)

    Article  MathSciNet  MATH  Google Scholar 

  40. Horst, R., Pardalos, P.M.: Handbook of Global Optimization, vol. 2. Springer Science & Business Media (2013)

    Google Scholar 

  41. Konak, A., Coit, D.W., Smith, A.E.: Multi-objective optimization using genetic algorithms: a tutorial. Reliab. Eng. Syst. Saf. 91(9), 992–1007 (2006)

    Article  Google Scholar 

  42. Kuhn, H.W., Tucker, A.W.: Nonlinear Programming, pp. 481–492. University of California Press, Berkeley, California (1951)

    Google Scholar 

  43. Lemarechal, C., Mifflin, R.: Nonsmooth optimization: proceedings of a IIASA workshop, March 28–April 8, 1977. Elsevier (2014)

    Google Scholar 

  44. Li, D.H., Fukushima, M.: A derivative-free line search and global convergence of Broyden-like method for nonlinear equations. Optim. Methods Softw. 13(3), 181–201 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  45. Luenberger, D.G.: Linear and Nonlinear Programming, 2nd edn. Addison–Wesley (1984)

    Google Scholar 

  46. Mangasarian, O.L.: Nonlinear Programming. McGraw-Hill, New York (1969)

    MATH  Google Scholar 

  47. Mangasarian, O.L., Fromovitz, S.: The Fritz John necessary optimality conditions in the presence of equality and inequality constraints. J. Math. Anal. Appl. 17, 37–47 (1967)

    Article  MathSciNet  MATH  Google Scholar 

  48. Meyer, C.D.: Matrix Analysis and Applied Linear Algebra. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA (2000)

    Book  Google Scholar 

  49. Nocedal, J., Wright, S.: Numerical Optimization. Springer Science & Business Media (2006)

    Google Scholar 

  50. Ortega, J.M., Rheinboldt, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press (1970)

    Google Scholar 

  51. Pinter, J.: Continuous global optimization: applications. In: C. Floudas, E.P.M. Pardalos (eds.) Encyclopedia of Optimization, pp. 482–486. Springer (2008)

    Google Scholar 

  52. Polak, E., Ribière, G.: Note sur la convergence de méthodes de directions conjuguées. ESAIM: Math. Model. Numer. Anal.-Modélisation Mathématique et Anal. Numérique 3(R1), 35–43 (1969)

    Google Scholar 

  53. Polyak, B.T.: The conjugate gradient method in extremal problems. USSR Comput. Math. Math. Phys. 9(4), 94–112 (1969)

    Article  MATH  Google Scholar 

  54. Pytlak, R.: Conjugate Gradient Algorithms in Nonconvex Optimization, vol. 89. Springer Science & Business Media (2008)

    Google Scholar 

  55. Robinson, S.M.: Perturbed Kuhn-Tucker points and rates of convergence for a class of nonlinear-programming algorithms. Math. Program. 7(1), 1–16 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  56. Rockafellar, R.T.: Convex Analysis. Princeton Mathematical Series, Princeton University Press, Princeton, N. J. (1970)

    Book  MATH  Google Scholar 

  57. Shanno, D.F.: Conditioning of Quasi-Newton methods for function minimization. Math. Comput. 24, 647–656 (1970)

    Article  MathSciNet  MATH  Google Scholar 

  58. Slater, M.: Lagrange multipliers revisited. Technical report, Cowles Foundation Discussion Paper No. 80, Cowles Foundation for Research in Economics, Yale University (1950)

    Google Scholar 

  59. Sun, W., Yuan, Y.X.: Optimization Theory and Methods: Nonlinear Programming, vol. 1. Springer Science & Business Media (2006)

    Google Scholar 

  60. Wang, Z., Fang, S.C., Xing, W.: On constraint qualifications: motivation, design and inter-relations. J. Ind. Manag. Optim. 9, 983–1001 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  61. Wolfe, P.: A duality theorem for non-linear programming. Q. Appl. Math. 19, 239–244 (1961)

    Article  MathSciNet  MATH  Google Scholar 

  62. Wolfe, P.: Convergence conditions for ascent methods. SIAM Rev. 11(2), 226–235 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  63. Zhang, H.C., Hager, W.W.: A nonmonotone line search technique and its application to unconstrained optimization. SIAM J. Optim. 14(4), 1043–1056 (2004)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The author wants to express his gratitude to prof. Nadaniela Egidi for reading a first version of the manuscript and providing many useful suggestions.

Author information

Authors and Affiliations

  1. School of Science and Technology, University of Camerino, Camerino (MC), Italy

    Renato De Leone

Authors
  1. Renato De Leone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Renato De Leone .

Editor information

Editors and Affiliations

  1. Dipartimento di Ingegneria Informatica, Modellistica, Elettronica e Sistemistica, University of Calabria, Rende, Italy

    Yaroslav D. Sergeyev

  2. Scuola di Scienze e Tecnologie, Università degli Studi di Camerino, Camerino, Italy

    Renato De Leone

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

De Leone, R. (2022). Nonlinear Optimization: A Brief Overview. In: Sergeyev, Y.D., De Leone, R. (eds) Numerical Infinities and Infinitesimals in Optimization. Emergence, Complexity and Computation, vol 43. Springer, Cham. https://doi.org/10.1007/978-3-030-93642-6_2

Download citation

Publish with us

Policies and ethics

Search

Navigation