Skip to main content

Part of the book series: Texts in Applied Mathematics ((TAM,volume 63))

  • 8120 Accesses


This chapter reviews the basic elements of optimization theory and practice, without going into the fine details of numerical implementation. Many UQ problems involve a notion of ‘best fit’, in the sense of minimizing some error function, and so it is helpful to establish some terminology for optimization problems. In particular, many of the optimization problems in this book will fall into the simple settings of linear programming and least squares (quadratic programming), with and without constraints.

We demand rigidly defined areas of doubt and uncertainty!

The Hitchhiker’s Guide to the Galaxy

Douglas Adams

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 99.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. 1.

    Or, more generally, Hausdorff, locally convex, topological vector spaces.

  2. 2.

    If Q is not positive definite, but merely positive semi-definite and self-adjoint, then existence of solutions to the associated least squares problems still holds, but uniqueness can fail.


  • B. M. Adams, L. E. Bauman, W. J. Bohnhoff, K. R. Dalbey, M. S. Ebeida, J. P. Eddy, M. S. Eldred, P. D. Hough, K. T. Hu, J. D. Jakeman, L. P. Swiler, J. A. Stephens, D. M. Vigil, and T. M. Wildey. DAKOTA, A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 6.1 User’s Manual. Technical Report SAND2014-4633, Sandia National Laboratories, November 2014.

  • E. Bishop and K. de Leeuw. The representations of linear functionals by measures on sets of extreme points. Ann. Inst. Fourier. Grenoble, 9: 305–331, 1959.

    Article  MathSciNet  MATH  Google Scholar 

  • S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, Cambridge, 2004.

    Book  MATH  Google Scholar 

  • F. J. Gould and J. W. Tolle. Optimality conditions and constraint qualifications in Banach space. J. Optimization Theory Appl., 15:667–684, 1975.

    Article  MathSciNet  MATH  Google Scholar 

  • W. Karush. Minima of Functions of Several Variables with Inequalities as Side Constraints. Master’s thesis, Univ. of Chicago, Chicago, 1939.

    Google Scholar 

  • S. Kirkpatrick, C. D. Gelatt, Jr., and M. P. Vecchi. Optimization by simulated annealing. Science, 220(4598):671–680, 1983. doi: 10.1126/ science.220.4598.671.

    Article  MathSciNet  MATH  Google Scholar 

  • S. G. Krantz. Convex Analysis. Textbooks in Mathematics. CRC Press, Boca Raton, FL, 2015.

    MATH  Google Scholar 

  • M. Krein and D. Milman. On extreme points of regular convex sets. Studia Math., 9:133–138, 1940.

    MathSciNet  Google Scholar 

  • H. W. Kuhn and A. W. Tucker. Nonlinear programming. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, 1950, pages 481–492. University of California Press, Berkeley and Los Angeles, 1951.

    Google Scholar 

  • M. M. McKerns, P. Hung, and M. A. G. Aivazis. Mystic: A Simple Model-Independent Inversion Framework, 2009.

  • M. M. McKerns, L. Strand, T. J. Sullivan, P. Hung, and M. A. G. Aivazis. Building a Framework for Predictive Science. In S. van der Walt and J. Millman, editors, Proceedings of the 10th Python in Science Conference (SciPy 2011), June 2011, pages 67–78, 2011.

    Google Scholar 

  • J. Nocedal and S. J. Wright. Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer, New York, second edition, 2006.

    MATH  Google Scholar 

  • R. R. Phelps. Lectures on Choquet’s Theorem, volume 1757 of Lecture Notes in Mathematics. Springer-Verlag, Berlin, second edition, 2001. doi: 10.1007/b76887.

    Google Scholar 

  • K. V. Price, R. M. Storn, and J. A. Lampinen. Differential Evolution: A Practical Approach to Global Optimization. Natural Computing Series. Springer-Verlag, Berlin, 2005.

    Google Scholar 

  • R. T. Rockafellar. Convex Analysis. Princeton Landmarks in Mathematics. Princeton University Press, Princeton, NJ, 1997. Reprint of the 1970 original.

    Google Scholar 

  • R. Storn and K. Price. Differential evolution — a simple and efficient heuristic for global optimization over continuous spaces. J. Global Optim., 11(4):341–359, 1997. doi: 10.1023/A:1008202821328.

    Article  MathSciNet  MATH  Google Scholar 

  • E. Zeidler. Applied Functional Analysis: Main Principles and Their Applications, volume 109 of Applied Mathematical Sciences. Springer-Verlag, New York, 1995.

    Google Scholar 

Download references

Author information

Authors and Affiliations


Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Sullivan, T.J. (2015). Optimization Theory. In: Introduction to Uncertainty Quantification. Texts in Applied Mathematics, vol 63. Springer, Cham.

Download citation

Publish with us

Policies and ethics