Numerical Methods for Unconstrained Optimization

Abstract

Unconstrained optimization is the search for the maximum or minimum of a function with no restriction on the values of the variables. At the same time, it forms the basis for methods of constrained optimization in the next chapter. Zero-order methods use only function values, progress made in the previous step pointing the way to the next step. The Hooke and Jeeves method is one such method, suitable for small problems with little programming effort. First-order methods employ the gradient of the function, usually obtained by finite difference, to derive a search direction. This is followed by a line search along this direction for the current maximum or minimum, performed either by progressive reduction of the region in which the maximum or minimum is to be found or by polynomial interpolation. In its simplest form, this is the steepest descent method. However, by the use of gradient data from the previous iteration, an improved search direction can be found, with faster convergence. This is the Fletcher–Reeves method. A more general formulation is based on a quadratic approximation to the objective function, referred to as a second-order method or quasi-Newton method. This involves progressively building up an approximation to the inverse of the Hessian matrix of second derivatives to deduce a search direction. A spreadsheet program for the Hooke and Jeeves method is also used in the next chapter for the penalty function method for constrained optimization.

Exercises

• 4.1 Repeat the first few steps of the problem in Example 4.1 for the same function $f({\mathbf{x}})$, from a different starting point.

  Follow the same analytical procedure as in the example. Plot values of the variables at each step in a figure similar to Fig. 4.1.

• 4.2 Perform the first few steps of the Fletcher–Reeves method to minimize the function:

  $$ f({\mathbf{x}}) = x_{1}^{2} + x_{2}^{4} + 1. $$

  Follow the same analytical procedure as in Example 4.2 . Choose a suitable initial point to start the procedure.

• 4.3 Repeat Exercise 4.2 using the DFP update formula in Eq. (4.4).

  Make an Excel spreadsheet to calculate the update formula. Use the MMULT function in Excel to perform the matrix multiplication.

• 4.4 Repeat Exercise 4.3 using the BFGS update formula in Eq. (4.5).

  Extend the spreadsheet in Exercise 4.3 to calculate the BFGS update formula.

• 4.5 Use the golden section method to find the minimum of the function:

  $$ f(x) = 2x^{3} - 3x + 2. $$

  Use initial lower and upper bounds $x_{L} = 0,\,x_{{{\kern 1pt} U}} = 1$ to start region elimination. Continue until the search interval has been reduced to about 10 per cent of its initial value.

• 4.6 Complete the minimization of the function in Exercise 4.5 by parabolic interpolation.

  Use the standard formulae for parabolic interpolation in Sect. 4.2.2 . Compare the result with the exact minimum found by differentiation of the original function.

• 4.7 Perform the first few steps of the procedure in the spreadsheet ‘Hooke and Jeeves Method’ by hand and compare the results with those in Table 4.3.

  Take the same quadratic function with the same initial values and step size. Observe the working of the series of ‘If’ and ‘ElseIf’ statements enabling a pattern move towards the end of function HJ.

• 4.8 Use the spreadsheet ‘Hooke and Jeeves Method’ to repeat the optimization of the quadratic function in Table 4.3 with different initial values and step sizes.

  Values of the variables and minimum of the function on completion of each step size can be seen by progressively increasing the number of reductions in step size entered on the spreadsheet. Observe the number of function evaluations necessary for the same accuracy with different initial values and step size.

• 4.9 Use the spreadsheet ‘Hooke and Jeeves Method’ to optimize the angles α and β of the seven-bar truss in Fig. 1.15 of Chap. 1.

  Formulae for analysis of the truss are given in Table 1.5 of Chap. 1. Replace the Visual Basic code in function FN with some lines of code to calculate the volume of the truss in terms of design variables $x_{1} = D$ and $x_{2} = H$ . Variable FN in the function FN must be the volume of the truss for any $x_{1}$ and $x_{2}$ . Choose convenient values for the load on the truss, its span and the allowable stress. Express the result in terms of angles α and β.