2. 84 b Unidimensional Search Methods Most algorithms for unconstrained and constrained optimisation use an efficient unidimensional optimisation technique to locate a local minimum of a function of one variable.

3. 85 Basic principle - acceleration and bracketing A search for the solution x * which minimizes f ( x ) is performed by computing the sequence: { x o, x 1, x 2,..., x k, x k+1,..., x N -2, x N -1, x N } by repeated application of the formula:- x k+1 = x k +  k starting from an initial guess x o, evaluating the function f ( x k ) each time with initial direction chosen such that f ( x 1 ) f ( x N-1 ).

4. 86 x f(x)f(x) xoxo x1x1 xkxk x k+1 x N -1 x*x* xNxN fofo f1f1 fkfk f k+1 fNfN f N -2 f*f* interval of uncertainty kk x N -2 f N -1

5. 87 Then the required solution x * will be bracketed by the range x N-2 to x N, and | x N - x N-2 | is known as the interval of uncertainty. Often the step size  k is variable, for example using an acceleration technique:  k = 2 k.  which doubles the step size every iteration. Once the minimum has been bracketed, an estimate of x * can be obtained using a polynomial approximation such as quadratic or cubic interpolation. This estimate can be improved, and the interval of uncertainty reduced, by repeated application of the interpolation.

6. 88 x f(x)f(x) x N-2 x N-1 xNxN x*x* f N-2 f N-1 fNfN It can be shown that: e.g. quadratic interpolation (without derivatives)

7. 89 Rates of Convergence When comparing different optimisation techniques it is useful to examine their rates of convergence - a common classification is as follows: quadratic : (fast) superlinear : (fast) linear : (slow)

8. 90 Newton’s Method Suppose that f ( x ) is approximated by a quadratic function at a point x k ( Taylor series :-) Then the stationary point, d f ( x )/d x = 0, is given as: yielding the next approximation x k +1 = x as:-

9. 91 x*x* x k+1 xkxk x 0

10. 92 disadvantages : 1. need to compute 2. if then slow convergence. advantages : 1. locally quadratically convergent. 2. for a quadratic function, x* is found in a single iteration.

11. 93 Quasi-Newton Methods A quasi-Newton method is one that imitates Newton’s method. For example, instead of computing and exactly, we can use finite difference approximations: where  is a small step size (chosen to suit the computer machine precision).

12. 94 This then provides the updating formula: The disadvantage is the need to perform small perturbations to x k at each iteration, hence, slowing down the progress towards the solution.

13. 95 Example: Minimisation of: (i) acceleration plus quadratic interpolation:

14. 96 Tabulated Results iter x f 1 3.0000 79.0000 2 2.9900 77.9354 3 2.9800 76.8815 4 2.9600 74.8056 5 2.9200 70.7795 6 2.8400 63.2139 7 2.6800 49.9069 8 2.3600 29.6604 9 1.7200 8.0321 10 0.4400 0.5975 iter x f 11 -2.1200 23.3196 12 0.3205 0.6900 13 0.4626 0.5832 14 0.7663 0.5785 15 0.6185 0.5278 16 0.6178 0.5279 17 0.6284 0.5275 18 0.6301 0.5275 19 0.6300 0.5275 interpolation phase

15. 97 (ii) Newton’s method:

16. 98

17. 99 (ii) Quasi Newton’s method:

18. 100

19. 101 How Unidimensional Search is Applied in a Multidimensional Problem In minimizing a function f ( x ) of several variables, a common procedure is to (a) calculate a search direction s (a vector) (b) take steps in that search direction to reduce the value of f ( x )

20. 102

21. 103 Unconstrained Multivariable Optimisation Univariate Search Select n fixed search directions, using the coordinate directions, then minimize f ( x ) in each direction sequentially using line search. In general, this method is not efficient.

22. 104 xoxo x1x1 x2x2 x3x3 x4x4

23. 105 Simplex Method This is not a line search method Use a regular geometric figure ( a simplex ) and evaluate f ( x ) at each vertex. In two dimensions a simplex is a triangle, in three dimensions it is a tetrahedron. At each iteration a new point is selected by reflecting the simplex opposite the vertex with the highest function value, which is then discarded to form a new simplex. The iterations proceed until the simplex straddles the optimum. The size is reduced and the procedure is repeated.

24. 106 3 1 2 4 highest f ( x ) The simplex can expand and contract continuously throughout the search ( Nelder and Mead Method )

25. 107 Conjugate Search Directions A set of n linearly independent search directions: s o, s 1,..., s n-1 are said to be conjugate with respect to a positive definite matrix Q if: where, in optimisation, Q is the Hessian matrix of the objective function. (i.e. Q = H ). If f ( x ) is quadratic you are guaranteed to reach the minimum in n stages ( n line searches along conjugate directions).

26. 108 soso s1s1 xoxo x1x1 x2x2

27. 109 Minimization of a Quadratic Function Using Line Search Consider the function f ( x ) approximated by a quadratic at point x k and consider the line search Then along this line and is an extremum, with respect to , when

28. 110 Example: Consider the minimization of: with initial direction: Find a direction s 1 which is conjugate to s o and verify that the minimum of f ( x ) can be reached in two line searches, firstly using s o and then using s 1.

29. 111 We require Hence, we can use Search along s o ( Note :

30. 112 Hence, Next stage - search along s 1 Then: and: which is clearly the minimum of f ( x ).

31. 113 soso s1s1 xoxo x1x1 x2x2

32. 114 Construction of conjugate directions without using derivatives xoxo xaxa s -s x1x1 xbxb Start at x o and locate x a as the minimum of f ( x ) in the direction s. Then start at another point x 1 and locate x b as the minimum of f ( x ) in the same parallel direction s (or - s ). The the vector ( x b - x a ) is conjugate to s ( provided f ( x ) is quadratic ). Also, the gradient of f ( x ) at x a is orthogonal to s. i.e.

33. 115 Powell’s Method (basic form) The k th stage employs n linearly independent search directions. Initially, these are usually the co-ordinate directions (Hence, stage 1 is univariate search). At subsequent stages these search directions change such that for a quadratic function they become conjugate. Step 1 From determine by line search in direction so that is a minimum. Let From determine so that is a minimum. Let Continue this procedure until all search directions starting always from the last immediate point in sequence, until all are determined. The final point is

34. 116 Step 2 Now search from along the direction to determine the point which minimizes This point then becomes the new starting point for the next stage. Step 3 Now replace one of the search directions by s k (e.g. replace by s k ), and repeat from step 1. Termination criterion Keep repeating the steps of the algorithm until: where  is a defined tolerance.

35. 117 For example, in two variables the first iteration would be:

36. 118 Gradient Search Methods Methods such as univariate, simplex and Powell do not require the use of derivative information in determining the search direction, and are known as direct methods. In general, they are not as efficient as indirect methods which make use of derivatives, first or second, in determining search directions. For minimization, a good search direction should reduce the objective function at each step of a linear search. i.e.

37. 119 For s k +1 to be a descent direction  > 90 o, hence: 10 6 3 sksk s k +1 region of descent directions 

38. 120 Method of Steepest Descent The gradient gives the direction of local greatest increase of f ( x ) and is normal to the contour of f ( x ) at x. Hence, a local search direction: will give the greatest local decrease of f(x). This is the basis of the steepest descent algorithm.

39. 121 Step 1 Choose an initial point x o (thereafter x k ) Step 2 Calculate (analytically or numerically) the partial derivatives Step 3 Calculate the search direction vector: Step 4 Perform a line search: to find x k +1 which minimizes f ( x k +  k s k ). Step 5 Repeat from step 2 until a desired termination criterion is satisfied.

40. 122 Notes : 1) Convergence is slow for a badly scaled f ( x ) 2) Exhibits zig-zag behaviour

41. 123 Example: First few iterations of the steepest descent algorithm xoxo

42. 124 Fletcher and Reeves Conjugate Gradient Method This method computes the new search direction by a linear combination of the current gradient and the previous search direction. For a quadratic function this produces conjugate search directions and, hence, will minimize such a function in at most n iterations. The method represents a major improvement over the steepest descent algorithm with only minor extra computational requirements.

43. 125 Step 1 : At x o calculate f ( x o ) and let Step 2 : Use line search to determine to minimize f ( x o +  o s o ). Compute f ( x 1 ), Step 3 : Compute the new search direction:- (at iteration k) Step 4 : Test for convergence and if not satisfied repeat from step 2.

44. 126 Example: The Fletcher and Reeves algorithm minimizes a two dimensional quadratic function in two iterations x*x* xoxo

45. 127 Example: Consider the same example as used earlier of minimizing: which is the same initial direction as used previously.

46. 128 The next search direction, using Fletcher and Reeves is: which provides the same search direction as previously. Hence, s o and s 1 are conjugate and the minimum will be obtained by searching along s o and s 1 in sequence.

47. 129 Newton’s Method This is a second-order method and makes use of second-order information of f ( x ) through a quadratic approximation of f ( x ) at x k. Differentiating f ( x ) with respect to  x k produces:- = 0 for a minimum when:- and defining  x k = x k+1 - x k provides the iteration: where is the search direction, and k, proportional to step length, is introduced to regulate convergence.

48. 130 disadvantage : need to determine, analytically or numerically, second order partial derivatives in order to compute the Hessian matrix H ( x ).

49. 131 = Newton search direction. Newton’s method will minimize a quadratic function in a single linear search. xk=x*xk=x* xkxk s

50. 132 Example : Investigate the application of Newton’s method for minimizing the convex quadratic function: Now consider the first step of the Newton algorithm with o = 1 :- Clearly, x 1 = x *, the minimum of f ( x ).

51. 133 Quasi-Newton Methods These methods approximate the Hessian matrix using only the first partial derivatives of f ( x ). Define:- Basically, the methods update an estimate of the Hessian matrix, or it’s inverse. Two particular updating mechanisms are:

52. 134 (i)DFP (Davidon-Fletcher-Powell) (ii)BFGS (Broyden-Fletcher-Goldfarb-Shanno) Note: It is important that in any Newton minimization method the Hessian matrix updates remain +ve def. Both DFP and BFGS have this property.

53. 135 Test Example - Rosenbrocks Banana Shaped Valley

54. 136 It is called the banana function because of the way the curvature bends around the origin. It is notorious in optimization examples because of the slow convergence with which most methods exhibit when trying to solve this problem.

55. 137

56. 138 Warning: Maximum number of iterations has been exceeded Final solution: x = 0.8842 0.7813 f = 0.0134 number of function evaluations = 1002

57. 139 final solution x = 1.0000 f = 8.0378e-015 number of function evaluations = 85

58. 140 final solution x = 1.0001 1.0003 f = 2.2498e-008 number of function evaluations = 86

59. 141 final solution x = 1.0000 1.0001 f = 1.0314e-009 number of function evaluations = 212

60. 142 Termination - Stopping Criteria In iterative optimisation, the following criteria, often used simultaneously, are recommended for testing convergence. (  i = scalar tolerance value (small))

61. 143 Nonlinear Programming With Constraints Quadratic Programming - equality constraints case Quadratic Programming (QP) is sometimes used within techniques employed for solving nonlinear optimisation problems with equality and inequality constraints. Here we will limit ourselves to equality constraints only. Consider the problem: where Q is +ve def. Forming the Lagrangian, and the necessary conditions for a minimum, gives:

62. 144 That is an analytical solution giving from (a) and substituting into (b) Hence:

63. 145 Sequential Quadratic Programming (SQP) The basic idea is that, at each iteration ( x = x k ), the objective function is approximated locally by a quadratic function and the constraints are approximated by linear functions. Then quadratic programming is used recursively. The method is applied to the general problem: but uses an active set strategy to replace it with: where contains h ( x ) and the active inequality constraints. (those of which are

64. 146 Example : Application of SQP to the problem:

65. 147 final solution x = 1.0013 4.8987 f = -31.9923 h = 0 g = 0 19.1949

66. 148 MATLAB Optimisation Toolbox Routines fminbnd scalar minimization on a fixed interval ( fmin ) Fminunc multivariable unconstrained minimization (Quasi Newton BFGS, or DFP; or steepest descent) ( fminu ) fminsearch multivariable unconstrained minimization (Simplex - Nelder and Mead) ( fmins ) fmincon constrained minimization (SQP) ( constr ) fsolve non-linear equation solver linprog linear programming ( lp ) (Earlier version names in italics)

67. 149 Example:

68. 150 MATLAB Program:

69. 151 where ‘fopt22’ is a function m file:

70. 152 Results : (443 function evaluations)