1. Tutorial 12 Unconstrained optimization Conjugate gradients

2. M4CS 20052 Tutorial 11 Suppose that we want to minimize the quadratic function where Q is a symmetric positive definite matrix, and x has n components. As we well know, the minimum x* is the solution to the linear system The explicit solution of this system (Newton’s Method) requires about O(n 3 ) operations and O(n 2 ) memory, which is very expensive. Method of Conjugate Gradients

3. M4CS 20053 Tutorial 11 We now consider an alternative solution method that does not need the inversion of Q, but only the gradient of f(x k ) (just like SD, but better) evaluated at n different points x 1, …, x n. Conjugate Gradients 2 Conjugate Gradient Gradient

4. M4CS 20054 Tutorial 11 Conjugate Gradients 3 Consider, for example, the case n=3, in which the variable x in f(x) is a three-dimensional vector. Then the quadratic function f(x) is constant over 3D ellipsoids, called isosurfaces, centered at the minimum x*. How can we start from a point x 0 on one of these ellipsoids and reach x* by a finite sequence of one-dimensional searches? In the steepest descent, for the poorly conditioned Hessians, orthogonal directions lead to many small steps, that lead to slow convergence.

5. M4CS 20055 Tutorial 11 Conjugate Gradients: Spherical Case In the spherical case, the very first step in the direction of the gradient takes us to x* right away. Suppose however that we cannot afford to compute this special direction p 1 orthogonal to p 0, but that we can only compute some direction p 1 orthogonal to p 0 (there is an n-1 dimensional space of such directions!) and reach the minimum of f(x) in this direction. In that case n steps will take us to x* of the sphere, since coordinate of the minimum in each on the n directions is independent of others.

6. M4CS 20056 Tutorial 11 Conjugate Gradients: Elliptical Case Any set of orthogonal directions, with a line search in each direction, will lead to the minimum for spherical isosurfaces. Given an arbitrary set of ellipsoidal isosurfaces, there is a one-to-one mapping with a spherical system: if Q = UΣU T is the SVD of the symmetric positive definite matrix Q, then we can write,where

7. M4CS 20057 Tutorial 11 Elliptical Case 2 Consequently, there must be a condition for the original problem (in terms of Q ) that is equivalent to orthogonality for the spherical problem. If two directions y i and y j are orthogonal in the spherical context, that is, if, where What does this translate into in terms of the directions x i and x j for the ellipsoidal problem? We have This condition is called Q -conjugacy, or Q -orthogonality : if this equation holds, then x i and x j are said to be Q -conjugate or Q - orthogonal to each other, or simply "conjugate". (2) (1)

8. M4CS 20058 Tutorial 11 Elliptical Case 3 In summary, if we can find n directions p 0,…,p n-1, that are mutually conjugate, i.e. comply with (2), and if we do line minimization along each direction p i we reach the minimum in at most n steps. Such algorithm, would be named “Conjugate Direction (CD)”. A special case we will consider is to drive the construction of these directions from the local gradients, thus giving birth to the “Conjugate Gradients”. Of course, we cannot use the transformation (1) in the algorithm, because Σ and especially U T are too large. So for computational efficiency we need to find a method for generating n conjugate directions without using SVD or other complex processes on the Hessian Q.

9. M4CS 20059 Tutorial 11 Hestenes Stiefel Procedure Here First step is like NSD with optimal line search Once we have the new solution, the gradient is evaluated there The next search direction is built by taking the current gradient vector and Q-orthogonalizing it with all previous directions

10. M4CS 200510 Tutorial 11 Hestenes Stiefel Procedure 2 Let us check that the directrions are indeed Q-conjugate. First, it is easy to see that p 1 and p 0 are conjugate. Now assume that p 0,…, p k are already mutually conjugate and let us verify that p k+1 is conjugate to each of them, i.e. for arbitrary j: One can see that the vectors p k are found by a generalization of Gram- Schmidt to produce conjugate rather than orthogonal vectors. In practical cases, it can be worth to dismiss except for. For the sake of simplicity, we will assume that this is the case.

11. M4CS 200511 Tutorial 11 Removing the Hessian In the described algorithm the expression for p k contains the Hessian Q, which is too large. We now show that p k can be rewritten in terms of the gradient values g k and g k+1 only. To this end, we notice that or Proof: So that

12. M4CS 200512 Tutorial 11 We can therefore write and Q has disappeared. This expression for p k can be further simplified by noticing that because the line along p k is tangent to an isosurface at x k+l, while the gradient g k+l is orthogonal to the isosurface at x k+l. Similarly, Then, the denominator of becomes: Removing the Hessian 2

13. M4CS 200513 Tutorial 11 In conclusion, we obtain the Polak-Ribiere formula Polak-Ribiere formula

14. M4CS 200514 Tutorial 11 When the function f(x) is arbitrary, the same algorithm can be used, but n iterations will not suffice, since the Hessian, which was constant for the quadratic case, now is a function of x k. Strictly speaking, we then lose conjugacy, since p k and p k+l are associated to different Hessians. That is the reason why it is worth to keep conjugacy only between p k+1 and p k, setting ). However, as the algorithm approaches the minimum x*, the quadratic approximation becomes more and more valid, and a few cycles of n iterations each will achieve convergence. General Case

15. M4CS 200515 Tutorial 11 Consider the elliptic function: f(x,y)=(x-1) 2 +2(y-1) 2 and find the first three terms of Taylor expansion. Find the first step of Steepest Descent, from (0,0). Conjugate gradients: example 1 2 -f’(0)

16. M4CS 200516 Tutorial 11 Conjugate gradients: example 1 1 -f’(0)