Exam 1 Oct 3, closed book Place ITE 119, Time:12:30-1:45pm One double-sided cheat sheet (8.5in x 11in) allowed Bring your calculator to the exam Chapters 1-11 Error Analysis Taylor Series Roots of Equations Linear Systems Numerical Methods, Lecture 8 Prof. Jinbo Bi CSE, UConn

Today’s class Optimization Multi-dimensional unconstrained problems Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Multi-dimensional unconstrained problems Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Multi-dimensional unconstrained problems Given a function f(x1, x2, x3, …, xn) find the set of values that minimize or maximize the function Solution techniques Direct or nongradient methods Gradient or descent/ascent methods Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Random Search Brute force search using randomly selected inputs Conduct a sufficient number of samples and the optimum will eventually be selected Guaranteed to converge Will always find the global minimum Very inefficient in terms of convergence Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Random Search Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Univariate search Try changing just one variable at a time Iteratively optimize each dimension until you have arrived at an optimum You can use one-dimensional searches to improve the approximation Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Univariate search Numerical Methods, Prof. Jinbo Bi CSE, UConn It is not efficient to search along the narrow ridge area. Why don’t we directly move from p1 to p3 or p5, or we directly move from p2 – p4 or p6. Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Univariate search Changing one variable at a time can be inefficient especially along the narrow ridge toward the optimum Use the pattern or conjugate vectors to help guide you to the optimal Powell’s Method The best known method is called Powell’s method. Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Powell’s method Numerical Methods, Prof. Jinbo Bi CSE, UConn The observation we are using here is that if p1 and p2 are obtained by 1-dimensional search in the same direction but from different starting point. Then the direction determined by these two points will be directed toward the maximum. This new direction is called conjugate direction. For a quadratic function, the search along conjugate directions is proved to always converge in a finite number of steps regardless of the starting point. As nonlinear functions can often be reasonably approximated by a quadratic function, the conjugate direction methods are usually efficient, can reach quadratic convergence rate. Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Powell’s method Numerical Methods, Prof. Jinbo Bi CSE, UConn Each iteration we maintain two directions, and search along the two directions sequentially. This is used to find conjugate directions. Start from 0, search along h1, then it gives us p1, then search along h2, it gives us p2. We connect p0 and p2 to form a new direction, (may or may not be conjugate direction). Then we get p3. From p3, we go to next iteration, we keep two directions, h2 and h3, we serach along h2, then we hit p4, we then search along h3 from p4, we hit p5. Then we connect p3 and p5. Then h4 is conjugate to h3. Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Powell’s method Example: find minimum of Start with initial point X0=(5.5,2) and initial vectors (1,0) and (0,1) Find minimum along (1,0) vector Minimum at γ1= π-5.5 (point 1) Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Powell’s method Find minimum along (0,1) vector Minimum at (point 1) Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Powell’s method Now minimize along the P2-P0 vector Minimum at γ=0.9817 (h3) Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Powell’s method Find minimum along U1=(0,1) vector (point 3) Find minimum along U1=(0,1) vector Minimum at γ1=0.0497 (point 4) (h3) Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Powell’s method Find minimum along U2=(-2.3584,2.7124) vector Minimum at γ2=0.00788 (point 5) (h4) Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Powell’s method Now minimize along the U2 vector Minimum at γ=0.8035684 Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Given a starting point, use the gradient to tell you which direction to proceed The gradient gives you the largest slope out from the current position Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Numerical Methods, Prof. Jinbo Bi CSE, UConn Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods The slope at point (a,b) is Maximize the slope Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Example: Steepest ascent of f(x,y)=xy2 at (2,2) Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Numerical Methods, Prof. Jinbo Bi CSE, UConn Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Second derivative tells you whether you have reached a minimum or maximum But in multi-dimensions, it is a little trickier Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Numerical Methods, Prof. Jinbo Bi CSE, UConn A saddle point. If you think to calculate partial derivative of f(x) with respect x, calculate the partial second derivative, it is positive, so you say we reach the minimum. Then you think what happens to the y diemsjjon, then you calculate f(x,y) with respect to y, calculate for second derivative, you like both are positive. OK, this also reaches the minimum along y dimension, so hooray, we are at the minimum point, but actually, when you look at the direction defined by y=x, this point is actually the maximum point Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Compute the Hessian determinant Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Numerical Methods, Prof. Jinbo Bi CSE, UConn Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Example: Maximize f(x,y)=2xy+2x-x2-2y2, with an initial solution of (-1,1) Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Gradient Methods Find the maximum along the vector (-1,1)+γ(6,-6) Now Maximize New point is now (-1,1)+γ(6,-6) = (0.2, -0.2) Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn

Next class Constrained Optimization Read Chapter 15 Numerical Methods, Lecture 10 Prof. Jinbo Bi CSE, UConn