Exam 1 Oct 3, closed book Place ITE 119, Time:12:30-1:45pm One double-sided cheat sheet (8.5in x 11in) allowed Bring your calculator to the exam Chapters 1-11 Error Analysis Taylor Series Roots of Equations Linear Systems Numerical Methods, Lecture 8 1 Prof. Jinbo Bi CSE, UConn

2 Today’s class Optimization One-dimensional unconstrained Numerical Methods, Fall 2011 Lecture 10 2 Prof. Jinbo Bi CSE, UConn

Optimization Given a function f(x 1, x 2, x 3, …, x n ) find the set of values that minimize or maximize the function Examples: Lowering power usage in a circuit while maximizing speed Least-cost management of supply chain 3 Numerical Methods, Fall 2011 Lecture 10 3 Prof. Jinbo Bi CSE, UConn

Optimization Optimization of a differential function means setting the derivative f’(x 1, x 2, x 3, …, x n ) to zero and finding solutions to that equation. If f’’(x 1, x 2, x 3, …, x n ) > 0, it is a minimum and if f’’ (x 1, x 2, x 3, …, x n ) < 0, it is a maximum To find where the derivative is zero, we can use the root-finding techniques we discussed before 4 Numerical Methods, Fall 2011 Lecture 10 4 Prof. Jinbo Bi CSE, UConn

Optimization 5 Numerical Methods, Fall 2011 Lecture 10 5 Prof. Jinbo Bi CSE, UConn

Optimization Finding the roots of the derivative requires knowing what the derivative is Often, the function is not well-defined and you can not analytically derive the derivative Requires finite-difference approximation of the derivative 6 Numerical Methods, Fall 2011 Lecture 10 6 Prof. Jinbo Bi CSE, UConn

Optimization General optimization problem: p 7 Numerical Methods, Fall 2011 Lecture 10 7 Prof. Jinbo Bi CSE, UConn

8 Optimization Unconstrained optimization Constrained optimization Degrees of freedom n-p-m Under-constrained p+m≤n Solution possible Over-constrained p+m>n Solution unlikely 8 Numerical Methods, Fall 2011 Lecture 10 8 Prof. Jinbo Bi CSE, UConn

Optimization Linear Programming Objective function and the constraints are all linear Quadratic Programming Objective function is quadratic and the constraints are linear Nonlinear Programming Objective function is not linear or quadratic and/or the constraints are nonlinear 9 Numerical Methods, Fall 2011 Lecture 10 9 Prof. Jinbo Bi CSE, UConn

Optimization Dimensionality One-dimensional Single dependent variable Multi-dimensional Function is dependent on more than one variable 10 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Optimization How do you know that the minimum or maximum that you found is the global minimum or maximum? Multimodal systems 11 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Optimization How do you find the global extremum? Use graphical methods to gain insight Use multiple guesses to find multiple local extrema and then pick the largest as global Slight perturbations of initial guesses to see if the extrema changes For large multi-dimensional problems it is sometimes too difficult to guarantee that you have found the global extremum 12 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

One-dimensional Unconstrained Optimization Similar to root-finding Bracketing methods Golden-section search Quadratic interpolation Open methods Newton’s method 13 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search Also called Golden Ratio search Start with an interval bracket around the maximum 14 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search Assume that the function is unimodal over interval 15 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search 16 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search Pick two interior points in the interval using the golden ratio 17 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search Two possibilities 18 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search Example maximum 19 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search 20 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search 21 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search Error analysis Worst-case error is when true value is at the far-end of the sub-interval Take case when optimum value is in upper interval True value is at the left 22 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Golden-section search Relative error is (considering the worst case) 23 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Quadratic interpolation Use a second order polynomial as an approximation of the function near the optimum 24 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Quadratic interpolation Use the above formula to get a new guess and then use same method as with Golden-search to get rid of one of the guesses x 3 is the value of x that corresponds to the maximum of the quadratic function 25 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Quadratic interpolation Example maximum 26 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Quadratic interpolation 27 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Newton’s Method Newton-Raphson could be used to find the root of an function When finding a function optimum, use the fact that we want to find the root of the derivative and apply Newton-Raphson 28 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Newton’s Method Example maximum 29 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Newton’s Method Example 30 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Newton’s Method Like Newton-Raphson for roots, Newton’s method for finding optima may also diverge Can use secant-like methods using finite- difference approximations if the derivative is not available Usually used only near the optima Use Hybrid methods Bracketing methods to get near the optimum Open methods to quickly converge to the optimum 31 Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn

Next class Multi-dimensional Optimization Read Chapter 13 and Numerical Methods, Fall 2011 Lecture Prof. Jinbo Bi CSE, UConn