# CS B553: A LGORITHMS FOR O PTIMIZATION AND L EARNING Univariate optimization.

## Presentation on theme: "CS B553: A LGORITHMS FOR O PTIMIZATION AND L EARNING Univariate optimization."— Presentation transcript:

CS B553: A LGORITHMS FOR O PTIMIZATION AND L EARNING Univariate optimization

x f(x)

K EY I DEAS Critical points Direct methods Exhaustive search Golden section search Root finding algorithms Bisection [More next time] Local vs. global optimization Analyzing errors, convergence rates

x f(x) Local maxima Local minima Inflection point Figure 1

x f(x) ab Figure 2a

x f(x) ab Find critical points, apply 2 nd derivative test Figure 2b

x f(x) ab Figure 2b

x f(x) ab Global minimum must be one of these points Figure 2c

x f(x) ab Exhaustive grid search Figure 3

x f(x) ab Exhaustive grid search

x f(x) Two types of errors x* xtxt f(x t ) f(x * ) Geometric error Analytical error Figure 4

x f(x) a b Does exhaustive grid search achieve  /2 geometric error?  x*

x f(x) a b Does exhaustive grid search achieve  /2 geometric error? Not necessarily for multi-modal objective functions Error x*

L IPSCHITZ CONTINUITY Slope +K Slope -K |f(x)-f(y)|  K|x-y| Figure 5

x f(x) a b Exhaustive grid search achieves K  /2 analytical error in worst case  Figure 6

x f(x) a b Golden section search m Bracket [a,b] Intermediate point m with f(m) < f(a),f(b) Figure 7a

x f(x) a b Golden section search m Candidate bracket 1 [a,m] c Candidate bracket 2 [c,b] Figure 7b

x f(x) a b Golden section search m Figure 7b

x f(x) a b Golden section search m c Figure 7b

x f(x) ab Golden section search m Figure 7b

x f(x) a b Optimal choice: based on golden ratio m Choose c so that (c-a)/(m-c) = , where  is the golden ratio => Bracket reduced by a factor of  -1 at each step c

N OTES

x f(x) Root finding: find x-value where f’(x) crosses 0 f’(x) Figure 8

Bisection g(x) ab Bracket [a,b] Invariant: sign(f(a)) != sign(f(b)) Figure 9a

Bisection g(x) ab Bracket [a,b] Invariant: sign(f(a)) != sign(f(b)) m Figure 9

Bisection g(x) ab Bracket [a,b] Invariant: sign(f(a)) != sign(f(b)) Figure 9

Bisection g(x) ab Bracket [a,b] Invariant: sign(f(a)) != sign(f(b)) m Figure 9

Bisection g(x) ab Bracket [a,b] Invariant: sign(f(a)) != sign(f(b)) Figure 9

Bisection g(x) ab Bracket [a,b] Invariant: sign(f(a)) != sign(f(b)) m Figure 9

Bisection g(x) ab Bracket [a,b] Invariant: sign(f(a)) != sign(f(b)) Linear convergence: Bracket size is reduced by factor of 0.5 at each iteration Figure 9

N EXT TIME Root finding methods with superlinear convergence Practical issues

Download ppt "CS B553: A LGORITHMS FOR O PTIMIZATION AND L EARNING Univariate optimization."

Similar presentations