Download presentation

Presentation is loading. Please wait.

1
Optimization 吳育德

2
**Unconstrained Minimization**

Def : f(x), x is said to be differentiable at a point x*, if it is defined in a neighborhood N around x* and if x* +h a vector n independent of h that where the vector a is called the gradient of f(x) evaluated at x*, denote it as The term <a,h> is called the 1-st variation. and

3
**Unconstrained Minimization**

Note if f(x) is twice differentiable, then where F(x) is an n*n symmetric, called the Hessian of f(x) Then 1st variation 2nd variation

4
**Directional derivatives**

Let w be a directional vector of unit norm || w|| =1 Now consider is a function of the scalar r. Def : The directional derivative of f(x) in the direction w (unit norm) at w* is defined as

5
**Directional derivatives**

Example : Let Then i.e. the partial derivative of f(x*) w.r.t xi is the directional derivative of f(x) in the direction ei. Interpretation of Consider Then The directional derivative along a direction w (||w||=1) is the length of the projection vector of on w.

6
**Unconstrained Minimization**

[Q] : What direction w yield the largest directional derivative? Ans : Recall that the 1st variation of is Conclusion 1 : The direction of the gradient is the direction that yields the largest change (1st -variation) in the function. This suggests in the steepest decent method which will be described later

7
**Directional derivatives**

Example: Sol : Let , w with unit norm =

8
**Directional derivatives**

The directional derivative in the direction of the gradient is Notes :

9
**Directional derivatives**

Def : f(x) is said to have a local (or relative) minimum at x*, if in a nbd N of x* Theorem: Let f(x) be differentiable ,If f(x) has a local minimum at x* , then pf : Note: is a necessary condition, not sufficient condition.

10
**Directional derivatives**

Theorem: If f(x) is twice diff and pf : Conclusion2: The necessary & Sufficient Conditions for a local minimum of f(x) is

11
**Minimization of Unconstrained function**

Prob. : Let y=f(x) , We want to generate a sequence and such that it converges to the minimum of f(x). Consider the kth guess, , we can generate provided that we have two of information (1) the direction to go (2) a scalar step size Then Basic descent methods (1) Steepest descent (2) Newton-Raphson method

12
Steepest Descent Steepest descent : Note 1.a. Optimum it minimizes

13
Steepest Descent Example :

14
Steepest Descent Example :

15
**Steepest Descent Optimum iteration Remark :**

The optimal steepest descent step size can be determined analytically for quadratic function.

16
**Steepest Descent 1.b. other possibilities for choosing**

Constant step size i.e. adv : simple disadv : no idea of which value of α to choose If α is too large diverge If α is too small very slow Variable step size

17
**Steepest Descent 1.b. other possibilities for choosing**

Polynomial fit methods (i) Quadratic fit gauss three values for α, say α1 , α2 , α3. Let Solve for a, b, c minimize by Check

18
**Steepest Descent 1.b. other possibilities for choosing**

Polynomial fit methods (ii) Cubic fit

19
**Steepest Descent 1.b. other possibilities for choosing**

Region elimination methods Assume g(α) is convex over [a,b] i.e. one minimum (a) g1>g (b)g1<g (c)g1=g2 eliminated eliminated eliminated eliminated initial interval of uncertainty [a,b] , next interval of uncertainty for (i) is [ ,b]; for (ii) is [a, ]; for (iii) is [ , ]

20
**Steepest Descent [Q] : how do we choose and ?**

(i) Two points equal interval search i.e. α1- a = α1- α2=b- α1 1st iteration 2nd iteration 3rd iteration kth iteration

21
**Steepest Descent [Q] : how do we choose and ?**

(ii) Fibonacci Search method For N-search iteration Example: Let N=5, initial a = 0 , b = 1 k=0

22
**Steepest Descent [Q] : how do we choose and ?**

(iii) Golden Section Method then use until Example: then then etc…

23
**x(k+1)c=x(k)- α(k) ▽f(x(k))**

Steepest Descent Flow chart of steepest descent Initial guess x(0) Stop! x(k) is minimum Compute ▽f(x(k)) ∥ ▽f(x(k)) ∥﹤ε Yes k=k+1 No α {α1，…αn} Polynomial fit : cubic ,… Region elimination : … Determine α(k) x(k+1)c=x(k)- α(k) ▽f(x(k))

24
**Steepest Descent [Q]: is the direction of the “best” direction to go?**

suppose the initial guess is x(0) Consider the next guess What should M be such that x(1) is the minimum, i.e ? Since we want If MQ=I，or M=Q-1 Thus，for a quadratic function，x(k+1)=x(k)-Q-1▽f(x(k)) will take us to the minimum in one iteration no matter what x(0) is.

25
**Newton-Raphson Method**

Minimize f(x) The necessary condition ▽f(x)=0 The N-R algorithm is to find the roots of ▽f(x)=0 Guess x(k)，then x(k+1) must satisfy Note not always converge

26
**Newton-Raphson Method**

A more formal derivation Min f(x(k)+h) w.r.t h

27
**Newton-Raphson Method**

Remarks： （1）computation of [F(x(k))]-1 at every iteration → time consuming → modify N-R algorithm to calculate [F(x(k))]-1 every M-th iteration （2）must check F(x(k)) is p.d. at every iteration. If not → Example ：

28
**Newton-Raphson Method**

The minimum of f(x) is at (0,0) In the nbd of (0,0) is p.d. Now suppose we start an initial guess Then diverges. Remark： （3）N-R algorithm is good(fast) when initial guess close to minimum ，but not very good when far from minimum.

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google