1. Performance Surfaces

2. Taylor Series ExpansionF x ( ) * d = – + 1 2 - n !

3. Example Taylor series of F(x) about x* = 0 :
Taylor series approximations:

4. Plot of Approximations

5. Vector Case F x ( ) * 1 = – 2 + n -

6. Matrix Form Gradient Hessian x x x x x x x x x x x x x x x x * F ( ) =+ Ñ F ( x ) ( x – x * ) x x * = 1 T + - - - ( x – x * ) Ñ 2 F ( x ) ( x – x * ) + 2 x x * =
Gradient
Hessian F x ( ) Ñ 2 1 n = F x ( ) Ñ 1 2 n =

7. Directional Derivatives
First derivative (slope) of F(x) along xi axis:
(ith element of gradient)
Second derivative (curvature) of F(x) along xi axis:
(i,i element of Hessian) p T F x ( ) Ñ -
First derivative (slope) of F(x) along vector p: T p Ñ 2 F ( x ) p
Second derivative (curvature) of F(x) along vector p: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - p 2

8. Example p T F x ( ) Ñ - 1 – 2 =

9. Plots
Directional
Derivatives
1.4
1.3 x2
1.0
0.5
0.0 x2 x1 x1

10. Minima
Strong Minimum
The point x* is a strong minimum of F(x) if a scalar d > 0 exists, such that F(x*) < F(x* + Dx) for all Dx such that d > ||Dx|| > 0.
Global Minimum
The point x* is a unique global minimum of F(x) if
F(x*) < F(x* + Dx) for all Dx ° 0.
Weak Minimum
The point x* is a weak minimum of F(x) if it is not a strong minimum, and a scalar d > 0 exists, such that F(x*) Š F(x* + Dx) for all Dx such that d > ||Dx|| > 0.

11. Scalar Example
Strong Maximum
Strong Minimum
Global Minimum

12. Vector Example

13. First-Order Optimality Conditionx ( ) * D + Ñ T = 1 2 -
For small Dx:
If x* is a minimum, this implies: If
then
But this would imply that x* is not a minimum. Therefore
Since this must be true for every Dx,

14. Second-Order Condition
If the first-order condition is satisfied (zero gradient), then
A strong minimum will exist at x* if
for any Dx ° 0.
Therefore the Hessian matrix must be positive definite. A matrix A is positive definite if:
for any z ° 0.
This is a sufficient condition for optimality.
A necessary condition is that the Hessian matrix be positive semidefinite. A matrix A is positive semidefinite if:
for any z.

15. Example x F ( ) + = (Not a function of x in this case.)1 2 + =
(Not a function of x
in this case.)
To test the definiteness, check the eigenvalues of the Hessian. If the eigenvalues
are all greater than zero, the Hessian is positive definite.
Both eigenvalues are positive, therefore strong minimum.

16. Quadratic Functions Gradient and Hessian: (Symmetric A)
Useful properties of gradients:
Gradient of Quadratic Function:
Hessian of Quadratic Function:

17. Eigensystem of the Hessian
Consider a quadratic function which has a stationary
point at the origin, and whose value there is zero.
Perform a similarity transform on the Hessian matrix,
using the eigenvalues as the new basis vectors.
Since the Hessian matrix is symmetric, its eigenvectors
are orthogonal. A ' B T [ ] l 1 2 n L =

18. Second Directional Derivativep T F x ( ) Ñ 2 - A =
Represent p with respect to the eigenvectors (new basis): p T A 2 - c B L ( ) l i 1 = n å l m i n p T A 2 - a x

19. Eigenvector (Largest Eigenvalue)c B T p z m a x 1 = z m a x T A 2 - l i c 1 = n å
The eigenvalues represent curvature
(second derivatives) along the eigenvectors
(the principal axes).

20. (Any two independent vectors in the plane would work.)
Circular Hollow
(Any two independent vectors in the plane would work.)

21. Elliptical Hollow

22. Elongated Saddle F x ( ) 1 4 - 2 – 3 T
0.5
1.5 =

23. Stationary Valley F x ( ) 1 2 - – + T =

24. Quadratic Function Summary
If the eigenvalues of the Hessian matrix are all positive, the function will have a single strong minimum.
If the eigenvalues are all negative, the function will have a single strong maximum.
If some eigenvalues are positive and other eigenvalues are negative, the function will have a single saddle point.
If the eigenvalues are all nonnegative, but some eigenvalues are zero, then the function will either have a weak minimum or will have no stationary point.
If the eigenvalues are all nonpositive, but some eigenvalues are zero, then the function will either have a weak maximum or will have no stationary point.
Stationary Point: