1. Multivariable optimization with no constraints
Definition: rth Differential of f
If all partial derivatives of the function f through order r ≥ 1 exist and are continuous at a point X*, the polynomial
is called the rth differential of f at X*.
r summations

2. Multivariable optimization with no constraints
Example: rth Differential of f
when r = 2 and n = 3, we have
r summations

3. Multivariable optimization with no constraints
Definition: rth Differential of f
The Taylor series expansion of a function f (X*) about a point X* is given by:
where the last term, called the remainder is given by:

4. Example
Find the second order Taylor’s series approximation of the function
about the point
Solution: The second order Taylor’s series approximation of the function f about point X* is given by

5. Example cont’d
where

6. Example cont’d
where

7. Example cont’d Thus, the Taylor’s series approximation is given by:
Where h1=x1-1, h2=x2, and h3=x3+2

8. Multivariable optimization with no constraints
Necessary condition
If f(X) has an extreme point (maximum or minimum) at X=X* and if the first partial derivatives of f (X) exist at X*, then
Sufficient condition
A sufficient condition for a stationary point X* to be an extreme point is that the matrix of second partial derivatives (Hessian matrix) of f (X*) evaluated at X* is
Positive definite when X* is a relative minimum point
Negative definite when X* is a relative maximum point

9. Semi-definite case
The sufficient conditions for the case when the Hessian matrix of the given function is semidefinite:
In case of a function of a single variable, the higher order derivatives in the Taylor’s series expansion are investigated

10. Semi-definite case
The sufficient conditions for a function of several variables for the case when the Hessian matrix of the given function is semidefinite:
Let the partial derivatives of f of all orders up to the order k ≥ 2 be continuous in the neighborhood of a stationary point X*, and
so that dk f |X=X* is the first nonvanishing higher-order differential of f at X*.
If k is even:
X* is a relative minimum if dk f |X=X* is positive definite
X* is a relative maximum if dk f |X=X* is negative definite
If dk f |X=X* is semidefinite, no general conclusions can be drawn
If k is odd, X* is not an extreme point of f(X*)

11. Saddle point
In the case of a function of two variables f (x,y), the Hessian matrix may be neither positive nor negative definite at a point (x*,y*) at which
In such a case, the point (x*,y*) is called a saddle point.
The characteristic of a saddle point is that it corresponds to a relative minimum or maximum of f (x,y) wrt one variable, say, x (the other variable being fixed at y=y* ) and a relative maximum or minimum of f (x,y) wrt the second variable y (the other variable being fixed at x*).

12. Saddle point
Example: Consider the function
f (x,y)=x2-y2. For this function:
These first derivatives are zero at x* = 0 and y* = 0. The Hessian matrix of f at (x*,y*) is given by:
Since this matrix is neither positive definite nor negative definite, the point ( x*=0, y*=0) is a saddle point.

13. Saddle point Example cont’d:
It can be seen from the figure that f (x, y*) = f (x, 0) has a relative minimum and f (x*, y) = f (0, y) has a relative maximum at the saddle point (x*, y*).

14. Example Find the extreme points of the function
Solution: The necessary conditions for the existence of an extreme point are:
These equations are satisfied at the points: (0,0), (0,-8/3), (-4/3,0), and (-4/3,-8/3)

15. Example
Solution cont’d: To find the nature of these extreme points, we have to use the sufficiency conditions. The second order partial derivatives of f are given by:
The Hessian matrix of f is given by:

16. Solution cont’d: Example
If J1=|6x1+4| and , the values of J1 and J2 and
the nature of the extreme point are as given in the next slide:

17. Example Point X Value of J1 Value of J2 Nature of J Nature of X f (X)
(0,0) +4
+32
Positive definite
Relative minimum 6
(0,-8/3)
-32
Indefinite
Saddle point
418/27
(-4/3,0) -4
194/27
(-4/3,-8/3)
Negative definite
Relative
maximum
50/3

18. Multivariable optimization with equality constraints
Problem statement:
Minimize f = f (X) subject to gj(X)=0, j=1,2,…..,m where
Here m is less than or equal to n, otherwise the problem becomes overdefined and, in general, there will be no solution.
Solution:
Solution by direct substitution
Solution by the method of constrained variation
Solution by the method of Lagrange multipliers

19. Solution by direct substitution
For a problem with n variables and m equality constraints:
Solve the m equality constraints and express any set of m variables in terms of the remaining n-m variables
Substitute these expressions into the original objective function, the result is a new objective function involving only n-m variables
The new objective function is not subjected to any constraint, and hence its optimum can be found by using the unconstrained optimization techniques.

20. Solution by direct substitution
Simple in theory
Not convenient from a practical point of view as the constraint equations will be nonlinear for most of the problems
Suitable only for simple problems

21. Solution by constrained variation
Minimize f (x1,x2)
subject to g(x1,x2)=0
A necessary condition for f to have a minimum at some point (x1*,x2*) is that the total derivative of f (x1,x2) wrt x1 must be zero at (x1*,x2*)
Since g(x1*,x2*)=0 at the minimum point, any variations dx1 and dx2 taken about the point (x1*,x2*) are called admissable variations provided that the new point lies on the constraint:

22. Solution by constrained variation
Taylor’s series expansion of the function about the point (x1*,x2*):
Since g(x1*, x2*)=0
Assuming
Substituting the above equation into

23. Solution by constrained variation
The expression on the left hand side is called the constrained variation of f
Since dx1 can be chosen arbitrarily:
This equation represents a necessary condition in order to have (x1*,x2*) as an extreme point (minimum or maximum)

24. Example
A beam of uniform rectangular cross section is to be cut from a log having a circular cross secion of diameter 2 a. The beam has to be used as a cantilever beam (the length is fixed) to carry a concentrated load at the free end. Find the dimensions of the beam that correspond to the maximum tensile (bending) stress carrying capacity.

25. Example
Solution: From elementary strength of materials, we know that the tensile stress induced in a rectangular beam  at any fiber located at a distance y from the neutral axis is given by
where M is the bending moment acting and I is the moment of inertia of the cross-section about the x axis. If the width and the depth of the rectangular beam shown in the figure are 2x and 2y, respectively, the maximum tensile stress induced is given by:
I = BD^3/12

26. Example
solution cont’d: Thus for any specified bending moment, the beam is said to have maximum tensile stress carrying capacity if the maximum induced stress (max) is a minimum. Hence we need to minimize k/xy2 or maximize Kxy2, where k=3M/4 and K=1/k, subject to the constraint
This problem has two variables and one constraint; hence the equation
can be applied for finding the optimum solution.

27. Example
Solution: Since
we have:
Equation gives:

28. Example Solution: that is
Thus the beam of maximum tensile stress carrying capacity has a depth of 2 times its breadth. The optimum values of x and y can be obtained from the above equation and
as:

29. Example of direct substitution method.
Find the dimensions of a box of largest volume that can be inscribed in a sphere of unit radius
Solution: Let the origin of the Cartesian coordinate system x1, x2, x3 be at the center of the sphere and the sides of the box be 2x1, 2x2, and 2x3. The volume of the box is given by:
Since the corners of the box lie on the surface of the sphere of unit radius, x1, x2 and x3 have to satisfy the constraint

30. Example
This problem has three design variables and one equality constraint. Hence the equality constraint can be used to eliminate any one of the design variables from the objective function. If we choose to eliminate x3:
Thus, the objective function becomes:
f(x1,x2)=8x1x2(1-x12-x22)1/2
which can be maximized as an unconstrained function in two variables.

31. Example
NOW differentiate the function f with respect to X1 and X2 and then find out the values of X1, X2 and X3.

32. Example The necessary conditions for the maximum of f give:
which can be simplified as:
From which it follows that x1*=x2*=1/3 and hence x3*= 1/3

33. Example This solution gives the maximum volume of the box as:
To find whether the solution found corresponds to a maximum or minimum, we apply the sufficiency conditions to f (x1,x2) of the equation f (x1,x2)=8x1x2(1-x12-x22)1/2. The second order partial derivatives of f at (x1*,x2*) are given by:

34. Example
The second order partial derivatives of f at (x1*,x2*) are given by:

35. Example
Since
the Hessian matrix of f is negative definite at (x1*,x2*). Hence the point (x1*,x2*) corresponds to the maximum of f.