1. Nonlinear Programming Models
In LP ... the objective function & constraints are linear and the problems are “easy” to solve.
Most real-world problems have nonlinear elements and are hard to solve.

2. General NLP Minimize f(x) s.t. gi(x) (, , =) bi, i = 1,…,m
x is the n-dimensional vector of decision variables
f(x) is the objective function
gi(x) are the constraint functions
bi are fixed known constants

3. “decreasing efficiencies”4
Example 1
Max
3x1 + 2x2 2
s.t. x1 + x2 £ 1, x1 ³ 0, x2 unrestricted
Example 2
Max e c x e c x … e c x 1 1 2 2 n n
s.t. Ax = b, x ³ 0 n
Example 3
Min å
fj(xj)
Problems with
“decreasing efficiencies”
j=1
s.t. Ax = b, x ³ 0
fj(xj)
where each fj(xj) is of the form xj
Examples 2 and 3 can be reformulated as LPs

4. NLP Graphical Solution Method
Max f(x1, x2) = x1x2
s.t. 4x1 + x2 £ 8
x1, x2 ³ 0 x2 8
f(x) = 2
f(x) = 1 2 x1
Optimal solution will lie on the line g(x) = 4x1 + x2 – 8 = 0.

5. Solution Characteristics
Gradient of f(x) = f(x1, x2)  (f/x1, f/x2)T
This gives f/x1 = x2, f/x2 = x1
and g/x1 = 4, g/x2 = 1
At optimality we have f(x1, x2) = g(x1, x2)
or x2* = 4 and x1* = 1
Solution is not a vertex of feasible region.
For this particular problem the solution is on the boundary of the feasible region.
This is not always the case.

6. f(x) x
local
min
global
max
stationary
point
Nonconvex Function
Let S  Rn be the set of feasible solutions to an NLP.
Definition: A global minimum is any x0  S such that
f(x0)  f(x)
for all feasible x not equal to x0.

7. Function with Unique Global Minimum at x = (1, –3)
What is the optimal solution if x1 ³ 0 and x2 ³ 0 ?

8. Function with Multiple Maxima and Minima
Min {f(x)= sin(x) : 0  x  5p}

9. Constrained Function with Unique Global Maximum and Unique Global Minimum

10. Convexity
Convex function: If you draw a straight line between any two points on f(x) the line will be above or on the line of f(x).
Concave function: If f(x) is convex than - f(x) is concave. d 2 f ( x )
≥ 0 for all x.
Convex for Univariate f :
Linear functions are both convex and concave.

11. Definition of Convexity
Let x1 and x2 be two points in S  Rn. A function f(x) is convex if and only if
f(lx1 + (1–l)x2) ≤ lf(x1) + (1–l)f(x2)
for all 0 < l < 1. It is strictly convex if the inequality sign ≤ is replaced with the sign <.
1-dimensional example

12. Nonconvex -- Nonconcave Function
f(x) x

13. Theoretical Result for Convex Functions
A positively weighted sum of convex functions is convex:
if fk(x) k =1,…,m are convex and 1,…,m ³ 0
then f(x) = å akfk(x) is convex. m
k=1 …
Hessian of f at x:
Example:
f(x) = 2x13 + 3x22 – 4x12x2 + 5x1-8

14. Determining Convexity
Single Dimensional Functions:
A function f(x) Î C1 is convex if and only if it is underestimated by linear extrapolation; i.e.,
f(x2) ≥ f(x1) + (df(x1)/dx)(x2 – x1) for all x1 and x2.
x x2
f(x)
A function f(x) Î C2 is convex if and only if its second derivative is nonnegative.
d2f(x)/dx2 ≥ 0 for all x
If the inequality is strict (>), the function is strictly convex.

15. Multiple Dimensional Functions
Definition: The Hessian matrix H(x) associated with f(x) is the n  n symmetric matrix of second partial derivatives of f(x) with respect to the components of x.
When f(x) is quadratic, H(x) has only constant terms; when f(x) is linear, H(x) does not exist.
Example: f(x) = 3(x1)2 + 4(x2)3 – 5x1x2 + 4x1

16. Properties of the Hessian
How can we use Hessian to determine whether or not f(x) is convex?
H(x) is positive semi-definite (PSD) if and only if xTHx ≥ 0 for all x and there exists an x  0 such that xTHx ≥ 0.
H(x) is positive definite (PD) if and only if xTHx > 0 for all x  0.
H(x) is indefinite if and only if xTHx > 0 for some x, and xTHx < 0 for some other x.

17. Multiple Dimensional Functions and Convexity
f(x) is convex if only if f(x2) ≥ f(x1) + ÑTf(x1)(x2 – x1) for all x1 and x2.
f(x) is convex (strictly convex) if its associated Hessian matrix H(x) is positive semi-definite (definite) for all x.
f(x) is concave if only if f(x2) ≤ f(x1) + ▽Tf(x1)(x2 – x1) for all x1 and x2.
f(x) is concave (strictly concave) if its associated Hessian matrix H(x) is negative semi-definite (definite) for all x.
f(x) is neither convex nor concave if its associated Hessian matrix H(x) is indefinite

18. Testing for Definiteness
Let Hessian, H =
Definition: The ith leading principal submatrix of H is the matrix formed taking the intersection of its first i rows and i columns. Let Hi be the value of the corresponding determinant:

19. Definition
The kth order principal submatrices of an n   n symmetric matrix A are the k    k matrices obtained by deleting n - k rows and the corresponding n - k columns of A (where k = 1, ... , n).
Example

21. Rules for Definiteness
H is positive definite if and only if the determinants of all the leading principal submatrices are positive; i.e., Hi > 0 for i = 1,…,n.
H is negative definite if and only if H1 < 0 and the remaining leading principal determinants alternate in sign:
H2 > 0, H3 < 0, H4 > 0, . . .
H is positive-semidefinite if and only if all principal
submatrices ( Hi ) have nonnegative determinants.
H is negative semi-definiteness if and only if
Hi  0 for i odd and Hi  0 for i even .

22. Quadratic Functions Example 1: f(x) = 3x1x2 + x12 + 3x22
so H1 = 2 and H2 = 12 – 9 = 3
Conclusion  f(x) is strictly convex because H(x) is positive definite.

23. Quadratic Functions Example 2: f(x) = 24x1x2 + 9x12 + 6x22
H1 = 18 and H2 = 576 – 576 = 0 → f is not PD
H is positive semi-definite (determinants of all principal submatrices are nonnegative) → f(x) is convex .
Note, xTHx = 18(x1 + (4/3)x2)2 ≥ 0.

24. Nonquadratic Functions
Example 3: f(x) = (x2 – x12)2 + (1 – x1)2
Thus the Hessian depends on the point under consideration:
At x = (1, 1), which is positive definite.
At x = (0, 1), which is indefinite.
Thus f(x) is not convex although it is strictly convex near (1, 1).

25. Example
Is matrix A PD or PSD or ND or NSD or Indefinite ?

26. x0 = lx1 + (1–l)x2 Î S for all l such that 0 ≤ l ≤ 1.
Convex Sets
Definition: A set S  n is convex if any point on the line segment connecting any two points x1, x2 Î S is also in S. Mathematically, this is equivalent to
x0 = lx1 + (1–l)x2 Î S for all l such that 0 ≤ l ≤ 1.  x1 x2 x1 x1   x2   x2 

27. (Nonconvex) Feasible Region
S = {(x1, x2) : (0.5x1 – 0.6)x2 ≤ 1
2(x1)2 + 3(x2)2 ≥ 27; x1, x2 ≥ 0}

28. Convex Sets and Optimization
Let S = { x Î n : gi(x) £ bi, i = 1,…,m }
Fact: If gi(x) is a convex function for each i = 1,…,m then S is a convex set.
Convex Programming Theorem: Let x  n and let f(x) be a convex function defined over a convex constraint set S. If a finite solution exists to the problem
Minimize{f(x) : x Î S}
then all local optima are global optima. If f(x) is strictly convex, the optimum is unique.

29. Note Let s = { x  n : g(x) b}.
Fact: If g (x) is a convex function, then s is a convex set.
Let S = { x  n : gi(x)  bi, i = 1,…,m }
Fact: If gi(x) is a convex function for each i = 1,…,m then S is a convex set.
Let t = { x  n : g(x) b}.
Fact: If g (x) is a concave function, then t is a convex set.
Let T = { x  n : gi(x)  bi, i = 1,…,m }
Fact: If gi(x) is a concave function for each i = 1,…,m then T is a convex set.

30. Convex Programming Min f(x1,…,xn) s.t. gi(x1,…,xn) £ bi i = 1,…,m
is a convex program if f is convex and each gi is convex.
Max f(x1,…,xn)
s.t. gi(x1,…,xn) £ bi
i = 1,…,m
x1 ³ 0,…,xn ³ 0
is a convex program if f is concave and each gi is convex.

31. Linearly Constrained Convex Function with Unique Global Maximum
Maximize f(x) = (x1 – 2)2 + (x2 – 2)2
subject to –3x1 – 2x2 ≤ –6
–x1 + x2 ≤ 3
x1 + x2 ≤ 7
2x1 – 3x2 ≤ 4

32. (Nonconvex) Optimization Problem

33. Importance of Convex Programs
Commercial optimization software cannot guarantee that a solution is globally optimal to a nonconvex program.
NLP algorithms try to find a point where the gradient of the Lagrangian function is zero – a stationary point – and complementary slackness holds.
Given L(x,m) = f(x) + m(g(x) – b)
we want
L(x,m) = 0, g(x) – b ≤ 0, m[g(x)-b] = 0, x ³ 0, m ³ 0
However, for a convex program, all local solutions are globally optima.

34. Example: Cylinder Design
We want to build a cylinder (with a top and a bottom) of maximum volume such that its surface area is no more than S units.
Max V(r,h) = pr2h
s.t. 2pr2 + 2prh = S
r ³ 0, h ³ 0 r h
There are a number of ways to approach this problem. One way is to solve the surface area constraint for h and substitute the result into the objective function.

35. Solution by Substitution
S - 2pr2
S - 2pr2 rS 
Volume = V = pr2
- pr3
h = [
] = 2 p r
2pr 2 dV S S S
1/2
1/2
= 0 
r = ( )
, h = -
r = 2( ) dr 6 p
2pr 6 p S
3/2 S
1/2 S
1/2
V = pr2h = 2p ( ) p
r = ( )
h = 2( ) 6 6 p 6 p
Is this a global optimal solution?

36. Test for Convexity rS dV(r) S d2V(r) - pr3  = - 3pr2  = -6pr V(r) =dr 2
dr2 d 2 V
£ 0 for all r ³ 0 dr 2
Thus V(r) is concave on r ³ 0 so the solution is a global maximum.

37. Advertising (with Diminishing Returns)
A company wants to advertise in two regions.
The marketing department says that if $x1 is spent in region 1, sales volume will be 6(x1)1/2.
If $x2 is spent in region 2 the sales volume will be 4(x2)1/2.
The advertising budget is $100.
Model: Max f(x) = 6(x1)1/2 + 4(x2)1/2
s.t. x1 + x2 £ 100, x1 ³ 0, x2 ³ 0
Solution: x1* = 69.2, x2* = 30.8, f(x*) = 72.1
Is this a global optimum?

38. Excel Add-in Solution

39. Portfolio Selection with Risky Assets (Markowitz)
Suppose that we may invest in (up to) n stocks.
Investors worry about (1) expected gain (2) risk.
Let mj = expected return
sjj = variance of return
We are also concerned with the covariance terms:
sij = cov (ri, rj)
If sij > 0 then returns on i and j are positively correlated.
If sij < 0 returns are negatively correlated.

40. Decision Variables: xj = # of shares of stock j purchased
R(x) = å mjxj n
j=1
Expected return of the portfolio: n
i=1 n
j=1
Variance (measure of risk):
V(x) = å å sijxixj
Example
If x1 = x2 = 1, we get
V(x) = s11x1x1 + s12x1x2 + s21x2x1 + s22x2x1
= (-2) + (-2) + 2 = 0
Thus we can construct a “risk-free” portfolio (from variance point of view) if we can find stocks “fully” negatively correlated.

41. If , then purchasing stock 2 is just like purchasing additional shares of stock 1.

42. Nonlinear optimization models …
Let pj = price of stock j,
b = our total budget
b = risk-aversion factor (b = 0 risk is not a factor)
Consider 3 different models:
1) Max f(x) = R(x) – bV(x)
s.t. å pjxj £ b, xj ³ 0, j = 1,…,n
where b ³ 0 determined by the decision maker n
j=1

43. s.t. V(x) £ a, å pjxj £ b, xj ³ 0, j = 1,…,n
Max f(x) = R(x)
s.t. V(x) £ a, å pjxj £ b, xj ³ 0, j = 1,…,n
where a ³ 0 is determined by the investor. Smaller values of a represent greater risk aversion. n
j=1
3) Min f(x) = V(x)
s.t. R(x) ³ g, å pjxj £ b, xj ³ 0, j = 1,…,n
where g ³ 0 is the desired rate of return (minimum expectation) is selected by the investor. n
j=1