1. Convex functions Lecture 7
Dr. Zvi Lotker

2. In the last lecture optimization problem in standard form
convex optimization problems
quasiconvex optimization

3. Optimization problem in standard form
minimize f0(x)
subject to
fi(x)≤0, i=1,…,m
hi(x) = 0; i = 1,…,p
xRn is the optimization variable
f0:RnR is the objective or cost function
fi:RnR, i=1,…,m, are the inequality constraint functions
hi:RnR, i=1,…,P, are the equality constraint functions
optimal value:
p*=inf{f0(x):fi(x)≤0, hj(x)=0}
p*=  if problem is infeasible (no x satisfies the constraints)
p*=- if problem is unbounded below

4. Convex optimization problem
standard form convex optimization problem
minimize f0(x)
subject to
fi(x)≤0, i=1,…,m
aTi(x) = bi; i = 1,…,p
xRn is the optimization variable
f0:RnR is the objective, is convex
fi:RnR, i=1,…,m, the inequality constraint functions are convex
The equality constraints are Affine.
A problem is quasiconvex if f0 is quasiconvex.
important property: the feasible set of a convex optimization problem is convex

5. Optimal and locally optimal points in convex problem
x is locally optimal if there is an R > 0 such that x is optimal for
Any locally optimal point of a convex problem is (globally) optimal

6. unconstrained problem
In an unconstrained convex optimization problem the minimal x* solution satisfy
f0(x*)=0

7. Example minimize f0(x) subject to Ax = b Feasibility problem
x is optimal if and only if there exists a v such that
xDom f, Ax=b, f0(x) +ATv=0
Feasibility problem
find x
subject to fi(x)≤0, i = 1,…,m
hi(x) = 0; i = 1,…,p

8. equality constraint problem
minimize f0(x) subject to Ax = b
x is optimal if and only if there exists a v such that
xDom f, Ax=b, f0(x) +ATv=0
Proof
Assume that the feasible set is nonempty.
For all y, f0(x)T(y-x)≥0
y=x+v where Av=0  f0(x)Tv≥0
If a linear function is nonnegative on a subspace
then it must be zero on the subspace and so
f0(x)Tv=0  f0(x)N(A)  f0(x)N(A)=R(AT)

9. Outline of the lecture Equivalent convex problems linear optimization
quadratic optimization
Duality
Lagrange dual function
The dual problem

10. Equivalent convex problems
two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa
some common transformations that preserve convexity:
minimize f0(x)
s.t.
fi(x)≤0, i=1,…,m
AT(x) = b
is equivalent to
minimize f0(Fz+x0)
fi(Fz+x0)≤0, i=1,…,m
where F and x0 are such that
Ax = b X=Fz+x0 For some z

11. Introducing equality constraints
The problem
minimize f0(Fz+x0)
s.t.
fi(Fz+x0)≤0, i=1,…,m
is equivalent to
minimize f0(y)
fi(yi)≤0, i=1,…,m
yi=ATi(x) + bi, i=1,…,m

12. introducing slack variables for linear inequalities
The problem
minimize f0(x)
s.t.
Ai(x) ≤ bi, i=1,…,m
is equivalent to
Ai(x)+si = bi, i=1,…,m
si≥0, i=1,…,m

13. epigraph form standard form convex problem is equivalent to
minimize (over x, t) t
S.t.
f0(x)-t ≤0
fi(x)≤0, i=1,…,m
AT(x) = b

14. Quasiconvex optimization
The function f0 is Quasiconvex
minimize f0(x)
s.t.
Ai(x) ≤ bi, i=1,…,m
with f0 : RnR quasiconvex,
f1,…,fm convex
can have locally optimal points that are not (globally) optimal

15. convex representation of sublevel sets of f0
if f0 is quasiconvex, there exists a family of functions t such that:
t(x) is convex in x for fixed t
t-sublevel set of f0 is 0-sublevel set of t(x), i.e.,
f0(x) ≤ t  t(x) ≤ 0
Example
f0(x) =p(x)/q(x),
with p convex, q concave, and p(x)≥0, q(x)>0 on domf0
can take t(x) = p(x)-tq(x)
for t≥0, t(x) is convex in x
p(x)/q(x)≤t iff t(x)≤0

16. quasiconvex optimization via convex feasibility problems
t(x)≤0, fi(x) ≤0, i=1,…,m, Ax=0
for fixed t, a convex feasibility problem in x
if feasible, we can conclude that t≥p*;
if infeasible, t≤p*

17. Bisection method for quasiconvex optimization
given l≤p*, u≥p*, tolerance  > 0.
repeat
1. t := (l + u)/2.
2. Solve the convex feasibility problem (1).
3. if (1) is feasible, u := t; else l := t.
until u - l≤
requires exactly log2((u - l)/e) iterations (where u, l are initial values)

18. Bisection method for quasiconvex optimization
given l≤p*, u≥p*, tolerance  > 0.
repeat
1. t := (l + u)/2.
2. Solve the convex feasibility problem (1).
3. if (1) is feasible, u := t; else l := t.
until u - l≤
requires exactly log2((u - l)/e) iterations (where u, l are initial values)

19. Linear program (LP) minimize cTx + d
S.t Gx≤ h
Ax = b
convex problem with affine objective and constraint functions
feasible set is a polyhedron

20. Examples diet problem: choose quantities x1,…,xn of n foods.
One unit of food j costs cj, contains amount aij of nutrient i
healthy diet requires nutrient i in quantity at least bi
to find cheapest healthy diet
minimize cTx
subject to Ax≥b, x≥0
piecewise-linear minimization
minimize maxi=1,…,m aTix+bi
equivalent to an LP

21. Assignment problem
The assignment problem is finding a maximum weight matching in a weighted bipartite graph.
Max cijxij
S.t. j=1,…,n xij=1, i=1,…,n
i=1,…,n xij=1, j=1,…,n
xij≥0

22. Minimum spanning tree
Given a connected, undirected graph, a spanning tree of that graph is a subgraph which is a tree and connects all the vertices together.
Max Ct x
S.t x(S)≤S-1 for all SV
x(E)=|V|-1
xe≥0 for all eE

23. Chebyshev center of a polyhedron
P={x:aix≤bi, i=1,…,m}
is center of largest inscribed ball
B={xc+u: ||u||≤r}
aTi x≤ bi for all xB iff
Sup{aTi(xc + u) : ||u||≤r}= aTixc + r||ai||≤b
hence, xc, r can be determined by solving the LP
maximize r
S.t. aTixc + r||ai||≤b; i=1,…,m

24. Quadratic program (QP)
minimize (1/2)xTPx + qTx + r
subject to Gx≤b
Ax = b
PSn+, so objective is convex quadratic
minimize a convex quadratic function over a polyhedron

25. Minimal surface Consider a dierentiable function f:R2R, domf = C.
The surface area of its graph is given by
A=C(1+||f(x)||22)1/2 dx=C||(f(x),1)||2
The minimal surface problem is to find the function The minimal surface problem is to nd the
function f that minimizes A subject to some constraint
for example, some given values of f on the boundary of C
We will approximate this problem by discretizing the function f.

26. Minimal surface f(x)k <fi+1,j-fi,j, fi,j+1-fi,j>
A  Adisc1/k2||<k(fi+1,j-fi,j), k(fi,j+1-fi,j),1>||2
Min Adisc
S.t
f0,j=lj
fK,j=rj

27. Linear Programming: Box constraints
Min <c,t>
Subject to li<x<ui

28. Examples least-squares Proof: minimize ||Ax-b||22
analytical solution x* = Ab
Where A=(ATA)-1AT or A=AT(AAT)-1
Proof:
||Ax-b|| 22 =xTATAx-2bTAx+bTb
 ||Ax-b|| 22 =2ATAx-2bTA=0
For example
2ATAx-2bTA=2ATA ATA-TA-1b -2bTA=2ATb -2bTA=0

29. Generalized inequality constraints
convex problem with generalized inequality constraints
minimize f0(x)
S.t
fi(x)≤ki0, i=1,…,m
Ax = b
f0:RnR convex; fi:RnRki ki-convex w.r.t. proper cone Ki
same properties as standard convex problem (convex feasible set, local optimum is global, etc.)

30. conic form problem special case with affine objective and constraints
minimize f0(x)
S.t
Fx+g≤k0, i=1,…,m
Ax = b

31. Semidefinite program (SDP)
minimize cTx
S.t
x1F1+x2F2+…+xnFn+G≤k0
Ax = b
Fi,G,KSn+
Standard and inequality form sdp
minimize Tr(CX)
Tr(AiX)=bi
X≥0
Ai,C,XSn+

32. Example Eigenvalue minimization minimize max(A(x)) equivalent SDP
Where A(x)=A0+x1A1+…+xnAn
AiSn+
equivalent SDP
minimize t
Where A0+x1A1+…+xnAn ≤t I
AiSn+, xRn, tR
follows from
max(A)≤t  A≤tI

33. Duality Lagrange dual problem weak and strong duality
geometric interpretation
optimality conditions
examples

34. Lagrangian standard form problem (not necessarily convex)
minimize f0(x)
S.t
fi(x)≤0, i=1,…,m
hi(x) = 0; i = 1,…,p
variable xRn, domain D, optimal value p*
Lagrangian: L:RnRmRpR,
with dom L=DRmRp
L(x,,)=f0(x)+ ifi(x), jhj(x)
weighted sum of objective and constraint functions
i is Lagrange multiplier associated with fi(x) ≤ 0
j is Lagrange multiplier associated with hj(x) = 0

35. Lagrange dual function
Lagrange dual function: g:RmRpR,
g(,)= infxD L(x,,)
= infxD f0(x)+ ifi(x)+ jhj(x)
g is concave, can be - for some ,
lower bound property:
if ≥0, then g(,)≤ p*
Proof if x’ is feasible and ≥0, then
F0(x’)≥L(x’,,)≥infxD L(x,,)=g(,)
minimizing over all feasible x’ gives p*≥g(,)

36. Least-norm solution of linear equations
minimize xTx
S.t Ax=b
dual function
Lagrangian is L(x,)=xTx+ T(Ax-b)
to minimize L over x, set gradient equal to zero:
xL=2x+ AT=0  x=-1/2 AT
plug in in L to obtain g:
g()=L(-1/2 AT, )=-1/4TAAT-bT
lower bound property: p*≥-1/4TAAT-bT

37. Standard form LP Min cTx dual function Lagrangian is
S.t Ax=b, x≥0
dual function
Lagrangian is
L(x,,)=cTx+ T(Ax-b)- Tx=
= -bT +(c+AT-)Tx
L is linear in x, hence
g(,)=infxD L(x,,)=-bT if c+AT-=0
otherwise g(,)=-

38. The dual problem maximize g (,) S.t.
≥0
Finds best lower bound on p*, obtained from Lagrange dual function
a convex optimization problem; optimal value denoted d*
, are dual feasible if ≥0, (,)dom g

39. equivalent problem g(,)=infxD -bT +(c+AT-)Tx
Min cT x
S.t Ax-b=0
x≥0
max bT 
S.t AT+c≥0
max bT 
S.t AT -+c=0
≥0

40. Karush-Kuhn-Tucker (KKT) conditions
the following four conditions are called KKT conditions (for a problem with
differentiable fi, hi):
primal constraints: ifi(x)≤0, hi(x)=0
dual constraints: i≥0
complementary slackness ifi(x)=0
gradient of Lagrangian with respect to x vanishes: f0(x)+i fi(x)+ j hj(x)=0