Download presentation

Presentation is loading. Please wait.

Published byAntoine Jersey Modified over 3 years ago

1
C&O 355 Lecture 15 N. Harvey TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A A A A A A A A

2
Topics Subgradient Inequality Characterizations of Convex Functions Convex Minimization over a Polyhedron (Mini)-KKT Theorem Smallest Enclosing Ball Problem

3
Subgradient Inequality Prop: Suppose f : R ! R is differentiable. Then f is convex iff f(y) ¸ f(x) + f(x)(y-x) 8 x,y 2 R Proof: ( : See Notes Section 3.2. ) : Exercise for Assignment 4. ¤

4
Convexity and Second Derivative Prop: Suppose f : R ! R is twice-differentiable. Then f is convex iff f(x) ¸ 0 8 x 2 R. Proof: See Notes Section 3.2.

5
Subgradient Inequality in R n Prop: Suppose f : R n ! R is differentiable. Then f is convex iff f(y) ¸ f(x) + r f (x) T (y-x) 8 x,y 2 R Proof: ( : Exercise for Assignment 4. ) : See Notes Section 3.2. ¤

6
Minimizing over a Convex Set Prop: Let C µ R n be a convex set. Let f : R n ! R be convex and differentiable. Then x minimizes f over C iff r f(x) T (z-x) ¸ 0 8 z 2 C. Proof: ( direction Direct from subgradient inequality. f(z) ¸ f(x) + r f(x) T (z-x) ¸ f(x) Subgradient inequality Our hypothesis

7
Minimizing over a Convex Set Prop: Let C µ R n be a convex set. Let f : R n ! R be convex and differentiable. Then x minimizes f over C iff r f(x) T (z-x) ¸ 0 8 z 2 C. Proof: ) direction Let x be a minimizer, let z 2 C and let y = z-x. Recall that r f(x) T y = f(x;y) = lim t ! 0 f(x+ty)-f(x). If limit is negative then we have f(x+ty)

8
Positive Semidefinite Matrices (again) Assume M is symmetric Old definition: M is PSD if 9 V s.t. M = V T V. New definition: M is PSD if y T My ¸ 0 8 y 2 R n. Claim: Old ) New. Proof: y T My = y T V T Vy = k Vy k 2 ¸ 0. Claim: New ) Old. Proof: Based on spectral decomposition of M.

9
Convexity and Hessian Prop: Let f: R n ! R be a C 2 -function. Let H(x) denote the Hessian of f at point x. Then f is convex iff H(x) is PSD 8 x 2 R n. Proof: ) direction Fix y 2 R n. Consider function g y ( ® ) = f(x+ ® y). Convexity of g y follows from convexity of f. Thus g y (0) ¸ 0. (convexity & 2 nd derivative) Fact: g y (0) = y T H(x) y (stated in Lecture 14) So y T H(x) y ¸ 0 8 y 2 R n ) H(x) is PSD.

10
Convexity and Hessian Prop: Let f: R n ! R be a C 2 -function. Let H(x) denote the Hessian of f at point x. Then f is convex iff H(x) is PSD 8 x 2 R n. Proof: ( direction. Fix x,y 2 R n. Define g : [0,1] ! R by g( ® ) = f(x+ ® (y-x)). For all ®, g( ® ) =(y-x) T H(x+ ® (y-x)) (y-x) ¸ 0. (The equality was stated in Lecture 14. The inequality holds since H is PSD.) So g is convex. (By Prop Convexity and Second Derivative) f((1- ® )x+ ® y) = g( ® ) · (1- ® )g(0) + ® g(1) = (1- ® )f(x) + ® f(y). ¥

11
Hessian Example Example: Let M be a symmetric n x n matrix. Let z 2 R n. Define f : R n ! R by f(x) = x T Mx - x T z. Note f(x) = § i § j M i,j x i x j - § i x i z i. Taking partial derivatives r f(x) i = 2 § j M i,j x j - z i. So r f(x) = 2Mx-z. Recall the Hessian at x is H(x) = r ( r f(x)). So H(x) = 2M.

12
Smallest Ball Problem Let {p 1,…,p n } be points in R d. Find (unique!) ball of smallest volume (not an ellipsoid!) that contains all the p i s. In other words, we want to solve: min { r : 9 y 2 R d s.t. p i 2 B(y,r) 8 i }

13
Smallest Ball Problem Let {p 1,…,p n } be points in R d. Find (unique!) ball of smallest volume (not an ellipsoid!) that contains all the p i s. In other words, we want to solve: min { r : 9 y 2 R d s.t. p i 2 B(y,r) 8 i } We will formulate this as a convex program. In fact, our convex program will be of the form min { f(x) : Ax=b, x ¸ 0 }, where f is convex. Minimizing a convex function over an (equality form) polyhedron To solve this, we will need optimality conditions for convex programs.

14
(Mini)-KKT Theorem j th column of A Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Proven by Karush in 1939 (his Masters thesis!), and by Kuhn and Tucker in 1951.

15
(Mini)-KKT Theorem Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Special Case: (Strong LP Duality) Let f(x) = -c T x. (So the convex program is max { c T x : Ax=b, x ¸ 0 }) Then x is optimal iff 9 y 2 R m s.t. 1) -c T + y T A ¸ 0, 2) For all j, if x j >0 then - c j + y T A j = 0. y is feasible for Dual LP Complementary Slackness holds

16
Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Proof: ( direction. Suppose such a y exists. Then ( r f(x) T + y T A) x = 0. (Just like complementary slackness) For any feasible z 2 R n, we have ( r f(x) T + y T A) z ¸ 0. Subtracting these, and using Ax=Az=b, we get r f(x) T (z-x) ¸ 0 8 feasible z. So x is optimal. (By earlier proposition Minimizing over a Convex Set)

17
Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Proof: ) direction. Suppose x is optimal. Let c=- r f(x). Then r f(x) T (z-x) ¸ 0 ) c T z · c T x for all feasible points z. By our earlier proposition Minimizing over a Convex Set

18
Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Proof: ) direction. Suppose x is optimal. Let c=- r f(x). Then r f(x) T (z-x) ¸ 0 ) c T z · c T x for all feasible points z. So x is optimal for the LP max { c T x : Ax=b, x ¸ 0 }. So there is an optimal solution y to dual LP min { b T y : A T y ¸ c }. So r f(x) T + y T A = -c T + y T A ¸ 0 ) (1) holds. Furthermore, x and y are both optimal so C.S. holds. ) whenever x j >0, the j th dual constraint is tight ) y T A j = c j ) (2) holds. ¥

19
Smallest Ball Problem Let P={p 1,…,p n } be points in R d. Let Q be d x n matrix s.t. Q i =p i. Let z 2 R n satisfy z i =p i T p i. Define f : R n ! R by f(x) = x T Q T Qx - x T z. Claim 1: f is convex. Consider the convex program min { f(x) : j x j = 1, x ¸ 0 }. Claim 2: This program has an optimal solution x. Claim 3: Let p * =Qx and r= -f(x). Then P ½ B(p *,r). Claim 4: B(p *,r) is the smallest ball containing P.

20
Let P={p 1,…,p n } be points in R d. Let Q be d x n matrix s.t. Q i =p i. Let z 2 R n satisfy z i =p i T p i. Define f : R n ! R by f(x) = x T Q T Qx - x T z. Claim 1: f is convex. Proof: By our earlier example, the Hessian of f at x is H(x) = 2 ¢ Q T Q. This is positive semi-definite. By our proposition Convexity and Hessian, f is convex. ¤

21
Let P={p 1,…,p n } be points in R d. Let Q be d x n matrix s.t. Q i =p i. Let z 2 R n satisfy z i =p i T p i. Define f : R n ! R by f(x) = x T Q T Qx - x T z. Consider the convex program min { f(x) : j x j = 1, x ¸ 0 }. Claim 2: This program has an optimal solution. Proof: The objective function is continuous. The feasible region is a bounded polyhedron, and hence compact. By Weierstrass Theorem, an optimal solution exists. ¤

22
Let Q be d x n matrix s.t. Q i =p i. Let z 2 R n satisfy z i =p i T p i. Define f : R n ! R by f(x) = x T Q T Qx - x T z. Let x be an optimal solution of min { f(x) : j x j =1, x ¸ 0 } Let p * = Qx and r 2 = -f(x) = j x j p j T p j - p *T p *. Claim 3: The ball B(p *,r) contains P. Proof: Note that r f(x) T = 2x T Q T Q - z T. By KKT, 9 y 2 R s.t. 2p j T p * - p j T p j + y ¸ 0 8 j. Furthermore, equality holds 8 j s.t. x j >0. So y = j x j y = j x j p j T p j - 2 j x j p j T p * = j x j p j T p j - 2p *T p *. So y + p *T p * = j x j p j T p j - p *T p * = -f(x) ) r = y + p *T p *

23
Let p * = Qx and r 2 = -f(x) = j x j p j T p j - p *T p *. Claim 3: The ball B(p *,r) contains P. Proof: Note that r f(x) = 2x T Q T Q - z. By KKT, 9 y 2 R s.t. 2p j T p * - p j T p j + y ¸ 0 8 j. Furthermore, equality holds 8 j s.t. x j >0. ) r = y + p *T p * It remains to show that B(p *,r) contains P. This holds iff k p j -p * k · r 8 j. Now k p j -p * k 2 = (p j -p * ) T (p j -p * ) = p *T p * -2p j T p * +p j T p j · y+p *T p * = r 2 8 j. ¤

24
Claim 4: B(p *,r) is the smallest ball containing P. Proof: See Textbook.

Similar presentations

OK

1 15.053 Tuesday, March 5 Duality – The art of obtaining bounds – weak and strong duality Handouts: Lecture Notes.

1 15.053 Tuesday, March 5 Duality – The art of obtaining bounds – weak and strong duality Handouts: Lecture Notes.

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on natural numbers 0 Ppt on social networking sites Ppt on forward rate agreement currency Ppt on power system analysis Ppt on merger and acquisition in india Ppt on rural and urban livelihood Ppt on leadership qualities of bill gates Ppt on bluetooth hacking softwares Ppt on google chrome web browser Ppt on different types of transport