Download presentation

Presentation is loading. Please wait.

Published byCory Marske Modified over 2 years ago

1
COSC 6114 Prof. Andy Mirzaian

2
General Overview Introduction Fundamentals Duality Major Algorithms Open Problems 2D Linear Programming O(n log n) time by computation of feasible region O(n) time Randomized Incremental Method O(n) time Deterministic Prune-&-Search Method Smallest Enclosing Disk TOPICS

3
References: [M. de Berge et al] chapter 4 [CLRS] chapter 29 [Edelsbrunner ‘87] chapter 10 Lecture Notes 4, 7-13 Reference books on “Optimization” listed on the course web-site

4
General Overview

5
The LP Problem subject to: maximize

6
Applications The most widely used Mathematical Optimization Model. Management science (Operations Research). Engineering, technology, industry, commerce, economics. Efficient resource allocation: –Airline transportation, –Communication network – opt. transmission routing, –Factory inventory/production control, –Fund management, stock portfolio optimization. Approximation of hard optimization problems....

7
Feasible Region Example in 2D optimum basic constraints max x 1 + 8x 2 subject to: (1) x 1 3 (2) x 2 2 (3)–3x 1 + 4x 2 14 (4) 4x 1 – 3x 2 25 (5) x 1 + x 2 15 x2x2 x1x1 (1) (2) (4) (5) (3) (5) (3) optimum x 1 + 8x 2 x 1 = 46/7 x 2 = 59/7 = 74

8
Example in 3D subject to: maximize z Optimum (x,y,z)=(0,0,3) x y z

9
History of LP BC: Egypt, Babylon, India, China, Greece: [geometry & algebra] Egypt: polyhedra & pyramids. India: Sulabha suutrah (Easy Solution Procedures) [2 equations, 2 unknowns] China: Jiuzhang suanshu (9 Chapters on the Mathematical Art) [Precursor of Gauss-Jordan elimination method on linear equations] Greece: Pythagoras, Euclid, Archimedes, …Sulabha suutrah Jiuzhang suanshu PythagorasEuclidArchimedes 825 AD: Persia: Muhammad ibn-Musa Alkhawrazmi (author of 2 influential books):Muhammad ibn-Musa Alkhawrazmi “Al-Maqhaleh fi Hisab al-jabr w’almoqhabeleh” (An essay on Algebra and equations) “Kitab al-Jam’a wal-Tafreeq bil Hisab al-Hindi” (Book on Hindu Arithmetic). originated the words algebra & algorithm for solution procedures of algebraic systems. Fourier [1826], Motzkin [1933] [Fourier-Motzkin elimination method on linear inequalities] Minkowski [1896], Farkas [1902], De la Vallée Poussin [1910], von Neumann [1930’s], Kantorovich [1939], Gale [1960] [LP duality theory & precursor of Simplex] George Dantzig [1947]: Simplex algorithm. Exponential time in the worst case, but effective in practice.George Dantzig Leonid Khachiyan [1979]: Ellipsoid algorithm.Leonid Khachiyan The first weakly polynomial-time LP algorithm: poly(n,d,L). Narendra Karmarkar [1984]: Interior Point Method. Also weakly polynomial-time. IPM variations are very well studied.Narendra Karmarkar Megiddo-Dyer [1984]: Prune-&-Search method. O(n) time if the dimension is a fixed constant. Super-exponential on dimension.

10
LP in matrix form: subject to: Ax b maximize c T x Canonical Form : We assume all vectors are column vectors. Transpose (T) if you need a row vector.

11
LP feasible region F = { x d | Ax b }

12
Primal Dual LPs: Primal: maximize c T x subject to: Ax b (canonical form) Dual: minimize y T b subject to: y T A = c T y 0 (standard form) A xTxT yb cTcT n d

13
Primal-Dual LP Correspondence PRIMALDUAL max c T xmin y T b Constraint: a i T x b i a j T x b j a k T x = b k ( a i T = i-th row of constraint matrix A) Variable: y i 0 y j 0 y k unrestricted Variable: x i 0 x j 0 x k unrestricted Constraint: y T A i c i y T A j c j y T A k = c k ( A i = i-th column of constraint matrix A) FACT: Dual of the Dual is the Primal.

14
Example 1: Primal-Dual PRIMAL: max 16 x 1 23 x x x 4 subject to: 3 x x 2 9 x x 4 x x x 3 14 x 4 = x x x x 4 -364 x 1 0, x 2 0, x 4 0 DUAL: min 239 y y 2 364 y 3 subject to: 3 y 1 9 y y 3 16 6 y y y 3 y y y 3 = 43 4 y 1 14 y y 3 82 y 1 0, y 3 0

15
Example 2: The Diet Problem This is one of the earliest Primal-Dual LP formulations. Dantzig attributes it to: [G. J. Stigler, “The cost of subsistence,” J. Farm Econ. 27, no. 2, May 1945, ]. Primal: A homemaker has n choices of food to buy, and each food has some of each of m nutrients: Given:a ij = amount of the ith nutrient in a unit of the jth food, i=1..m, j=1..n, r i = yearly minimum requirement of the ith nutrient, i=1..m, c j = cost per unit of the jth food, j=1..n. Determine: x j = yearly consumption of the jth food, j=1..n. Objective: Minimize total food purchasing cost c T x for the homemaker. Constraints: Meet yearly nutritional requirement (Ax r) with nonnegative amount of each food (x 0). Primal: min c T x subject to: Ax r, x 0.

16
Example 2: The Diet Problem Primal:min c T xsubject to: Ax r, x 0. Dual:max r T ysubject to: A T y c, y 0. Dual interpretation: A pill-maker wishes to market pills containing each of the m nutrients, y i = the price per unit of nutrient i, i=1..m. He wishes to be competitive with the price of real food, and maximize his profit from the sale of pills for an adequate diet. The dual constraints express the fact that the cost in pill form of all the nutrients in the j th food is no greater than the cost of the j th food itself. The objective function r T y is simply the pill maker’s profit for an adequate diet.

17
Example 3: Network Max Flow Flow Network: N=(V,E,s,t,c) V = vertex set E = directed edge set s = source vertex t = terminal vertex c: E + edge capacities find max flow f: E + satisfying flow conservation & capacity constraints see also AAW maximize v Maximize Net flow out of source subject to: v for x=s y f xy y f yx = -v for x=t Flow Conservation Law 0 for x V {s,t} f e c e for e E Flow Capacity Constraints f e 0 for e E Flow Non-negativity Constraints Assume flow is 0 on non-edges.

18
Example 3: Network Max Flow e i j i j e fefe cece A = A: vertex-edge incidence matrix Rows indexed by vertices V, columns by edges E. For edge e=(i,j) E : A ie = +1, A je = -1, A ke = 0 k V-{i,j}.

19
Example 3: Network Max Flow e i j fefe cece e ii jj ee cece

20
LP: Fundamental Facts

21
LP optimal solution set Feasible region F = { x d | Ax b } is the intersection of n half-spaces. F is a convex polyhedron (possibly empty or unbounded). Convexity: local optimality global optimality. A supporting hyper-plane is one that does not intersect the interior of F. A face of F is the intersection of F with some supporting hyper-plane. We consider F itself as a (full dimensional) face of F. is also a face of F. Vertex 0 dimensional face. Edge (bounded) 1 dimensional face. optimum objective value z* = max { c T x : x F} (may not exist or be unbounded) supporting hyper-plane c T x = z* optimum face F* = { x : c T x = z*, x F} F* interior(F) = F is pointed if it doesn’t contain any line. Hence, F= or has a vertex (i.e., rank(A)=d). F is pointed every face of F is pointed. F* & pointed F* contains a vertex of F (an optimal vertex). [In that case, we can focus on searching for an optimal vertex.] For further details see the upcoming slides.

22
Separation Lemma The Hahn-Banach Separation Theorem: Consider any non-empty closed convex set P and a point b in n. b P there is a hyper-plane that separates b from P, i.e., hyper-plane y T x = for some y n and , such that y T b < and y T z z P. y T x = P b Proof: Consider the closest point p P to point b. Take the hyper-plane through p and orthogonal to segment bp. p

23
Extreme Points FACT: A non-empty closed convex set P has an extreme point if and only if P does not contain any line. Proof ( ): Suppose P contains a line L. We must prove that no point x P is an extreme point of P. Let L’ be the line parallel to L that passes through x. CLAIM: P contains L’. This claim implies x is not an extreme point of P. (Why?) Refutation of the Claim leads to a contradiction as follows: Suppose there is a point b L’ – P. By the Separation Lemma, there is a hyperplane H that separates b from P. H intersects L’ (at some point between x and b). H must also intersect line L (which is parallel to L’). That would contradict the fact that H separates b from P L. This contradiction proves the Claim. L x L’ b H

24
Extreme Points FACT: A non-empty closed convex set P has an extreme point if and only if P does not contain any line. Proof ( ): Suppose P does not contain any line. We apply induction on the dimension. If P is a single point, then it is an extreme point. Now suppose dim(P) > 0. Since P does not contain any line, P is a proper subset of aff(P) (affine hull of P). So, there is a point b aff(P) – P. By the Separation Lemma, there is a hyperplane H separating b from P in aff(P). Move H parallelwise toward P until H is tangent to P. Let f = H P. dim(f) dim(H) < dim(P). f is a non-empty closed convex set that contains no line. So, by the induction hypothesis, f must have an extreme point x. x is an extreme point of P also. (Why?) H P f

25
Extreme Points The Krein-Milman Theorem: A closed bounded convex set is equal to the convex hull of its extreme points. Proof: By convexity & closedness, the given set P contains the convex hull of its extreme points. The converse, i.e., every point x P can be represented as a convex combination of extreme points of P, is proved by induction on the dimension. If x is an extreme point of P, we are done. Otherwise, draw a line L through x. Let yz be the maximal segment of L in P. x is a convex combination of the boundary points y and z (as shown). By induction, y & z can be represented as convex combinations of extreme points of the lower dimensional closed bounded convex sets P H 1 and P H 2, respectively. Hence, x is a convex combination of extreme points of P (details for exercise). H1H1 H2H2 y x z P L

26
Farkas Lemma Farkas Lemma: Let A n d, b n. Then exactly one of the following must hold: (1) x d : Ax b. (2) y n : y T A = 0, y T b < 0, y 0. Proof: [(1) & (2) cannot both hold] Else: 0 = 0x = (y T A)x = y T (Ax) y T b < 0. [ (1) (2)]: P = { z n | z=Ax+s, x d, s n, s 0 } is convex & non-empty(0 P). (1) bPbP y, : y T b < y T z z P by Separation Lemma y, : y T b < y T (Ax+s) s 0, x y, : y T b < (y T A)x+ y T s s 0, x y, : y T b < y T s s 0, & y T A = 0 if y T A 0, take x=( -1- y T s) A T y ||y T A|| -2 y, : y T b < y T s s 0, & y T A = 0, & y 0 if y i <0, take s i = - (1+| |) / y i & s j =0 for j i y, : y T b 0, take s = ( /2)y ||y|| -2 y: y T b < 0, y T A = 0, y 0 (2)

27
Farkas Lemma Farkas Lemma (alternative versions): Let A n d, b n. 1) x d : Ax b y n : y T A = 0, y T b < 0, y 0. 2) x d : Ax b, x 0 y n : y T A 0, y T b < 0, y 0. 3) x d : Ax = b y n : y T A = 0, y T b < 0. 4) x d : Ax = b, x 0 y n : y T A 0, y T b < 0.

28
Convex Cones, Sets & Polyhedra cone(R) = { 1 r 1 + 2 r 2 + … + k r k | i 0 } cone(R)={0} if R= . Extreme rays of the cone are shown bold. 0 Minkowski sum (of vector sets): P+Q = {x+y | x P, y Q} CH(V) = { 1 v v 2 + … + m v m | i i =1, i 0 } Recession Cone of the polyhedron Convex Cone: C ( d ), 0 C, C + C C, and 0: C C. Finitely Generated Convex Cones and Convex Sets: Let V = {v 1, …,v m } and R = {r 1, …,r k } += CH(V) + cone(R): polyhedron

29
Caratheodory’s Theorem Let X be a non-empty set in d. (a)x cone(X) & x 0 if and only if x= 1 x 1 + 2 x 2 + … + k x k for some 1, …, k > 0 and for some linearly independent x 1, …,x k X. (Hence, k d.) (b) x CH(X) if and only if x= 1 x 1 + 2 x 2 + … + k x k for some 1, …, k > 0, i i =1. and for some affinely independent x 1, …,x k X. (I.e., x 2 -x 1, …,x k -x 1 are linearly independent. Hence, k d+1.) 0 x cone(X) (a) CH(X) x (b) X x1x1 x2x2 x1x1 x2x2 x3x3

30
Caratheodory’s Theorem - Proof (a) Consider a minimal k such that x = 1 x 1 + 2 x 2 + … + k x k, i >0, x i X, i=1..k. (x i ) are linearly dependent 1, …, k : at least one i >0 & 0 = 1 x 1 + 2 x 2 + … + k x k. So, x = ( 1 – 1 ) x 1 + ( 2 – 2 ) x 2 + … + ( k – k ) x k. Take = min { i / i | i >0}. All coefficients remain non-negative and at least one becomes 0. This contradicts minimality of k. CH(X) cone(Y) Y x (b) Apply proof of (a) in d+1 to cone(Y) where Y = CH(X) {1} = { (x,1) d+1 | x CH(X) }.

31
Polyhedra - Theorem Fundamental Theorem of Polyhedra [Minkowski-Weyl]: A subset P d is the intersection of a finite set of half-spaces: P = { x d | Ax b } for some A n d, b n if and only if P is the sum of convex hull of a finite set of points plus conical combination of a finite set of vectors: P = CH(V) + cone(R) for some V d m, R d m’ = { x d | x = V + R , 0, 0, i i = 1 }. P is a polytope P is bounded R= , V = vertex set of P. Corollary: A polytope can be viewed in 2 equivalent ways: 1) as intersection of half spaces (defined by its facets), 2) as convex hull of its vertices (cf. Krein-Milman Theorem). If P is pointed, columns of V are vertices of P, columns of R are extreme rays of P.

32
Polyhedra – Example V = (v 1, v 2 ) R = ( r 1, r 2 v2v2 v1v1 r2r2 r1r1 P x1x1 x2x2 RV

33
Polyhedra – Proof Proof of ( ): [Assume polyhedron P is pointed.] Let Q = { x d | x = V + R , 0, 0, i i = 1 } where columns v i of V are extreme points of P, and columns r i of R are extreme rays of P. We claim Q = P: Q P: x Q implies x =y + z for some y=V 0, i i = 1, and z= R , 0. So y P, since y is a convex combination of extreme points of P. Also, z is a conic combination of extreme rays of P, so y+z P. So x P. Q P: WLOG assume P . If x Q, then ( ) m m’ such that v v …+ v m m + r 1 1 + r 2 2 +… + r m’ m’ x + + …+ m 0, 0. So, by Farkas Lemma ( , o ) d such that i: T v i o 0, T r i 0, T x o 0. Now consider the LP max { T y | y P}. For all rays r i of P, T r i 0. Hence, the LP has a bounded optimum. So, the optimum must be achieved at an extreme point of P. But for every extreme point v i of P we have T x > - o T v i. So, T x > - o max { T y | y P}. (The hyperplane T y o separates x from P.) Therefore, x P. [What happens if P is not pointed? See exercise 7.]

34
Polyhedra – Proof Proof of ( ): Q is the projection onto x of the polyhedron { (x, ) d m m’ | x - V – R , 0, 0, i i = 1 }. Hence, Q is also a polyhedron (i.e, intersection of a finite number of half-spaces).

35
Bases & Basic Solutions Bases & Basic Solutions: Basis B: Any set of d linearly independent rows of A. B d d and det(B) 0. N = remaining non-basic rows of A. Basic Solution: Bx B = b B x B = B -1 b B. Basic Feasible Solution (BFS): If x B is feasible, i.e., Nx B b N. Each BFS corresponds to a vertex of the feasible polyhedron. xTBxTB bBbB bNbN B N Ab

36
Fundamental Theorem of LP For any instance of LP exactly one of the following three possibilities holds: (a) Infeasible. (b) Feasible but no bounded optimum. (c) Bounded optimum. [Note: Feasible polyhedron could be unbounded even if optimum is bounded. It depends on the direction of the objective vector.] Moreover, if A has full rank (i.e., basis), then every nonempty face of the feasible polyhedron contains a BFS, and this implies: (1) feasible solution BFS. (2) optimum solution optimum solution that is a BFS. c Proof: If there is a basis, the basic cone contains the feasible region but does not contain any line. So the feasible region does not contain any line, hence it is pointed. So every non-empty face of it (including the optimal face, if non-empty) is pointed, and thus contains a vertex. (For details see exercise 4.)

37
LP Duality Theorem Primal: max { c T x | Ax b } Dual: min { y T b | y T A = c T, y 0 } Weak Duality: x Primal feasible & y Dual feasible c T x y T b. Proof: c T x = (y T A)x = y T (Ax) y T b. Corollary: (1) Primal unbounded optimum Dual infeasible (2) Dual unbounded optimum Primal infeasible (3) Primal & Dual both feasible both have bounded optima. [Note: Primal & Dual may both be infeasible. See Exercise 5.] Strong Duality: If x & y are Primal/Dual feasible solutions, then x & y are Primal/Dual optima c T x = y T b. Proof: By Farkas Lemma. See Exercise 8.

38
Example 1 P: max -18 x 1 27 x 2 subject to: -4 x x 2 8 x 1 + x 2 7 x 1 0, x 2 0 D:min 8 y y 2 subject to: -4 y 1 + y 2 y 1 + y 2 27 y 1 0, y 2 0 Optimum Solution for P: x 1 = 3, x 2 = 4 opt objective value = -18*3 + 27*4 = 54 Optimum Solution for D: y 1 = 5, y 2 = 2 opt objective value = 8*5 + 7*2 = 54

39
Example 2: The Diet Problem Primal:min c T xsubject to: Ax r, x 0.[Homemaker] Dual:max r T ysubject to: A T y c, y 0.[Pill Maker] Week Duality: For every primal feasible x and dual feasible y: c T x r T y. Interpretation : the pill maker cannot beat the real food market for an adequate diet. Strong Duality: min {c T x | Ax r, x 0} = max { r T y | A T y c, y 0}. Interpretation: homemaker’s minimum cost food budget is equal to pill maker’s maximum profit for an adequate diet.

40
LP Optimization Feasibility Primal: maximize c T x subject to: Ax b Dual: minimize y T b subject to: y T A = c T y 0 Is there (x,y) that satisfies: b T y - c T x 0 Ax b A T y = c y 0

41
Complementary Slackness Primal: maximize c T x subject to: a T i x b i, i=1..n x j 0, j=1..d Dual: minimize y T b subject to: A T j y c j, j=1..d y i 0, i=1..n Complementary Slackness Condition (CSC): y i (b i - a T i x ) = 0, i=1..n x j (A T j y - c j ) = 0, j=1..d FACT: Suppose x & y are Primal-Dual feasible. Then the following statements are equivalent: (a)x and y are Primal-Dual optimal, (b)c T x = y T b, (c)x & y satisfy the CSC.

42
Example 3: Max-Flow Min-Cut See AAW for animation: S ( v net out-flow) t f = c, =1 f = 0, = 0 = 0 = 0 = 1

43
Min-Max Saddle-Point Property FACT: For all functions f : X Y : max min f (x,y) min max f (x,y). x X y Y y Y x X Proof: Suppose LHS = f(x 1,y 1 ) and RHS = f(x 2,y 2 ). Then, f(x 1,y 1 ) ≤ f(x 1,y 2 ) ≤ f(x 2,y 2 ). x1x1 x2x2 y1y1 y2y2

44
Min-Max Saddle-Point Property FACT: f has the SPP if and only if max min f (x,y) = min max f (x,y). x X y Y y Y x X Function f has the Saddle-Point Property (SPP) if it has a saddle-point, i.e., a point (x 0, y 0 ) X Y such that x X f (x, y 0 ) ≤ f (x 0, y 0 ) ≤ f(x 0, y) y Y.

45
Duality via Lagrangian P* = max {c T x | Ax b } x D* = min { y T b | A T y = c } y 0 Lagrangian: L(x,y) = c T x + y T (b – Ax) = y T b + x T (c – A T y) Fact: P* = max min L(x,y), D* = min max L(x,y). x y 0 y 0 x Proof: See Exercise 9. Weak Duality : max min L(x,y) min max L(x,y) x y 0 y 0 x P* D*. Strong Duality : P or D feasible Lagrangian has SPP. max min L(x,y) = min max L(x,y) x y 0 y 0 x P* = D*.

46
George Dantzig [1947]: Simplex algorithm. Exponential time in the worst case, but effective in practice.George Dantzig Leonid Khachiyan [1979]: Ellipsoid algorithm.Leonid Khachiyan The first weakly polynomial-time LP algorithm: poly(n,d,L). Narendra Karmarkar [1984]: Interior Point Method. Also weakly polynomial-time. IPM variations are very well studied.Narendra Karmarkar Dantzig – Khachiyan – Karmarkar

47
Dantzig: Simplex Algorithm Pivots along the edges of the feasible polyhedron, monotonically improving the objective value. c Pivot Rule: Decides which row leaves & which row enters the basis (i.e., which edge of the feasible polyhedron to follow next). Each BFS corresponds to a vertex of the feasible polyhedron. Simplex Pivot: A basic row leaves the basis and a suitable row enters the basis. A non-degenerate pivot moves the BFS along an edge of the feasible polyhedron from one vertex to an adjacent one.

48
Center of Gravity A set of points K d is called a convex body if it is a full dimensional, closed, bounded, convex set. FACT: [Grünbaum 1960] Let point c be the center of gravity of a convex body K and H be an arbitrary hyperplane passing through point c. Suppose H divides K into two bodies K + and K –. Then the volumes vol(K + ) and vol(K – ) satisfy: c K+K+ K–K– H

49
Iterative Central Bisection FACT: Suppose the feasible set is a convex body F which is contained in a ball C 0 of radius R and contains a ball C 1 of radius r. Suppose we start with (the center of) the larger ball and iteratively perform central bisection until center of gravity of the remaining convex body falls within F. The number of such iterations is O(d log R/r). Proof: vol(C 0 ) = v d R d vol(C 1 ) = v d r d v d = volume of the unit ball in d. With each iteration, the volume drops by a constant factor. # iterations = O( log vol(C 0 )/vol(C 1 ) ). r R F C0C0 C1C1

50
Iterative Central Bisection The # central bisection iterations O(d log R/r) is polynomial in the bit-size of the LP instance. However, each iteration involves computing the volume or center of gravity of the resulting convex body. This task seems to be hard even for LP instances. We don’t know how to compute them in polynomial time. So, we resort to approximation 1)approximate the center of gravity 2)approximate the convex body What simple approximating “shape” to use? Bounding Box, Ball, ellipsoid, …?

51
Ellipsoids z o = center of ellipsoid E. A = a d d non-singular matrix. An ellipsoid is an affine transform of the Euclidean unit ball. zozo E x y 0

52
Ellipsoids Example:

53
Ellipsoids ( v d = volume of the unit ball in d.) zozo E c x opt

54
Löwner-John Ellipsoid FACT 1: [K. Löwner 1934, Fritz John, 1948] Let K be a convex body in d. Then 1.There is a unique maximum volume ellipsoid E in (K) contained in K. 2.K is contained in the ellipsoid that results from E in (K) by centrally expanding E in (K) by a factor of d (independent of K). 3.If K is centrally symmetric, then the factor d in part (2) can be improved to d.

55
Löwner-John Ellipsoid FACT 2: [K. Löwner 1934, Fritz John, 1948] Let K be a convex body in d. Then 1.There is a unique minimum volume ellipsoid E out (K) that contains K. 2.K contains the ellipsoid that results from E out (K) by centrally shrinking E out (K) by a factor of d (independent of K). 3.If K is centrally symmetric, then the factor d in part (2) can be improved to d.

56
Löwner-John metric approximation Corollary: Any metric in d can be approximated by a Löwner-John metric with distortion factor at most d. Proof: Let ||. || B denote an arbitrary metric norm with (centrally symmetric) unit ball B. || x || B = min { | x B } for any x d. Let E LJ = { x d | || A –1 x || 2 1 } be the min volume Löwner-John ellipsoid that contains B. The metric norm with E LJ as the unit ball is || x || LJ = || A –1 x || 2 for any x d. E LJ B

57
c x k-1 x opt zozo Iterative Löwner-John Ellipsoidal Cut xkxk x out

58
Iterative Löwner-John Ellipsoidal Cut # iterations is polynomial in the bit-size of the LP instance. However, each iteration is expensive and involves computing Löwner-John Ellipsoid of a convex polytope. What next? Approximate the convex body by an ellipsoid.

59
Khachiyan: Ellipsoid Method Circumscribed ellipsoid volume reduction by cuttings. EkEk zkzk E k+1 z k+1 # iterations = O(d 2 log R/r) Time per iteration = O(d n)

60
The New York Times, November 7, 1979 Continued on next slide.

61

62
Karmarkar: Interior Point Method maximize c T x + i log s i subject to Ax+s=b, s > 0 == =1 =0.01 == =1 =0.01 Central path Logarithmic Barrier and Central Path

63
IPM: Primal-Dual Central Path (P) max c T x (D) min b T y s.t. Ax+s=b, s 0 s.t. A T y=c, y 0 Optimality Conditions on (x,s,y): Ax+s=b, s 0 (Primal feasibility) A T y=c, y 0 (Dual feasibility) s i y i = 0 i (Complementary Slackness) Primal-Dual Central Path parameterized by > 0: Ax+s=b, s > 0 (Primal strict feasibility) A T y=c, y > 0 (Dual strict feasibility) s i y i = i (Centrality)

64
IPM: Primal-Dual Central Path Algorithmic Framework: Assume both (P) and (D) have strictly feasible solutions. Then, for any > 0, the above non-linear system of equations has a unique solution (x( ), s( ), y( )). These points form the Primal-Dual Central Path { (x( ), s( ), y( )) | (0, ) }. lim 0 (x( ), s( ), y( )) = (x*, s*, y*) is optimal solution of the Primal-Dual LP. Update approximate solution to the system while iteratively reducing towards 0. Central Path: Ax+s=b, s > 0 A T y=c, y > 0 s i y i = i

65
Open Problem Can LP be solved in strongly polynomial time? Strongly Polynomial: (i)Time = poly(n,d) [i.e., independent of input coefficient sizes] (ii)Bit-length of each program variable = poly(n,d,L). Weakly Polynomial: (i)Time = poly(n,d,L), [L = bit-length of input coefficients] (ii)Bit-length of each program variable = poly(n,d,L). [ Steve Smale, “Mathematical Problems for the Next Century’’, Mathematical Intelligencer, 20:7–15, 1998] lists 18 famous open problems and invites the scientific community to solve them in the 21 st century. Problem 9 on that list is the above open problem on LP.Steve SmaleMathematical Problems for the Next Century

66
Best known upper-bound [Kalai, Kleitman 1992]: (d,n) n log d + 2 For a recent survey on this topic click here.here Hirsch’s Conjecture [1957] Hirsch’s Conjecture: Diameter of any d-polyhedron with n facets satisfies (d,n) n – d. That is, any pair of its vertices are connected by a path of at most n d edges. This conjecture was disproved by Francisco Santos in June Click here to see Santos’s paper.here

67
LP in dimension 2

68
2D Linear Programming F y x Objective function = c 1 x + c 2 y, c T =(c 1, c 2 ) Feasible region F c optimum max = c 1 x + c 2 y

69
2D Linear Programming Feasible region F is the intersection of n half-planes. F is (empty, bounded or unbounded) convex polygon with n vertices. F can be computed in O(n log n) time by divide-&-conquer (See Lecture-Slide 3). If F is empty, then LP is infeasible. Otherwise, we can check its vertices, and its possibly up to 2 unbounded edges, to determine the optimum. The latter step can be done by binary search in O(log n) time. If objective changes but constraints do not, we can update the optimum in only O(log n) time. (We don’t need to start from scratch). Improvement Next: Feasible region need not be computed to find the optimum vertex. Optimum can be found in O(n) time both randomized & deterministic.

70
2D LP Example: Manufacturing with Molds

71
The Geometry of Casting: Is there a mold for an n-faceted 3D polytope P such that P can be removed from the mold by translation? Lemma: P can be removed from its mold with a single translation in direction d d makes an angle 90 with the outward normal of all non-top facets of P. mold P f f ’ (f ‘) = - (f) d Corollary: Many small translations possible Single translation possible.

72
2D LP Example: Manufacturing with Molds The Geometry of Casting: Is there a mold for an n-faceted 3D polytope P such that P can be removed from the mold by translation? mold P f f ’ (f ‘) = - (f) d (f) = ( x (f), y (f), z (f) ) outward normal to facet f of P. d T. (f) ≤ 0 non-top facet f of P x (f). x + y (f). y + z (f) ≤ 0 f n-1 constraints x y z z=1 d=(x,y,1) THEOREM: The mold casting problem can be solved in O(n log n) time. (This will be improved to O(n) time on the next slides.)

73
Randomized Incremental Algorithm

74
Randomization Random(k): Returns an integer i 1..k, each with equal probability 1/k. [Use a random number generator.] Algorithm RandomPermute (A) O(n) time Input: Array A[1..n] Output: A random permutation of A[1..n] with each of n! possible permutations equally likely. for k n downto 2 do Swap A[k] with A[Random(k)] end. This is a basic “initial” part of many randomized incremental algorithms.

75
2D LP: Incremental Algorithm Method: Add constraints one-by-one, while maintaining the current optimum vertex. Input: (H, c), H = { H(1), H(2), …, H(n)} n half-planes, c = objective vector Output: Infeasible: (i,j,k), or Unbounded: , or Optimum: v = argmax x { c T x | x H(1) H(2) … H(n) }. Define: C(i) = H(1) H(2) … H(i), for i = 1..n v(i) = optimum vertex of C(i), for i=2..n. c T v(i) = max { c T x | x C(i) } Note: C(1) C(2) … C(n). e v InfeasibleUnboundedNon-unique optimumUnique optimum c Possible Outcomes: H(i) H(j) H(k)

76
2D LP: Incremental Algorithm LEMMA: (1) v(i-1) H(i) v(i-1) C(i) v(i) v(i-1). (2) v(i-1) H(i) (2a) C(i) = , or (2b) v(i) L(i) C(i-1), L(i) =bounding-line of H(i). H(i) v(i)=v(i-1) (1) C(i-1) H(i) (2a) C(i-1) v(i-1) H(j) H(k) H(i) (2b) C(i-1) v(i-1) v(i) L(i) c

77
2D LP: Incremental Algorithm Algorithm PreProcess (H,c) O(n) time 1. min { angle between c and outward normal of H(i) | i=1..n } H(i) = most restrictive constraint with angle Swap H(i) with H(1) (L(1) is bounding-line of H(1)). 2. If (parallel) H(j) with angle and H(1) H(j)= then return “infeasible”. 3. If L(1) H(j) is unbounded for all H(j) H, then most restrictive L(1) H(j) over all H(j) H return (“unbounded”, ) 4. If L(1) H(j) is bounded for some H(j) H, then Swap H(j) with H(2) and return “bounded”. c L(1) H(1) L(1) H(j) H(2) H(1) (1)(3) (4) L(1) H(1) (2) H(j) Disjoint parallel. v(2)

78
Input: (H, c), H = { H(1), H(2), …, H(n)} n half-planes, c = objective vector Output: Solution to max { c T x | x H(1) H(2) … H(n) } 1. if PreProcess(H,c) returns (“unbounded”, ) or “infeasible” then return the same answer (* else bounded or infeasible *) 2. v(2) vertex of H(1) H(2) 3. RandomPermute (H[3..n]) 4. for i 3..n do 5. if v(i-1) H(i) then v(i) v(i-1) 6. else v(i) optimum vertex p of L(i) (H(1) … H(i-1)) (* 1D LP *) 7. if p does not exist then return “infeasible” 8. end-for 9. return (“optimum”, v(n)) end. Randomized Incremental 2D LP Algorithm

79
THEOREM: 2D LP Randomized Incremental algorithm has the following complexity: Space complexity = O(n) Time Complexity: (a) O(n 2 ) worst-case (b) O(n) expected-case. Randomized Incremental 2D LP Algorithm Proof of (a): Line 6 is a 1D LP with i-1constraints and takes O(i) time. Total time over for-loop of lines 4-8:

80
Proof of (b): Define 0/1 random variables Lines 5-7 take O( i*X(i) +1) time. Total time is Randomized Incremental 2D LP Algorithm v(i) is defined by 2 H(j)’s. The probability that one of them is H(i) is 2/(i-2). This does not depend on C(i). Hence, remove the “Fix” assumption. Expected time: Backwards Analysis linearity of expectation “Fix” C(i) = { H(1), H(2)} {H(3), …, H(i)} Random : C(i-1) = C(i) – {H(i)} random

81
Randomized Incremental LP in d dimension Randomized Incremental algorithm can be used (with minor modifications) in d dimensions, recursively. For example, for 3D LP line 6 becomes a 2D LP with i-1 constraints. In general, it will be a (d-1) dimensional LP. Define: T(n,d) = running time of algorithm for n constraints & d variables. This would be exponential in both n and d. Expected time (by backwards analysis):

82
Randomized Incremental LP in d dimension [For n d, T(n,d) = O(dn 2 ), by Gaussian elimination.]

83
Randomized Incremental Algorithm for Smallest Enclosing Disk

84
Smallest Enclosing Disk p i+1 Lemma: (1) p i D(i-1) D(i) = D(i-1) (2) p i D(i-1) p i lies on the boundary of D(i). pipi D(i+1) D(i)=D(i-1) Input: A set P={p 1, p 2, …, p n } of n points in the plane. Output: Smallest enclosing disk D of P. Lemma: Output is unique. Incremental Construction: P[1..i] = {p 1, p 2, …, p i } D(i)= smallest enclosing disk of P[1..i].

85
Smallest Enclosing Disk LEMMA: Let P and R be disjoint point sets in the plane. p P, R possibly empty. Define MD(P, R) = minimum disk D such that P D & R D ( D = boundary of D). (1)If MD(P, R) exists, then it’s unique, (2)p MD(P-{p}, R) MD(P,R) = MD(P-{p}, R), (3)p MD(P-{p}, R) MD(P, R) = MD(P-{p}, R {p}). D(0) D(1) R p D( ) (3)D(0) MD(P-{p}, R) D(1) MD(P, R) D( ) (1- ) D(0) + D(1) 0 1 As goes from 0 to 1, D( ) continuously deforms from D(0) to D(1) s.t. D(0) D(1) D( ). p D(1) – D(0) by continuity smallest *, 0 < * 1 s.t. p D( *) p D( *). P D( *) & R D( *) *=1 by uniqueness. Therefore, p is on the boundary of D(1). Proof: (1) If non-unique smaller such disk: (2) is obvious.

86
Smallest Enclosing Disk Algorithm MinDisk (P[1..n]) 1. RandomPermute(P[1..n]) 2. D(2) smallest enclosing disk of P[1..2] 3. for i 3..n do 4. if p i D(i-1) then D(i) D(i-1) 5. else D(i) MinDiskWithPoint (P[1..i-1], p i ) 6. return D(n) Procedure MinDiskWithPoint (P[1..j],q) 1. RandomPermute(P[1..j]) 2. D(1) smallest enclosing disk of p 1 and q 3. for i 2..j do 4. if p i D(i-1) then D(i) D(i-1) 5. else D(i) MinDiskWith2Points (P[1..i-1], q, p i ) 6. return D(j) Procedure MinDiskWith2Points (P[1..j],q 1,q 2 ) 1. D(0) smallest enclosing disk of q 1 and q 2 2. for i 1..j do 3. if p i D(i-1) then D(i) D(i-1) 4. else D(i) Disk (q 1, q 2, p i ) 5. return D(j) Show random permutation only once is enough

87
Smallest Enclosing Disk Proof: Space O(n) is obvious. MinDiskWith2Points (P,q 1,q 2 ) takes O(n) time. MinDiskWithPoint (P,q) takes time: where Apply this idea once more: expected running time of MinDisk is also O(n). P[1..i] q “Fix” P[1..i] = {p 1, …, p i } backwards P[1..i-1] = {p 1, …, p i } - {p i } one of these is p i with prob. 2/i THEOREM: The smallest enclosing disk of n points in the plane can be computed in randomized O(n) expected time and O(n) space.

88
2D LP Prune-&-Search

89
2D LP: Megiddo(83), Dyer(84): Prune-&-Search STEP 1: Transform so that the objective hyper-plane becomes horizontal. (a,b) (0,0), so WLOG b 0. Coordinate transform: Y = ax + by i = a i – (a/b) b i X = x i = b i /b min { Y | i X + i Y + c i 0, i=1..n } X Y F feasible region optimum Original LP: min { ax + by | a i x + b i y + c i 0, i=1..n }

90
2D LP: Prune-&-Search STEP 2: Partition the constraints (the index set [1..n]) I 0 = { i | i = 0 } vertical line constraints determine feasible x-interval I - = { i | i < 0 } I + = { i | i > 0 } STEP 3: X Y F U1U1 U2U2 STEP 4: min { Y | i X + i Y + c i 0, i=1..n }

91
2D LP: Prune-&-Search STEP 5: F+F+ F-F- So, the transformed constraints are: F - (X) Y F + (X). Since we want to minimize Y, our objective function is F - (X). STEP 6: Our new (transformed) problem is: X Y F U1U1 U2U2 optimum F+F+ F-F-

92
2D LP: Prune-&-Search 7(a): Given X [U 1, U 2 ], we can determine in O(n) time: F - (X) = max { i X + i | i I - } F + (X) = min { i X + i | i I + } f - (L) (X) = left slope of F - (X) at X f - (R) (X) = right slope of F - (X) at X f + (L) (X) = left slope of F + (X) at X f + (R) (X) = right slope of F + (X) at X Note: if only one i I - achieves the “max” in F - (X), then f - (L) (X) = f - (R) (X) = i. Similarly with F + (X). X’X F-F- STEP 7: An Evaluation Stage:

93
2D LP: Prune-&-Search 7(b): In O(n) time we can determine for X [U 1, U 2 ], whether: X achieves the minimum of F - (X) and is feasible (hence optimum). X is infeasible & there is no solution to the LP X is infeasible and we know which side (left/right) of X the feasible region F lies. X is feasible and we know which side of X the optimum solution lies. In the latter two cases, shrink the interval [U 1, U 2 ] accordingly. F - (X) F + (X) X X X X f - (L) (X) > f + (L) (X) infeasible F to the left f - (R) (X) < f + (R) (X) F to the right f - (L) (X) f + (L) (X) & f - (R) (X) f + (R) (X) F = ; LP is infeasible OR STEP 7: An Evaluation Stage:

94
2D LP: Prune-&-Search METHOD: A series of evaluation stages. Prune at least a fixed fraction (1- ) of the constraints in each stage. These constraints won’t play a role in determining the optimum vertex. After the k-th stage, # constraints remaining k n. At most log n stages. Total time:

95
2D LP: Prune-&-Search METHOD: (continued) = 3/4 is achievable. Which X to select to evaluate then prune? Pair up constraints in I + (do the same for I - ). Suppose i,j I + are paired up. parallel eliminate U1U1 U2U2 X ij U1U1 U2U2 eliminate X ij U1U1 U2U2 eliminate neither one

96
METHOD: (continued) Next evaluation point: X* = median of the X ij ’s of the non-eliminated pairs. O(n) time. Half of the X ij ’s will be on the wrong side of X*. We will eliminate one line from at least half of the pairs. So, at least ¼ of the lines will be eliminated in this stage. At most = ¾ of the lines remain. THEOREM: A 2-variable LP with n constraints can be solved in O(n) time in the worst-case by Megiddo-Dyer’s Prune-&-Search method. GENERALIZATION: Megiddo[84] showed d dim. LP can also be solved by this technique in O(n) time assuming d is a fixed constant. [However, the hidden constant in O(n) is super-exponential in d.] 2D LP: Prune-&-Search

97
Extensions & Applications The Smallest Enclosing Disk: THEOREM [Megiddo]: The smallest enclosing disk can be solved in O(n) time in the worst-case by the Prune-&-Search method. See Lecture Notes 7-13 for more on this topic. A Quadratic Program. See Exercise 16.

98
Exercises

99
1.A Non-Linear Primal Dual Example: [Rochat, Vecten, Fauguier, Pilatte, 1811] We are given 3 points A,B,C in general position in the plane. Primal: Consider any equilateral triangle PQR whose 3 sides pass through A,B,C, respectively. (See Figure below.) Let H denote the altitude of the equilateral triangle PQR. Find PQR to maximize H. Dual: Consider any point S, with L as the sum of the lengths of the 3 segments L = SA + SB + SC. Find S to minimize L. [The optimum S is called the Steiner point of ABC.] Prove the following claims: a)For any primal-dual feasible solutions H L. b)At optimality max H = min L. c)At optimality SA QR, SB RP, SC PQ, where means perpendicular. A B CP Q R S

100
2.LP Primal Dual Example: Give the dual of the following LP instance: max 5 x x x x x 5 subject to: -2 x x x x x 5 = x x x x x 5 x x x x x 5 43 8 x x x x x 5 61 x 1 0, x 3 0, x 4 0, x 5 0 3.Fractional Knapsack Problem (FKP): FKP is the LP max { v T x | w T x W, 0 x i 1, i=1..n }, where v T =(v 1,…, v n ) and w T =(w 1,…, w n ) are vectors of item values and weights, and W is the knapsack capacity. Assume these are all positive reals. The real variable x i, 0 x i 1, is the selected fraction of the i-th item, for i=1..n. a) Give the dual of FKP. b) Use the Duality Theorem to show that a simple greedy algorithm obtains optimal primal-dual solutions. [Hint: sort v i / w i, for i=1..n.]

101
4.Polyhedral Vertices versus Basic Solutions: Consider the (possibly empty) polyhedron F = { x d | Ax b } for some A n d, b n, d n. Assume rank(A)=d (i.e., A contains d linearly independent rows). Show the following facts: a)F has a basic solution (not necessarily feasible). b)The cone defined by the constraints corresponding to a basis is pointed (i.e., does not contain any line). c)F is pointed. d)Every non-empty face of F has a vertex. e)Every vertex of F corresponds to at least one basic feasible solution. f)A vertex could correspond to more than one basic feasible solution. g)Every vertex of F is the unique optimum LP solution for some linear objective vector and with F as the feasible region. 5.Give LP Primal-Dual examples where: a)Both the Primal and the Dual are infeasible. b)The Primal is unbounded and the Dual is infeasible. c)Both the Primal and the Dual have bounded optima.

102
6.Farkas Lemma (Alternative versions): Prove the remaining 3 versions of Farkas Lemma. 7.Fundamental Theorem of Polyhedra [Minkowski-Weyl] when P is not pointed: a) Example 1: Consider the non-pointed polyhedron P={(x 1, x 2 ) T 2 | x 1 1, -x 1 1}. Show that P={ x 2 | x = V + R , 0, 0, i i = 1 }, where V consists of the 2 columns v 1 =(1,0) T, v 2 =(-1,0) T, and R consists of the 2 columns r 1 =(0,1) T, r 2 =(0,-1) T. [Note that v 1 and v 2 are not extreme points of P.] b) Example 2: Show the alternative representation (according to the Theorem) for the non-pointed polyhedron P = {(x 1, x 2, x 3 ) T 3 | x 1 +x 2 -x 3 1, -x 1 +x 2 -x 3 1}. c) Prove the ( ) part of the Theorem when P is not pointed. [Our referenced book [Ziegler, “Lectures on Polytopes”] contains a proof using the Fourier-Motzkin elimination method.] 8.Strong Duality Theorem: Prove the Strong Duality Theorem using Farkas Lemma. [Hint: = min{y T b : y T A=c, y 0} implies the system {y T b – < 0, y T A c = 0, y 0} has no solution in y. Reformulate this as F.L.(2) using an extra variable y 0. What is F.L.(1)?]

103
9.LP Duality via Lagrangian: Consider the primal-dual pair of linear programs (Primal) P* = max { c T x | Ax 0 } and their Lagrangian L(x,y) = c T x + y T (b – Ax) = b T y + x T (c – A T y). Define the functions p(x) = min y>0 L(x,y) and d(y) = max x L(x,y). That is, for any x, p(x) is obtained by a y > 0 that minimizes the Lagrangian (for the given x). Similarly, for any y > 0, d(y) is obtained by an x that maximizes the Lagrangian (for the given y). a)Show that b)Show that P* = max x p(x) = max x min y>0 L(x,y) and D* = min y>0 d(y) = min y>0 max x L(x,y). c)Show that if at least one of the Primal or the Dual LP has a feasible point, then the Lagrangian has the Saddle Point Property, i.e., max x min y>0 L(x,y) = min y>0 max x L(x,y). [Can you prove this without assuming the LP Strong Duality Theorem?]

104
10.In the restricted version of the mold casting problem we insist that the object is removed from its mold using a vertical translation (perpendicular to the top facet). (a) Prove that in this case only a constant number of top facets are possible. (b) Give a linear time algorithm that determines all such possible top facets. 11.Let H be a set of at least 3 half-planes. A half-plane h H is called redundant if its removal does not change H (the intersection of half-planes in H). Prove that h is redundant if and only if h’ h’’ h, for some h’,h’’ H – {h}. 12.Prove that RandomPermute(A[1..n]) is correct, i.e., each of the n! permutations of A can be the output with equal probability 1/n!. 13.Show how to implement MinDisk using a single routine MinDiskWithPoints(P,R) that computes MD(P,R) as defined before. Your algorithm should compute only a single random permutation during the entire computation. 14.Show the “bridge finding” problem in Kirkpatrick-Seidel’s O(n log h) time 2D CH algorithm can be solved in O(n) time by Megiddo-Dyer’s technique.

105
15.Minimum Area Annulus: An annulus is the portion of the plane contained between two concentric circles. Let a set P of n points in the plane be given. Give an efficient algorithm to find the minimum area annulus that contains P. [Hint: This can be solved by an LP. We want to find a center (x,y) and two radii R and r to minimize R 2 – r 2 subject to the 2n constraints r 2 (x-x i ) 2 + (y – y i ) 2 R 2. Now instead of the variables x,y,r,R, use the variables x, y, w x 2 +y 2 - R 2 and z x 2 +y 2 - r 2. Also ensure that the LP solution satisfies x 2 +y 2 = w + R 2 = z + r 2 max{w,z}. How?] 16.Minimum Enclosing Disk in 2D: Show this problem can be solved in O(n) time by prune-&-search. [Hint: First show the restricted problem where the disk center lies on a given line can be solved in O(n) time. Then show it can be decided in O(n) time on which side of this line the center of the (unrestricted) minimum enclosing disk lies. Then apply prune-&-search to this scheme.] 17.Maximum Inscribed Sphere: Let P d be a convex polytope, given as intersection of n half-spaces. Give a formulation of the problem to determine the largest sphere inscribed in P. You don’t need to solve the problem; just give a good formulation of it. [The center of this sphere is called the Chebyshev center of the polytope.]

106
END

Similar presentations

© 2016 SlidePlayer.com Inc.

All rights reserved.

Ads by Google