COSC 6114 Prof. Andy Mirzaian Linear Programming.

COSC 6114 Prof. Andy Mirzaian Linear Programming

TOPICS General Overview 2D Linear Programming Introduction
Fundamentals Duality Major Algorithms Open Problems 2D Linear Programming O(n log n) time by computation of feasible region O(n) time Randomized Incremental Method O(n) time Deterministic Prune-&-Search Method Smallest Enclosing Disk

References: [M. de Berge et al] chapter 4 [CLRS] chapter 29
[Edelsbrunner ‘87] chapter 10 Lecture Notes 4, 7-13 Reference books on “Optimization” listed on the course web-site

General Overview

The LP Problem maximize subject to:

Applications The most widely used Mathematical Optimization Model.
Management science (Operations Research). Engineering, technology, industry, commerce, economics. Efficient resource allocation: Airline transportation, Communication network – opt. transmission routing, Factory inventory/production control, Fund management, stock portfolio optimization. Approximation of hard optimization problems. . . .

Example in 2D x1 = 46/7 x2 = 59/7 max x1 + 8x2 subject to:
= 74 x2 x1 (1) (2) (4) (5) (3) max x x2 subject to: x  3 x2  2 –3x x2  14 4x1 – 3x2  25 x x2  15 optimum x x2 Feasible Region optimum basic constraints

Example in 3D Optimum (x,y,z)=(0,0,3) x y z z maximize subject to:

History of LP BC: Egypt, Babylon, India, China, Greece: [geometry & algebra] Egypt: polyhedra & pyramids. India: Sulabha suutrah (Easy Solution Procedures) [2 equations, 2 unknowns] China: Jiuzhang suanshu (9 Chapters on the Mathematical Art) [Precursor of Gauss-Jordan elimination method on linear equations] Greece: Pythagoras, Euclid, Archimedes, … 825 AD: Persia: Muhammad ibn-Musa Alkhawrazmi (author of 2 influential books): “Al-Maqhaleh fi Hisab al-jabr w’almoqhabeleh” (An essay on Algebra and equations) “Kitab al-Jam’a wal-Tafreeq bil Hisab al-Hindi” (Book on Hindu Arithmetic). originated the words algebra & algorithm for solution procedures of algebraic systems. Fourier [1826], Motzkin [1933] [Fourier-Motzkin elimination method on linear inequalities] Minkowski [1896], Farkas [1902], De la Vallée Poussin [1910], von Neumann [1930’s], Kantorovich [1939], Gale [1960] [LP duality theory & precursor of Simplex] George Dantzig [1947]: Simplex algorithm Exponential time in the worst case, but effective in practice. Leonid Khachiyan [1979]: Ellipsoid algorithm. The first weakly polynomial-time LP algorithm: poly(n,d,L). Narendra Karmarkar [1984]: Interior Point Method Also weakly polynomial-time. IPM variations are very well studied. Megiddo-Dyer [1984]: Prune-&-Search method O(n) time if the dimension is a fixed constant. Super-exponential on dimension.

LP in matrix form: Canonical Form: maximize cT x subject to: Ax  b
We assume all vectors are column vectors. Transpose (T) if you need a row vector.

LP feasible region F = { xd | Ax  b }

Primal Dual LPs: Primal: maximize cT x
subject to: Ax  b (canonical form) xT y A b n Dual: minimize yT b subject to: yT A = cT y  0 (standard form) cT d

Primal-Dual LP Correspondence
max cTx min yTb Constraint: aiT x  bi ajT x  bj akT x = bk (aiT = i-th row of constraint matrix A) Variable: yi  yj  0 yk unrestricted Variable: xi  xj  0 xk unrestricted Constraint: yTAi  ci yTAj  cj yTAk = ck (Ai = i-th column of constraint matrix A) FACT: Dual of the Dual is the Primal.

Example 1: Primal-Dual PRIMAL: max 16 x1 - 23 x2 + 43 x3 + 82 x4
subject to: 3 x x x x  -9 x x x x = 5 x x x x  x10, x20, x40 DUAL: min y y y3 subject to: 3 y y y  16 6 y y y  -23 -9 y y y = 43 4 y y y  82 y10, y30

Example 2: The Diet Problem
This is one of the earliest Primal-Dual LP formulations. Dantzig attributes it to: [G. J. Stigler, “The cost of subsistence,” J. Farm Econ. 27, no. 2, May 1945, ]. Primal: A homemaker has n choices of food to buy, and each food has some of each of m nutrients: Given: aij = amount of the ith nutrient in a unit of the jth food, i=1..m, j=1..n, ri = yearly minimum requirement of the ith nutrient, i=1..m, cj = cost per unit of the jth food, j=1..n. Determine: xj = yearly consumption of the jth food, j=1..n. Objective: Minimize total food purchasing cost cTx for the homemaker. Constraints: Meet yearly nutritional requirement (Ax  r) with nonnegative amount of each food (x  0). Primal: min cTx subject to: Ax  r , x  0.

Primal: min cTx subject to: Ax  r , x  0. Dual: max rTy subject to: ATy  c , y  0. Dual interpretation: A pill-maker wishes to market pills containing each of the m nutrients, yi = the price per unit of nutrient i, i=1..m. He wishes to be competitive with the price of real food, and maximize his profit from the sale of pills for an adequate diet. The dual constraints express the fact that the cost in pill form of all the nutrients in the jth food is no greater than the cost of the jth food itself. The objective function rTy is simply the pill maker’s profit for an adequate diet.

Example 3: Network Max Flow
Flow Network: N=(V,E,s,t,c) V = vertex set E = directed edge set s = source vertex t = terminal vertex c: E  + edge capacities find max flow f: E  + satisfying flow conservation & capacity constraints see also AAW maximize v Maximize Net flow out of source subject to: v for x=s Sy fxy - Sy fyx = -v for x=t Flow Conservation Law 0 for xV - {s,t} fe  ce for e  E Flow Capacity Constraints fe  for e  E Flow Non-negativity Constraints Assume flow is 0 on non-edges.

A: vertex-edge incidence matrix Rows indexed by vertices V, columns by edges E. For edge e=(i,j)E : Aie = +1, Aje = -1, Ake = 0 kV-{i,j}. e j e i +1 fe A = ce j -1 i

j e fe i ce j e e i ce

LP: Fundamental Facts

LP optimal solution set
Feasible region F = { xd | Ax  b } is the intersection of n half-spaces. F is a convex polyhedron (possibly empty or unbounded). Convexity: local optimality  global optimality. A supporting hyper-plane is one that does not intersect the interior of F. A face of F is the intersection of F with some supporting hyper-plane. We consider F itself as a (full dimensional) face of F.  is also a face of F. Vertex  0 dimensional face. Edge  (bounded) 1 dimensional face. optimum objective value z* = max { cTx : xF} (may not exist or be unbounded) supporting hyper-plane cTx = z* optimum face F* = { x : cTx = z* , xF} F*  interior(F) = F is pointed if it doesn’t contain any line. Hence, F= or has a vertex (i.e., rank(A)=d). F is pointed  every face of F is pointed. F* & pointed  F* contains a vertex of F (an optimal vertex) [In that case, we can focus on searching for an optimal vertex.] For further details see the upcoming slides.

Separation Lemma The Hahn-Banach Separation Theorem: P
Consider any non-empty closed convex set P and a point b in n . bP  there is a hyper-plane that separates b from P, i.e.,  hyper-plane yTx =  for some yn and  , such that yTb <  and yTz   zP. yTx =  P b Proof: Consider the closest point pP to point b Take the hyper-plane through p and orthogonal to segment bp. p

Extreme Points FACT: A non-empty closed convex set P has an extreme point if and only if P does not contain any line. Proof (): Suppose P contains a line L. We must prove that no point xP is an extreme point of P. Let L’ be the line parallel to L that passes through x. CLAIM: P contains L’. This claim implies x is not an extreme point of P. (Why?) Refutation of the Claim leads to a contradiction as follows: Suppose there is a point b  L’ – P. By the Separation Lemma, there is a hyperplane H that separates b from P. H intersects L’ (at some point between x and b). H must also intersect line L (which is parallel to L’). That would contradict the fact that H separates b from P  L. This contradiction proves the Claim. L x L’ b H

Extreme Points FACT: A non-empty closed convex set P has an extreme point if and only if P does not contain any line. Proof (): Suppose P does not contain any line. We apply induction on the dimension. If P is a single point, then it is an extreme point. Now suppose dim(P) > 0. Since P does not contain any line, P is a proper subset of aff(P) (affine hull of P). So, there is a point b  aff(P) – P. By the Separation Lemma, there is a hyperplane H separating b from P in aff(P). Move H parallelwise toward P until H is tangent to P. Let f = HP. dim(f)  dim(H) < dim(P). f is a non-empty closed convex set that contains no line. So, by the induction hypothesis, f must have an extreme point x. x is an extreme point of P also. (Why?) H P f

Extreme Points The Krein-Milman Theorem:
A closed bounded convex set is equal to the convex hull of its extreme points. Proof: By convexity & closedness, the given set P contains the convex hull of its extreme points. The converse, i.e., every point xP can be represented as a convex combination of extreme points of P, is proved by induction on the dimension. If x is an extreme point of P, we are done. Otherwise, draw a line L through x. Let yz be the maximal segment of L in P. x is a convex combination of the boundary points y and z (as shown). By induction, y & z can be represented as convex combinations of extreme points of the lower dimensional closed bounded convex sets PH1 and PH2, respectively. Hence, x is a convex combination of extreme points of P (details for exercise). H1 H2 y x z P L

Farkas Lemma Farkas Lemma:
Let And, bn. Then exactly one of the following must hold: (1) xd : Ax  b. (2) yn : yTA = 0, yTb < 0, y0. Proof: [(1) & (2) cannot both hold] Else: 0 = 0x = (yTA)x = yT(Ax)  yTb < 0. [ (1)  (2)]: P = { zn | z=Ax+s, xd , sn , s0 } is convex & non-empty(0P).  (1) bP  y,: yTb <   yTz zP by Separation Lemma y,: yTb <   yT(Ax+s) s  0,x y,: yTb <   (yTA)x+ yTs s  0,x y,: yTb <   yTs s  0, & yTA = if yTA0, take x=(-1- yTs) ATy ||yTA||-2 y,: yTb <   yTs s  0, & yTA = 0, & y  if yi <0, take si= - (1+||)/yi & sj=0 for ji y,: yTb <   0  yTs s  0, & yTA = 0, & y  if >0, take s = (/2)y ||y||-2 y: yTb < 0, yTA = 0, y  0 (2)

Farkas Lemma Farkas Lemma (alternative versions): Let And, bn.
xd : Ax  b  yn : yTA = 0, yTb < 0, y0. xd : Ax  b, x   yn : yTA  0, yTb < 0, y0. xd : Ax = b  yn : yTA = 0, yTb < 0. xd : Ax = b, x   yn : yTA  0, yTb < 0.

Convex Cones, Sets & Polyhedra
Minkowski sum (of vector sets): P+Q = {x+y | xP, yQ} Convex Cone: C ( d), 0  C, C + C  C, and   0: C  C. Finitely Generated Convex Cones and Convex Sets: Let V = {v1, … ,vm} and R = {r1, … ,rk} CH(V) = {1v1+2v2+ … +mvm | Si i=1,i0 } cone(R) = {m1r1+m2r2+ … +mkrk | mi0 } cone(R)={0} if R=. Extreme rays of the cone are shown bold. + = CH(V) + cone(R): polyhedron Recession Cone of the polyhedron

Caratheodory’s Theorem
Let X be a non-empty set in d. xcone(X) & x0 if and only if x= m1x1+m2x2+ … +mkxk for some m1, … ,mk > 0 and for some linearly independent x1, … ,xk X. (Hence, k  d.) (b) xCH(X) if and only if x= m1x1+m2x2+ … +mkxk for some m1, … ,mk > 0, Si mi =1. and for some affinely independent x1, … ,xk  X. (I.e., x2-x1, … ,xk-x1 are linearly independent. Hence, k  d+1.) x1 x2 cone(X) x1 CH(X) X x x x2 x3 (a) (b)

Caratheodory’s Theorem - Proof
(a) Consider a minimal k such that x = m1x1+m2x2+ … +mkxk , mi >0, xiX, i=1..k. (xi) are linearly dependent  r1, … ,rk   : at least one ri >0 & = r1x1+r2x2+ … +rkxk. So, x = (m1 – a r1) x1+ (m2 – a r2) x2+ … + (mk – a rk) xk . Take a = min { mi / ri | ri >0}. All coefficients remain non-negative and at least one becomes 0. This contradicts minimality of k. CH(X) cone(Y) Y x (b) Apply proof of (a) in d+1 to cone(Y) where Y = CH(X){1} = { (x,1)  d+1 | xCH(X) }.

Polyhedra - Theorem Fundamental Theorem of Polyhedra [Minkowski-Weyl]:
A subset P  d is the intersection of a finite set of half-spaces: P = { xd | Ax  b } for some A  nd, b  n if and only if P is the sum of convex hull of a finite set of points plus conical combination of a finite set of vectors: P = CH(V) + cone(R) for some V  dm , R  dm’ = { x d | x = Vl + Rm, m0, l0, Si li = 1 }. If P is pointed, columns of V are vertices of P, columns of R are extreme rays of P. P is a polytope  P is bounded  R=, V = vertex set of P. Corollary: A polytope can be viewed in 2 equivalent ways: 1) as intersection of half spaces (defined by its facets), 2) as convex hull of its vertices (cf. Krein-Milman Theorem).

Polyhedra – Example P V R x2 r2 v2 x1 V = (v1 , v2) R = ( r1 , r2) r1

Polyhedra – Proof Proof of (): [Assume polyhedron P is pointed.]
Let Q = { x d | x = Vl + Rm, m0, l0, Si li = 1 } where columns vi of V are extreme points of P, and columns ri of R are extreme rays of P. We claim Q = P: Q P: xQ implies x =y + z for some y=Vl, l0, Si li = 1, and z= Rm, m0. So yP, since y is a convex combination of extreme points of P. Also, z is a conic combination of extreme rays of P, so y+z P. So xP. Q P: WLOG assume P. If xQ, then (l,m) mm’ such that v1 l1 + v2 l2 + …+ vm lm + r1 m1 + r2 m2 +… + rm’ mm’ = x l l2 + … lm = l  0, m  0. So, by Farkas Lemma (p,po)d such that i: pTvi + po  0, pTri  0, pTx + po > 0. Now consider the LP max {pTy | yP}. For all rays ri of P, pTri  0. Hence, the LP has a bounded optimum. So, the optimum must be achieved at an extreme point of P. But for every extreme point vi of P we have pTx > -po  pTvi. So, pTx > -po  max {pTy | yP}. (The hyperplane pTy = -po separates x from P.) Therefore, xP [What happens if P is not pointed? See exercise 7.]

Polyhedra – Proof Proof of (): Q is the projection onto x of the polyhedron { (x,l,m) dmm’ | x - Vl – Rm = 0, m0, l0, Si li = 1 }. Hence, Q is also a polyhedron (i.e, intersection of a finite number of half-spaces).

Bases & Basic Solutions
Basis B: Any set of d linearly independent rows of A. B dd and det(B)  N = remaining non-basic rows of A. Basic Solution: BxB = bB  xB = B-1 bB . Basic Feasible Solution (BFS): If xB is feasible, i.e., NxB  bN . Each BFS corresponds to a vertex of the feasible polyhedron. xTB B  bB N  bN A b

Fundamental Theorem of LP
For any instance of LP exactly one of the following three possibilities holds: (a) Infeasible. (b) Feasible but no bounded optimum. (c) Bounded optimum. [Note: Feasible polyhedron could be unbounded even if optimum is bounded. It depends on the direction of the objective vector.] Moreover, if A has full rank (i.e.,  basis), then every nonempty face of the feasible polyhedron contains a BFS, and this implies: (1)  feasible solution   BFS. (2)  optimum solution   optimum solution that is a BFS. c Proof: If there is a basis, the basic cone contains the feasible region but does not contain any line. So the feasible region does not contain any line, hence it is pointed. So every non-empty face of it (including the optimal face, if non-empty) is pointed, and thus contains a vertex. (For details see exercise 4.)

LP Duality Theorem Primal: max { cTx | Ax  b } Dual: min { yTb | yTA = cT, y  0 } Weak Duality: x Primal feasible & y Dual feasible  cTx  yTb. Proof: cTx = (yTA)x = yT(Ax)  yTb. Corollary: (1) Primal unbounded optimum  Dual infeasible (2) Dual unbounded optimum  Primal infeasible (3) Primal & Dual both feasible  both have bounded optima. [Note: Primal & Dual may both be infeasible. See Exercise 5.] Strong Duality: If x & y are Primal/Dual feasible solutions, then x & y are Primal/Dual optima  cTx = yTb. Proof: By Farkas Lemma. See Exercise 8.

Example 1 P: max -18 x1 + 27 x2 subject to: -4 x1 + 5 x2  8
D: min 8 y y2 subject to: -4 y1 + y2  -18 5 y1 + y2  27 y10, y2  0 Optimum Solution for P: x1 = 3, x2 = 4 opt objective value = -18*3 + 27*4 = 54 Optimum Solution for D: y1 = 5, y2 = 2 opt objective value = 8*5 + 7*2 = 54

Primal: min cTx subject to: Ax  r , x  0. [Homemaker] Dual: max rTy subject to: ATy  c , y  0. [Pill Maker] Week Duality: For every primal feasible x and dual feasible y: cTx  rTy. Interpretation: the pill maker cannot beat the real food market for an adequate diet. Strong Duality: min {cTx | Ax  r , x  0} = max { rTy | ATy  c , y  0}. Interpretation: homemaker’s minimum cost food budget is equal to pill maker’s maximum profit for an adequate diet.

LP Optimization  Feasibility
Dual: minimize yT b subject to: yT A = cT y  0 Primal: maximize cT x subject to: Ax  b Is there (x,y) that satisfies: bT y - cT x  0 Ax  b AT y = c y  0

Complementary Slackness
Primal: maximize cT x subject to: aTi x  bi , i=1..n xj  0 , j=1..d Dual: minimize yT b subject to: ATj y  cj , j=1..d yi  0 , i=1..n Complementary Slackness Condition (CSC): yi (bi - aTi x ) = 0 , i=1..n xj (ATj y - cj ) = 0 , j=1..d FACT: Suppose x & y are Primal-Dual feasible Then the following statements are equivalent: x and y are Primal-Dual optimal, cTx = yTb, x & y satisfy the CSC.

Example 3: Max-Flow Min-Cut
See AAW for animation: S (v net out-flow) t f = c ,  =1 f = 0 ,  = 0  = 0  = 0  = 1

Min-Max Saddle-Point Property
FACT: For all functions f : XY   : max min f (x,y)  min max f (x,y) . xX yY yY xX Proof: Suppose LHS = f(x1,y1) and RHS = f(x2,y2). Then, f(x1,y1) ≤ f(x1,y2) ≤ f(x2,y2) . x1 x2 y1 y2

Min-Max Saddle-Point Property
Function f has the Saddle-Point Property (SPP) if it has a saddle-point, i.e., a point (x0, y0)  XY such that xX f (x , y0) ≤ f (x0 , y0) ≤ f(x0 , y) yY . FACT: f has the SPP if and only if max min f (x,y) = min max f (x,y) . xX yY yY xX

Duality via Lagrangian
P* = max {cT x | Ax  b } x D* = min { yT b | ATy = c } y0 Lagrangian: L(x,y) = cT x + yT(b – Ax) = yT b + xT(c – ATy) Fact: P* = max min L(x,y) , D* = min max L(x,y) . x y y x Proof: See Exercise 9. Weak Duality : max min L(x,y)  min max L(x,y) x y y x  P*  D* . Strong Duality : P or D feasible  Lagrangian has SPP. max min L(x,y) = min max L(x,y) x y y x  P* = D* .

Dantzig – Khachiyan – Karmarkar
George Dantzig [1947]: Simplex algorithm Exponential time in the worst case, but effective in practice. Leonid Khachiyan [1979]: Ellipsoid algorithm. The first weakly polynomial-time LP algorithm: poly(n,d,L). Narendra Karmarkar [1984]: Interior Point Method Also weakly polynomial-time. IPM variations are very well studied.

Dantzig: Simplex Algorithm
Pivots along the edges of the feasible polyhedron, monotonically improving the objective value. Each BFS corresponds to a vertex of the feasible polyhedron. c Simplex Pivot: A basic row leaves the basis and a suitable row enters the basis. A non-degenerate pivot moves the BFS along an edge of the feasible polyhedron from one vertex to an adjacent one. Pivot Rule: Decides which row leaves & which row enters the basis (i.e., which edge of the feasible polyhedron to follow next).

Center of Gravity A set of points K  d is called a convex body
if it is a full dimensional, closed, bounded, convex set. FACT: [Grünbaum 1960] Let point c be the center of gravity of a convex body K and H be an arbitrary hyperplane passing through point c. Suppose H divides K into two bodies K+ and K– . Then the volumes vol(K+) and vol(K–) satisfy: c K+ K– H

Iterative Central Bisection
FACT: Suppose the feasible set is a convex body F which is contained in a ball C0 of radius R and contains a ball C1 of radius r. Suppose we start with (the center of) the larger ball and iteratively perform central bisection until center of gravity of the remaining convex body falls within F. The number of such iterations is O(d log R/r). r R F C0 C1 Proof: vol(C0) = vdRd vol(C1) = vdrd vd = volume of the unit ball in d . With each iteration, the volume drops by a constant factor. # iterations = O( log vol(C0)/vol(C1) ).

Iterative Central Bisection
The # central bisection iterations O(d log R/r) is polynomial in the bit-size of the LP instance. However, each iteration involves computing the volume or center of gravity of the resulting convex body. This task seems to be hard even for LP instances. We don’t know how to compute them in polynomial time. So, we resort to approximation approximate the center of gravity approximate the convex body What simple approximating “shape” to use? Bounding Box, Ball, ellipsoid, …?

Ellipsoids An ellipsoid is an affine transform of the Euclidean unit ball. zo = center of ellipsoid E. A = a dd non-singular matrix. y x E zo

Ellipsoids Example:

( vd = volume of the unit ball in d .)
Ellipsoids zo E c xopt ( vd = volume of the unit ball in d .)

Löwner-John Ellipsoid
FACT 1: [K. Löwner 1934, Fritz John, 1948] Let K be a convex body in d. Then There is a unique maximum volume ellipsoid Ein(K) contained in K. K is contained in the ellipsoid that results from Ein(K) by centrally expanding Ein(K) by a factor of d (independent of K). If K is centrally symmetric, then the factor d in part (2) can be improved to d .

Löwner-John Ellipsoid
FACT 2: [K. Löwner 1934, Fritz John, 1948] Let K be a convex body in d. Then There is a unique minimum volume ellipsoid Eout(K) that contains K. K contains the ellipsoid that results from Eout(K) by centrally shrinking Eout(K) by a factor of d (independent of K). If K is centrally symmetric, then the factor d in part (2) can be improved to d .

Löwner-John metric approximation
Corollary: Any metric in d can be approximated by a Löwner-John metric with distortion factor at most d . Proof: Let || . ||B denote an arbitrary metric norm with (centrally symmetric) unit ball B. || x ||B = min { l | x  lB } for any x  d . Let ELJ = { xd | || A–1 x ||2  1 } be the min volume Löwner-John ellipsoid that contains B. The metric norm with ELJ as the unit ball is || x ||LJ = || A–1 x ||2 for any x  d . ELJ B

Iterative Löwner-John Ellipsoidal Cut
xk xout zo c xk-1 xopt

Iterative Löwner-John Ellipsoidal Cut
# iterations is polynomial in the bit-size of the LP instance. However, each iteration is expensive and involves computing Löwner-John Ellipsoid of a convex polytope. What next? Approximate the convex body by an ellipsoid.

Khachiyan: Ellipsoid Method
Circumscribed ellipsoid volume reduction by cuttings. Ek Ek+1 zk+1 zk # iterations = O(d2 log R/r) Time per iteration = O(d n)

The New York Times, November 7, 1979
Continued on next slide.

Karmarkar: Interior Point Method
Logarithmic Barrier and Central Path maximize cTx +  i log si subject to Ax+s=b, s > 0 = =1 =0.01 Central path

IPM: Primal-Dual Central Path
(P) max cTx (D) min bTy s.t. Ax+s=b, s  0 s.t. ATy=c, y  0 Optimality Conditions on (x,s,y): Ax+s=b, s  0 (Primal feasibility) ATy=c, y  0 (Dual feasibility) si yi = 0 i (Complementary Slackness) Primal-Dual Central Path parameterized by  > 0: Ax+s=b, s > 0 (Primal strict feasibility) ATy=c, y > 0 (Dual strict feasibility) si yi =  i (Centrality)

IPM: Primal-Dual Central Path
Ax+s=b, s > 0 ATy=c, y > 0 si yi =  i Algorithmic Framework: Assume both (P) and (D) have strictly feasible solutions. Then, for any  > 0, the above non-linear system of equations has a unique solution (x(), s(), y()). These points form the Primal-Dual Central Path { (x(), s(), y()) |   (0, ) }. lim 0 (x(), s(), y()) = (x*, s*, y*) is optimal solution of the Primal-Dual LP. Update approximate solution to the system while iteratively reducing  towards 0.

Open Problem Can LP be solved in strongly polynomial time?
Weakly Polynomial: Time = poly(n,d,L), [L = bit-length of input coefficients] Bit-length of each program variable = poly(n,d,L). Strongly Polynomial: Time = poly(n,d) [i.e., independent of input coefficient sizes] Bit-length of each program variable = poly(n,d,L). [ Steve Smale, “Mathematical Problems for the Next Century’’, Mathematical Intelligencer, 20:7–15, 1998] lists 18 famous open problems and invites the scientific community to solve them in the 21st century. Problem 9 on that list is the above open problem on LP.

Hirsch’s Conjecture [1957]
Hirsch’s Conjecture: Diameter of any d-polyhedron with n facets satisfies D(d,n)  n – d. That is, any pair of its vertices are connected by a path of at most n  d edges. Best known upper-bound [Kalai, Kleitman 1992]: D(d,n)  nlog d + 2 For a recent survey on this topic click here. This conjecture was disproved by Francisco Santos in June 2010. Click here to see Santos’s paper.

LP in dimension 2

2D Linear Programming Objective function  = c1 x + c2 y, cT =(c1 , c2) Feasible region F c max  = c1 x + c2 y optimum F y x

2D Linear Programming Feasible region F is the intersection of n half-planes. F is (empty, bounded or unbounded) convex polygon with  n vertices. F can be computed in O(n log n) time by divide-&-conquer (See Lecture-Slide 3). If F is empty, then LP is infeasible. Otherwise, we can check its vertices, and its possibly up to 2 unbounded edges, to determine the optimum. The latter step can be done by binary search in O(log n) time. If objective changes but constraints do not, we can update the optimum in only O(log n) time. (We don’t need to start from scratch). Improvement Next: Feasible region need not be computed to find the optimum vertex. Optimum can be found in O(n) time both randomized & deterministic.

2D LP Example: Manufacturing with Molds

The Geometry of Casting: Is there a mold for an n-faceted 3D polytope P such that P can be removed from the mold by translation? Lemma: P can be removed from its mold with a single translation in direction d  d makes an angle  90 with the outward normal of all non-top facets of P. mold P f f ’ (f ‘) = - (f) d Corollary: Many small translations possible  Single translation possible.

The Geometry of Casting: Is there a mold for an n-faceted 3D polytope P such that P can be removed from the mold by translation? z f d=(x,y,1) d z=1 P f ’ y mold (f ‘) = - (f) x (f) = ( x(f) , y(f) , z(f) ) outward normal to facet f of P. dT.(f) ≤ 0  non-top facet f of P  x(f) . x + y(f) . y + z(f) ≤ 0 f n-1 constraints THEOREM: The mold casting problem can be solved in O(n log n) time. (This will be improved to O(n) time on the next slides.)

Randomized Incremental Algorithm

Randomization Algorithm RandomPermute (A) O(n) time
Random(k): Returns an integer i  1..k, each with equal probability 1/k [Use a random number generator.] Algorithm RandomPermute (A) O(n) time Input: Array A[1..n] Output: A random permutation of A[1..n] with each of n! possible permutations equally likely. for k  n downto 2 do Swap A[k] with A[Random(k)] end. This is a basic “initial” part of many randomized incremental algorithms.

2D LP: Incremental Algorithm
Method: Add constraints one-by-one, while maintaining the current optimum vertex. e v Infeasible Unbounded Non-unique optimum Unique optimum  c Possible Outcomes: H(i) H(j) H(k) Input: (H, c), H = { H(1), H(2), … , H(n)} n half-planes, c = objective vector Output: Infeasible: (i,j,k), or Unbounded: , or Optimum: v = argmaxx { cTx | x  H(1)H(2)…H(n) }. Define: C(i) = H(1)H(2)…H(i) , for i = 1..n v(i) = optimum vertex of C(i) , for i=2..n cTv(i) = max { cTx | x  C(i) } Note: C(1)  C(2)  …  C(n).

LEMMA: (1) v(i-1)  H(i)  v(i-1)  C(i)  v(i)  v(i-1). (2) v(i-1)  H(i)  (2a) C(i) = , or (2b) v(i)  L(i)  C(i-1), L(i) =bounding-line of H(i). c v(i-1) v(i-1) v(i)=v(i-1) H(i) L(i) L(i) H(i) H(i) C(i-1) C(i-1) C(i-1) H(k) v(i) H(j) (2b) (2a) (1)

Algorithm PreProcess (H,c) O(n) time   min { angle between c and outward normal of H(i) | i=1..n } H(i) = most restrictive constraint with angle  Swap H(i) with H(1) (L(1) is bounding-line of H(1)). If (parallel) H(j) with angle  and H(1)H(j)= then return “infeasible”. If L(1)  H(j) is unbounded for all H(j) H, then   most restrictive L(1)  H(j) over all H(j) H return (“unbounded”,  ) If L(1)  H(j) is bounded for some H(j)H, then Swap H(j) with H(2) and return “bounded”. v(2)  c L(1) L(1) L(1) H(j)  H(j) H(1) H(2) Disjoint parallel. H(1) H(1) (4) (1) (2) (3)

Randomized Incremental 2D LP Algorithm
Input: (H, c), H = { H(1), H(2), … , H(n)} n half-planes, c = objective vector Output: Solution to max { cTx | x  H(1)H(2)…H(n) } if PreProcess(H,c) returns (“unbounded”, ) or “infeasible” then return the same answer (* else bounded or infeasible *) v(2)  vertex of H(1)  H(2) RandomPermute (H[3..n]) for i  3..n do if v(i-1)  H(i) then v(i)  v(i-1) else v(i)  optimum vertex p of L(i)(H(1)…H(i-1)) (* 1D LP *) if p does not exist then return “infeasible” end-for return (“optimum”, v(n)) end.

THEOREM: 2D LP Randomized Incremental algorithm has the following complexity: Space complexity = O(n) Time Complexity: (a) O(n2) worst-case (b) O(n) expected-case. Proof of (a): Line 6 is a 1D LP with i-1constraints and takes O(i) time. Total time over for-loop of lines 4-8:

Proof of (b): Define 0/1 random variables Lines 5-7 take O( i*X(i) +1) time. Total time is Expected time: linearity of expectation Backwards Analysis “Fix” C(i) = { H(1), H(2)}  {H(3), …, H(i)} Random : C(i-1) = C(i) – {H(i)} random v(i) is defined by 2 H(j)’s. The probability that one of them is H(i) is  2/(i-2). This does not depend on C(i). Hence, remove the “Fix” assumption.

Randomized Incremental LP in d dimension
Randomized Incremental algorithm can be used (with minor modifications) in d dimensions, recursively. For example, for 3D LP line 6 becomes a 2D LP with i-1 constraints. In general, it will be a (d-1) dimensional LP. Define: T(n,d) = running time of algorithm for n constraints & d variables. This would be exponential in both n and d. Expected time (by backwards analysis):

Randomized Incremental LP in d dimension
[For nd, T(n,d) = O(dn2), by Gaussian elimination.]

Randomized Incremental Algorithm for Smallest Enclosing Disk

Smallest Enclosing Disk
Input: A set P={p1, p2, … , pn } of n points in the plane. Output: Smallest enclosing disk D of P. Lemma: Output is unique. Incremental Construction: P[1..i] = {p1, p2, … , pi } D(i)= smallest enclosing disk of P[1..i] . pi+1 Lemma: (1) pi  D(i-1)  D(i) = D(i-1) (2) pi  D(i-1)  pi lies on the boundary of D(i). pi D(i+1) D(i)=D(i-1)

LEMMA: Let P and R be disjoint point sets in the plane. pP, R possibly empty. Define MD(P, R) = minimum disk D such that P  D & R   D ( D = boundary of D). If MD(P, R) exists, then it’s unique, p  MD(P-{p}, R)  MD(P,R) = MD(P-{p}, R), p  MD(P-{p}, R)  MD(P, R) = MD(P-{p}, R  {p}). Proof: (1) If non-unique   smaller such disk: (2) is obvious. D(0) D(1) R p D() D(0)  MD(P-{p}, R) D(1)  MD(P, R) D()  (1-) D(0) +  D(1)    1 As  goes from 0 to 1, D() continuously deforms from D(0) to D(1) s.t.  D(0)   D(1)   D(). p  D(1) – D(0)  by continuity  smallest *, 0 < *  1 s.t. p  D(*)  p   D(*).  P  D(*) & R   D(*)  *=1 by uniqueness. Therefore, p is on the boundary of D(1).

Algorithm MinDisk (P[1..n]) RandomPermute(P[1..n]) D(2)  smallest enclosing disk of P[1..2] for i  3..n do if pi  D(i-1) then D(i)  D(i-1) else D(i)  MinDiskWithPoint (P[1..i-1] , pi) return D(n) Procedure MinDiskWithPoint (P[1..j],q) RandomPermute(P[1..j]) D(1)  smallest enclosing disk of p1 and q for i  2..j do if pi  D(i-1) then D(i)  D(i-1) else D(i)  MinDiskWith2Points (P[1..i-1] , q, pi) return D(j) Show random permutation only once is enough Procedure MinDiskWith2Points (P[1..j],q1,q2) D(0)  smallest enclosing disk of q1 and q2 for i  1..j do if pi  D(i-1) then D(i)  D(i-1) else D(i)  Disk (q1, q2, pi) return D(j)

THEOREM: The smallest enclosing disk of n points in the plane can be computed in randomized O(n) expected time and O(n) space. Proof: Space O(n) is obvious. MinDiskWith2Points (P,q1,q2) takes O(n) time. MinDiskWithPoint (P,q) takes time: where Apply this idea once more: expected running time of MinDisk is also O(n). P[1..i] q “Fix” P[1..i] = {p1, … , pi} backwards P[1..i-1] = {p1, … , pi} - {pi} one of these is pi with prob.  2/i

2D LP Prune-&-Search

2D LP: Megiddo(83), Dyer(84): Prune-&-Search
Original LP: min { ax + by | ai x + bi y + ci  0 , i=1..n } STEP 1: Transform so that the objective hyper-plane becomes horizontal. (a,b)  (0,0), so WLOG b  0. Coordinate transform: Y = ax + by i = ai – (a/b) bi X = x i = bi /b min { Y | i X + i Y + ci  0 , i=1..n } X Y F feasible region optimum

min { Y | i X + i Y + ci  0 , i=1..n }
2D LP: Prune-&-Search min { Y | i X + i Y + ci  0 , i=1..n } STEP 2: Partition the constraints (the index set [1..n]) I0 = { i | i = 0 } vertical line constraints determine feasible x-interval I- = { i | i < 0 } I+ = { i | i > 0 } STEP 3: X Y F U1 U2 STEP 4:

2D LP: Prune-&-Search F+ F F- F+ F- STEP 5:
So, the transformed constraints are: F-(X)  Y  F+ (X). Since we want to minimize Y, our objective function is F-(X). STEP 6: Our new (transformed) problem is: X Y F U1 U2 optimum F+ F-

2D LP: Prune-&-Search STEP 7: An Evaluation Stage:
7(a): Given X [U1 , U2 ] , we can determine in O(n) time: F-(X) = max { i X + i | i  I- } F+(X) = min { i X + i | i  I+ } f- (L) (X) = left slope of F-(X) at X f- (R) (X) = right slope of F-(X) at X f+ (L) (X) = left slope of F+(X) at X f+ (R) (X) = right slope of F+(X) at X Note: if only one i  I- achieves the “max” in F-(X), then f- (L) (X) = f- (R) (X) = i . Similarly with F+(X). X’ X F-

f- (L) (X)  f+ (L) (X) & f- (R) (X)  f+ (R) (X)
2D LP: Prune-&-Search STEP 7: An Evaluation Stage: 7(b): In O(n) time we can determine for X  [U1 , U2 ], whether: X achieves the minimum of F-(X) and is feasible (hence optimum). X is infeasible & there is no solution to the LP X is infeasible and we know which side (left/right) of X the feasible region F lies. X is feasible and we know which side of X the optimum solution lies. In the latter two cases, shrink the interval [U1 , U2 ] accordingly. F-(X) F+(X) X f- (L) (X) > f+ (L) (X) infeasible F to the left f- (R) (X) < f+ (R) (X) F to the right f- (L) (X)  f+ (L) (X) & f- (R) (X)  f+ (R) (X)  F = ; LP is infeasible OR

2D LP: Prune-&-Search METHOD: A series of evaluation stages .
Prune at least a fixed fraction (1-) of the constraints in each stage. These constraints won’t play a role in determining the optimum vertex. After the k-th stage, # constraints remaining  k n. At most log n stages. Total time:

2D LP: Prune-&-Search METHOD: (continued)  = 3/4 is achievable.
Which X to select to evaluate then prune? Pair up constraints in I+ (do the same for I- ) Suppose i,j  I+ are paired up. parallel eliminate U1 U2 Xij eliminate neither one

2D LP: Prune-&-Search METHOD: (continued) Next evaluation point: X* = median of the Xij’s of the non-eliminated pairs. O(n) time. Half of the Xij’s will be on the wrong side of X*. We will eliminate one line from at least half of the pairs. So, at least ¼ of the lines will be eliminated in this stage. At most  = ¾ of the lines remain. THEOREM: A 2-variable LP with n constraints can be solved in O(n) time in the worst-case by Megiddo-Dyer’s Prune-&-Search method. GENERALIZATION: Megiddo[84] showed d dim. LP can also be solved by this technique in O(n) time assuming d is a fixed constant. [However, the hidden constant in O(n) is super-exponential in d.]

Extensions & Applications
The Smallest Enclosing Disk: A Quadratic Program. See Exercise 16. THEOREM [Megiddo]: The smallest enclosing disk can be solved in O(n) time in the worst-case by the Prune-&-Search method. See Lecture Notes 7-13 for more on this topic.

Exercises

Prove the following claims:
A Non-Linear Primal Dual Example: [Rochat, Vecten, Fauguier, Pilatte, 1811] We are given 3 points A,B,C in general position in the plane. Primal: Consider any equilateral triangle PQR whose 3 sides pass through A,B,C, respectively. (See Figure below.) Let H denote the altitude of the equilateral triangle PQR Find PQR to maximize H. Dual: Consider any point S, with L as the sum of the lengths of the segments L = SA + SB + SC. Find S to minimize L [The optimum S is called the Steiner point of ABC.] Prove the following claims: For any primal-dual feasible solutions H  L. At optimality max H = min L. At optimality SA  QR, SB  RP, SC  PQ, where  means perpendicular. A B C P Q R S

LP Primal Dual Example: Give the dual of the following LP instance:
max x x x x x5 subject to: -2 x x x x x = 53 11 x x x x x  31 -4 x x x x x5  x x x x x5  61 x10, x3  0, x40, x50 Fractional Knapsack Problem (FKP): FKP is the LP max { vTx | wTx  W, 0xi1, i=1..n }, where vT=(v1 ,…, vn ) and wT =(w1 ,…, wn ) are vectors of item values and weights, and W is the knapsack capacity. Assume these are all positive reals. The real variable xi , 0xi1, is the selected fraction of the i-th item, for i=1..n. a) Give the dual of FKP. b) Use the Duality Theorem to show that a simple greedy algorithm obtains optimal primal-dual solutions. [Hint: sort vi / wi , for i=1..n.]

Polyhedral Vertices versus Basic Solutions: Consider the (possibly empty) polyhedron F = { xd | Ax  b } for some And, bn, d  n. Assume rank(A)=d (i.e., A contains d linearly independent rows). Show the following facts: F has a basic solution (not necessarily feasible). The cone defined by the constraints corresponding to a basis is pointed (i.e., does not contain any line). F is pointed. Every non-empty face of F has a vertex. Every vertex of F corresponds to at least one basic feasible solution. A vertex could correspond to more than one basic feasible solution. Every vertex of F is the unique optimum LP solution for some linear objective vector and with F as the feasible region. Give LP Primal-Dual examples where: Both the Primal and the Dual are infeasible. The Primal is unbounded and the Dual is infeasible. Both the Primal and the Dual have bounded optima.

Farkas Lemma (Alternative versions): Prove the remaining 3 versions of Farkas Lemma.
Fundamental Theorem of Polyhedra [Minkowski-Weyl] when P is not pointed: a) Example 1: Consider the non-pointed polyhedron P={(x1, x2)T2 | x11, -x11} Show that P={ x 2 | x = Vl + Rm, m0, l0, Si li = 1 }, where V consists of the columns v1=(1,0)T, v2=(-1,0)T, and R consists of the 2 columns r1=(0,1)T, r2=(0,-1)T. [Note that v1 and v2 are not extreme points of P.] b) Example 2: Show the alternative representation (according to the Theorem) for the non-pointed polyhedron P = {(x1, x2, x3)T3 | x1+x2 -x31, -x1+x2 -x31}. c) Prove the () part of the Theorem when P is not pointed. [Our referenced book [Ziegler, “Lectures on Polytopes”] contains a proof using the Fourier-Motzkin elimination method.] Strong Duality Theorem: Prove the Strong Duality Theorem using Farkas Lemma. [Hint: a = min{yTb : yTA=c, y0} implies the system {yTb – a < 0, yTA - c = 0, y0} has no solution in y. Reformulate this as F.L.(2) using an extra variable y0. What is F.L.(1)?]

LP Duality via Lagrangian: Consider the primal-dual pair of linear programs (Primal) P* = max { cTx | Ax < b } (Dual) D* = min { bTy | ATy = c and y > 0 } and their Lagrangian L(x,y) = cTx + yT (b – Ax) = bTy + xT (c – ATy) . Define the functions p(x) = miny>0 L(x,y) and d(y) = maxx L(x,y). That is, for any x, p(x) is obtained by a y > 0 that minimizes the Lagrangian (for the given x). Similarly, for any y > 0, d(y) is obtained by an x that maximizes the Lagrangian (for the given y). Show that Show that P* = maxx p(x) = maxx miny>0 L(x,y) and D* = miny>0 d(y) = miny>0 maxx L(x,y). Show that if at least one of the Primal or the Dual LP has a feasible point, then the Lagrangian has the Saddle Point Property, i.e., maxx miny>0 L(x,y) = miny>0 maxx L(x,y). [Can you prove this without assuming the LP Strong Duality Theorem?]

In the restricted version of the mold casting problem we insist that the object is removed from its mold using a vertical translation (perpendicular to the top facet). (a) Prove that in this case only a constant number of top facets are possible. (b) Give a linear time algorithm that determines all such possible top facets. Let H be a set of at least 3 half-planes. A half-plane hH is called redundant if its removal does not change H (the intersection of half-planes in H). Prove that h is redundant if and only if h’h’’  h, for some h’,h’’H – {h}. Prove that RandomPermute(A[1..n]) is correct, i.e., each of the n! permutations of A can be the output with equal probability 1/n!. Show how to implement MinDisk using a single routine MinDiskWithPoints(P,R) that computes MD(P,R) as defined before. Your algorithm should compute only a single random permutation during the entire computation. Show the “bridge finding” problem in Kirkpatrick-Seidel’s O(n log h) time 2D CH algorithm can be solved in O(n) time by Megiddo-Dyer’s technique.

Minimum Area Annulus: An annulus is the portion of the plane contained between two concentric circles. Let a set P of n points in the plane be given. Give an efficient algorithm to find the minimum area annulus that contains P. [Hint: This can be solved by an LP. We want to find a center (x,y) and two radii R and r to minimize R2 – r2 subject to the 2n constraints r2  (x-xi)2 + (y – yi)2  R2 . Now instead of the variables x,y,r,R, use the variables x, y, w  x2 +y2 - R2 and z  x2 +y2 - r2. Also ensure that the LP solution satisfies x2 +y2 = w + R2 = z + r2  max{w,z}. How?] Minimum Enclosing Disk in 2D: Show this problem can be solved in O(n) time by prune-&-search. [Hint: First show the restricted problem where the disk center lies on a given line can be solved in O(n) time. Then show it can be decided in O(n) time on which side of this line the center of the (unrestricted) minimum enclosing disk lies. Then apply prune-&-search to this scheme.] Maximum Inscribed Sphere: Let P  d be a convex polytope, given as intersection of n half-spaces. Give a formulation of the problem to determine the largest sphere inscribed in P. You don’t need to solve the problem; just give a good formulation of it. [The center of this sphere is called the Chebyshev center of the polytope.]

COSC 6114 Prof. Andy Mirzaian Linear Programming.

Similar presentations

Presentation on theme: "COSC 6114 Prof. Andy Mirzaian Linear Programming."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

COSC 6114 Prof. Andy Mirzaian Linear Programming.

Similar presentations

Presentation on theme: "COSC 6114 Prof. Andy Mirzaian Linear Programming."— Presentation transcript:

Similar presentations

About project

Feedback