1. Hon Wai Leong, NUS (CS6234, Spring 2009) Page 1 Copyright © 2009 by Leong Hon Wai CS6234: Lecture 4  Linear Programming  LP and Simplex Algorithm [PS82]-Ch2  Duality [PS82]-Ch3  Primal-Dual Algorithm [PS82]-Ch5  Additional topics: uReading/Presentation by students Lecture notes adapted from Comb Opt course by Jorisk, Math Dept, Maashricht Univ,

2. 12/25/2015Combinatorial Optimization Masters OR Combinatorial Optimization Chapter 3 of [PS82] Duality

3. 12/25/2015Combinatorial Optimization Masters OR LP in general form min c´ x s.t. a´ i x= b i i ε M a´ i x ≤ b i i ε M´ x j ≥0j ε N x j freej ε N´ 1.Introduce surplus variables for inequality constraints 2.Replace free variables by two non-negative variables.

4. 12/25/2015Combinatorial Optimization Masters OR In standard form… minc ~ ´ x ~ s.t.A ~ x ~ = b x ~ ≥ 0 where A ~ has extra columns for x´s in N´, and for slack variables, x ~ has extra variables for x´s in N´ and for slack variables, c ~ has extra elements for x´s in N´.

5. 12/25/2015Combinatorial Optimization Masters OR Starting the dual The simplex method gives an optimal solution x 0 ~ to the LP in standard form, with a basis Q, satisfying c ~ ´ – (c B ~ ´ B ~-1 )A ~ ≥ 0. Thus, π ´ = c B ~ ´ B ~-1 is a feasible solution to the problem π ´ A ~ ≤ c ~ ´ where π ε R m.

6. 12/25/2015Combinatorial Optimization Masters OR The dual 1.π´ A j ≤ c j, for j ε N, 2.π´ A j ≤ c j, for j ε N´, and π´ A j ≤ –c j, for j ε N´, and hence π´ A j = c j, for j ε N´. 3.–π i ≤ 0, for i ε M´, or π i ≥ 0, for i ε M´. Objective: max π´ b.

7. 12/25/2015Combinatorial Optimization Masters OR Definition of the Dual Definition 3.1: Given an LP in general form, called the primal, the dual is defined as follows PrimalDual Min c ´ xMax π ´ b a ´ i x = b i i ε M π i free a ´ i x ≥ b i i ε M ´ π i ≥ 0 x j ≥ 0 j ε N π ´ A j ≤ c j x j free j ε N ’ π ´ A j = c j

8. 12/25/2015Combinatorial Optimization Masters OR Strong duality Theorem 3.1: If an LP has an optimal solution, so does its dual, and at optimality their costs are equal. Proof. Let x and π be optimal solutions to the primal and the dual resp., then c ´ x ≥ π ´ Ax ≥ π ´ b. (*) From the dual From the primal

9. 12/25/2015Combinatorial Optimization Masters OR Strong duality proof Thus the primal has cost at least as high as the dual. Then, if the primal has a feasible solution, the cost of the dual cannot be unbounded. It has feasible solution π´, and since it is not unbounded, it must have an optimum. The cost of π´ is π´ b = c B ~ ´ B ~-1 b = c B ~ ´ x 0 ~ which is the optimal solution of the primal. Together with (*) this suffices to prove the Theorem.

10. 12/25/2015Combinatorial Optimization Masters OR The dual of the dual Theorem 3.2. The dual of the dual is the primal. Proof. Left as an exercise. Theorem 3.3: Given a primal-dual pair, either of the following holds: 1.Both have a finite optimum 2.Both are infeasible 3.One is unbounded the other is infeasible Proof. Skip.

11. 12/25/2015Combinatorial Optimization Masters OR Complementary slackness Theorem 3.4: A pair, x, π, respectively feasible in a primal-dual pair is optimal if and only if: u i = π i (a´ i x - b i ) = 0 for all i (1) v j = (c j - π´ A j ) x j = 0 for all j. (2)

12. 12/25/2015Combinatorial Optimization Masters OR Complementary slackness (2) Proof: Recall u i = π i (a´ i x - b i ) & v j = (c j - π´A j ) x j Define u =  u i (≥ 0) & v =  v j (≥ 0) Thus, u = 0 if and only if (1) holds (each u i = 0) and v = 0 if and only if (2) holds (each v j = 0). Now, notice that (u + v) = c´x – π´b. Thus, if (1) and (2) hold, (u + v) = 0, and c´x = π´b. This implies that x and π have the same objective value, and hence are both optimal.

13. 12/25/2015Combinatorial Optimization Masters OR Complementary slackness (3) Consequences: If an inequality constraint in the dual is not binding, then the corresponding primal variable must have value zero and vice versa. Likewise, if a nonnegative variable has strictly positive value, then the corresponding constraint must be binding.

14. 12/25/2015Combinatorial Optimization Masters OR Tableau (for example 2.6) Min Z = x 1 + x 2 + x 3 + x 4 + x 5 s.t. 3x 1 + 2x 2 +x 3 = 1 5x 1 + 2x 2 + x 3 + x 4 = 3 2 x 1 + 5 x 2 + x 3 + x 5 = 4 x1x1 x2x2 x3x3 x4x4 x5x5 – z011111 132100 351110 425101

15. 12/25/2015Combinatorial Optimization Masters OR Simplex Algorithm (Tableau Form) x1x1 x2x2 x3x3 x4x4 x5x5 –z–z011111 132100 351110 425101 x1x1 x2x2 x3x3 x4x4 x5x5 –z–z-6-3 000 x3=x3=132100 x4=x4=22010 x5=x5=3 3001 x1x1 x2x2 x3x3 x4x4 x5x5 –z–z011111 x3=x3=132100 x4=x4=22 010 x5=x5=3 3001 Diagonalize to get bfs { x 3, x 4, x 5 } Determine reduced cost by making c ~ j = 0 for all basic col

16. 12/25/2015Combinatorial Optimization Masters OR

17. 12/25/2015Combinatorial Optimization Masters OR Farkas Lemma Skip 3.3

18. 12/25/2015Combinatorial Optimization Masters OR The shortest path problem and its dual Definition 3.3 Given a directed graph G=(V,E) and a nonnegative weight c j ≥ 0 associated with each arc e j ε E, an instance of the shortest path problem (SP) is the problem of finding a directed path from a distinguished source node s to a distinguished terminal node t, with the minimum total weight.

19. 12/25/2015Combinatorial Optimization Masters OR Shortest path problem Feasible set F is given by F = {P = (e j 1,..,e j k ) : P is a directed path from s to t in G}. Cost c(P) = Σ i=1 k c j i. Formulation as LP uses node-arc incidence matrix A: 1if arc j leaves node i a ij = {-1-f arc j enters node i 0otherwise

20. 12/25/2015Combinatorial Optimization Masters OR SP Example 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5 e1e1 e2e2 e3e3 e4e4 e5e5 s11000 A=t000 a 0110 b0 01

21. 12/25/2015Combinatorial Optimization Masters OR LP formulation Mine 1 +2e 2 +2e 3 +3e 4 + e 5 s.t. e 1 + e 2 = 1 - e 4 - e 5 = -1 - e 1 e 3 + e 4 = 0 - e 2 - e 3 + e 5 = 0 0 ≤ e i ≤ 1, i =1…5

22. 12/25/2015Combinatorial Optimization Masters OR Redundancy Claim: Any solution that satisfies 3 of the four constraints, satisfies all 4 of them. Easily checked. Thus, we omit the constraint for the sink t. Next, we consider the tableau of a feasible solution

23. 12/25/2015Combinatorial Optimization Masters OR Starting Tableau e1e1 e2e2 e3e3 e4e4 e5e5 -z012231 s111000 a00110 b00 01 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5

24. 12/25/2015Combinatorial Optimization Masters OR Summary of LP for SP Simplex algorithm in tableau form –Cost vector c is also included as a constraint (Read details in [PS82]-Ch-2) Each basis has (n-1) edges; –Not all are positive; –Edges with +ve flows are on the (s-t) path Pivoting – bring in a new edge; –Based on relative cost (row0 of tableau) Clever idea!

25. 12/25/2015Combinatorial Optimization Masters OR Starting Tableau e1e1 e2e2 e3e3 e4e4 e5e5 -z012231 s111000 a00110 b00 01 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5 To prepare basis { e 1, e 4, e 5 }: row2  row2 + row1;

26. 12/25/2015Combinatorial Optimization Masters OR Initial feasible bfs (1) e1e1 e2e2 e3e3 e4e4 e5e5 -z012231 s111000 a101110 b00 01 Basis { e 1, e 4, e 5 }: e 1 =1, e 4 =1, e 5 =0, this bfs is degenerate Now, to adjust reduced costs; First, row-0  row-0 - row-1 After we have row-2  row-1 + row-2; 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5

27. 12/25/2015Combinatorial Optimization Masters OR Initial feasible bfs (2) e1e1 e2e2 e3e3 e4e4 e5e5 -z01231 s111000 a101110 b00 01 After we have row 2  row 1 + row 2; After, row 0  row 0 - row 1; Then, row-0  row-0 – 3*(row-2); 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5

28. 12/25/2015Combinatorial Optimization Masters OR Initial feasible bfs (3) e1e1 e2e2 e3e3 e4e4 e5e5 -z-40-201 s111000 a101110 b00 01 After we have row 2  row 1 + row 2; Then, row 0  row 0 + row 1; Then, row 0  row 0 – 3*(row 2); Then, row-0  row-0 – row-3; 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5

29. 12/25/2015Combinatorial Optimization Masters OR Initial feasible bfs (4) e1e1 e2e2 e3e3 e4e4 e5e5 -z-40000 s111000 a101110 b00 01 After we have row 2  row 1 + row 2; Then, row 0  row 0 + row 1; Then, row 0  row 0 – 3*(row 2); Then, row 0  row 0 + *(row 3) Basis { e 1, e 4, e 5 }: e 1 =1, e 4 =1, e 5 =0, this bfs is degenerate 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5

30. 12/25/2015Combinatorial Optimization Masters OR Pivoting (1) e1e1 e2e2 e3e3 e4e4 e5e5 -z-40000 s111000 a101110 b00 01 Choose col with -ve relative-cost; (bring col e 2 into the basis) Choose pivot from that column; Now, perform: (a) row1  row1 – row2; (b) row3  row3 + row1; 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5

31. 12/25/2015Combinatorial Optimization Masters OR Pivoting (2) e1e1 e2e2 e3e3 e4e4 e5e5 -z-40000 s010 0 a101110 b100011 Now, to bring e 2 into the basis; After (a) and (b); Now, get reduced cost: (c) row0  row0 + row2; 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5

32. 12/25/2015Combinatorial Optimization Masters OR Pivoting (3) e1e1 e2e2 e3e3 e4e4 e5e5 -z-300110 s010 0 a101110 b100011 Now, to bring e 2 into the basis; After (a) and (b); And (c); This solution is optimal. All relative cost (row0) are positive! 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5

33. 12/25/2015Combinatorial Optimization Masters OR The dual of SP Max π s – π t s.t. π ´ A ≤ c ´. π free. In case of SP: the constraints of the dual are: π i – π j ≤ c ij. 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5

34. 12/25/2015Combinatorial Optimization Masters OR Complementary slackness for SP A path f as defined by primal variables, and an assignment of dual variables π are jointly optimal if and only if: 1.A primal variable with positive value corresponds to a dual constraint that is satisfied at equality. 2.A dual constraint not satisfied at equality corresponds to a primal variable with value zero.

35. 12/25/2015Combinatorial Optimization Masters OR Min e 1 + 2e 2 + 2e 3 + 3e 4 + e 5 s.t. e 1 + e 2 = 1 - e 4 - e 5 = -1 - e 1 + e 3 + e 4 = 0 - e 2 - e 3 + e 5 = 0 0 ≤ e i ≤ 1, for i =1…5 Min π s – π t s.t. π s - π a  1 π s - π b  2 π a - π b  2 π a - π t  3 π b - π t  1 each π i  0 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5 3 1 ? 0 1 2 2 3 1 s a b t e1e1 e2e2 e3e3 e4e4 e5e5 Compare dual variables π with label used in Dijkstra algorithm for shortest path!

36. 12/25/2015Combinatorial Optimization Masters OR Read up the following sections on your own. Sect 3.5: Dual Info in the Tableau Sect 3.6: Dual Simplex Algorithm Sect 3.7: Intrepretation of Dual Simplex Algorithm

37. 12/25/2015Combinatorial Optimization Masters OR Some Suggested Exercises: All from [PS82] Theorem 3.3 3 11 12 13 17

38. Hon Wai Leong, NUS (CS6234, Spring 2009) Page 38 Copyright © 2009 by Leong Hon Wai Thank you. Q & A