2. 1 Network Flow & Linear Programming Jeff Edmonds York University COSC 3101 Lecture 5 Optimization Problems Network Flow Def n Network Flow Def n Simple Example Application: Matching Flow Strategy Hill Climbing Augmenting Flow Primal-Dual Hill Climbing Min Cut Max Flow = Min Cut Running Time Linear Programming

3. 2 Ingredients: Instances: The possible inputs to the problem. Solutions for Instance: Each instance has an exponentially large set of solutions. Cost of Solution: Each solution has an easy to compute cost or value. Specification : The input is one instance. : A valid solution with optimal cost. (minimum or maximum) Optimization Problems

4. 3 Instance: A Network is a directed graph G Edges represent pipes that carry flow Each edge has a maximum capacity c A source node s out of which flow leaves A sink node t into which flow arrives Goal: Max Flow Network Flow

5. 4 Instance: A Network is a directed graph G Edges represent pipes that carry flow Each edge has a maximum capacity c A source node s out of which flow leaves A sink node t into which flow arrives Network Flow

6. 5 Solution: The amount of flow F through each edge. Flow can’t exceed capacity i.e. F  c. Unidirectional flow F  0 and F = 0 or F = 0 and F  0 Some texts: F = -F Network Flow

7. 6 Solution: The amount of flow F through each edge. Flow F can’t exceed capacity c. Unidirectional flow No leaks, no extra flow. Network Flow

8. 7 Solution: The amount of flow F through each edge. Flow F can’t exceed capacity c. Unidirectional flow No leaks, no extra flow. For each node v: flow in = flow out  u F =  w F Except for s and t. Network Flow

9. 8 Value of Solution: Flow from s into the network minus flow from the network back into s. rate(F) =  u F = flow from network into t minus flow back in. =  u F -  v F -  v F What about flow back into s? Goal: Max Flow Network Flow

10. 9 A network with its edge capacities Network Flow What is the maximum that can flow from s to t?

11. 10 A network with its edge capacities Network Flow The max total rate of the flow is 1+2-0 = 3. flow/capacity = 2/5 Prove that total can not be higher.

12. 11 No more flow can be pushed along the top path because the edge is at capacity. Similarly, the edge. No flow is pushed along the bottom path because this would decrease the total from s to t. Network Flow

13. 12 is a minimum cut Its capacity is the sum of the capacities crossing the cut = 1+2 = 3. is not included in because it is going in the wrong direction. Network Flow

14. 13 The edges crossing forward across the cut are at capacity those crossing backwards have zero flow. This is always true. Network Flow

15. 14 The maximum flow is 1+2=3 The minimum cut is 1+2=3. These are always equal. Network Flow

16. 15 An Application: Matching SamMary BobBeth JohnSue FredAnn Who loves whom. Who should be matched with whom so as many as possible matched and nobody matched twice? 3 matches Can we do better? 4 matches

17. 16 An Application: Matching st c = 1 Total flow out of u  flow into u  1 Boy u matched to at most one girl. 1 c = 1 Total flow into v = flow out of v  1 Girl v matched to at most one boy. 1 u v

18. 17 Network Flow Strategy: Push flow into s. Must make decisions. Get stuck and must backtrack. Difficult and time consuming. s c=100 f=100 c=100 c=1 f=100

19. 18 Network Flow Strategy: Find a path for a single drop. Push as much flow through as fits. w = augment = Min  Path c

20. 19 Network Flow Strategy: Find a path for a single drop. Push as much flow through as fits. w = augment = Min  Path c flow/capacity = 20/21

21. 20 Network Flow Given Flow F Construct Augmenting Graph G F Find path P using BFS, DFS, or generic search algorithm Let w be the max amount flow can increase along path P. Increase flow along path P by w. i.e newF = oldF + w × P +w

22. 21 Network Flow Given Flow F Construct Augmenting Graph G F Find path P using BFS, DFS, or generic search algorithm No path Stop.

23. 22 Hill Climbing We have a valid solution. (not necessarily optimal) Take a step that goes up. measure progress Value of our solution. Problems: Exit Cant take a step that goes up. Running time? Initially have the “zero Local Max Global Max Can our Network Flow Algorithm get stuck in a local maximum? Make small local changes to your solution to construct a slightly better solution. If you take small step, could be exponential time.

24. 23 Network Flow Previous Input Previous Output Same Input Can our Network Flow Algorithm get stuck in local max? Worse Output Need only one example. Yes!

25. 24 Network Flow

26. 25 Network Flow Previous Input Previous Output Same Input Yes! Our Network Flow Algorithm can get stuck. Worse Output Need only one example.

27. 26 Hill Climbing Avoiding getting stuck in a local maximum Good ExecutionBad Execution Made better choices of direction Hard Back up an retry Exponential time Define a bigger step

28. 27 Hill Climbing Different Solutions Current Solution Alg defines to where alg can step i.e. what small local changes can be made to current solution This defines the topography

29. 28 Hill Climbing Different Solutions Current Solution Define a slightly bigger step This defines the topography Perhaps removes some local maxima

30. 29 Network Flow Previous Input Previous Output Same Input Worse Output Mistake? Putting 2 through this edge

31. 30 Network Flow Mistake? Putting 2 through this edge We need to decrease the flow in this edge. But if we decrease the total flow then the algorithm might run exponentially or forever. We need to decrease the flow in this edge AND increase the total flow.

32. 31 Network Flow How much flow can I have in each direction? u v 0/75 0/10 Where can I stand in each direction?

33. 32 Network Flow How much flow can I have in each direction? 75 to the right u v 75/75 0/10 Where can I stand in each direction?

34. 33 Network Flow How much flow can I have in each direction? 75 to the right u v 0/75 10/10 10 to the left Where can I stand in each direction?

35. 34 Network Flow u v 15/75 0/10 Add 5 flow to right Walking 5 u v 20/75 0/10 15 20

36. 35 Network Flow u v 15/75 0/10 Add 75-15=60 flow to right Walking 75-15=60 u v 75/75 0/10 15

37. 36 Network Flow u v 15/75 0/10 How much flow can I add to right How much can I change my position to the right? 75-15=60 15 u v 75-15=60 Augmentation Graph

38. 37 Network Flow u v f/c 0/10 How much flow can I add to right How much can I change my position to the right? 75-15=60 15 f-c u v Augmentation Graph

39. 38 Network Flow u v 15/75 0/10 Add 5 flow to left Walking 5 u v 15/75 5/10 Edge shouldn’t have flow in both directions. 15 10

40. 39 Network Flow u v 15/75 0/10 Add 5 flow to left Walking 5 u v 15-5=10/75 0/10 15 10

41. 40 Network Flow u v 15/75 0/10 Add 15 flow to left Walking 15 u v 15-15=0/75 0/10 15

42. 41 Network Flow u v 15/75 0/10 Add 15+5=20 flow to left Walking 15+5=20 u v -5/75 0/10 Edge shouldn’t have negative flow. 15 -5

43. 42 Network Flow u v 15/75 0/10 Add 15+5=20 flow to left Walking 15+5=20 u v 0/75 5/10 15 -5

44. 43 Network Flow u v 15/75 0/10 Add 15+10=25 flow to left Walking 15+10=25 u v 0/75 10/10 15

45. 44 Network Flow u v 15/75 0/10 How much flow can I add to left How much can I change my position to the left? 15+10=25 15 u v 15+10=25 Augmentation Graph

46. 45 Network Flow u v f/75 0/c How much flow can I add to left How much can I change my position to the left? 15+10=25 15 u v f+c Augmentation Graph

47. 46 u v F<u,v>/c<u,v>F<u,v>/c<u,v> 0/c Flow Graph u v Augmentation Graph F<u,v>+wF<u,v>+w w Walking c<u,v>-F<u,v>c<u,v>-F<u,v> F<u,v>+c<v,u>F<u,v>+c<v,u> c<u,v>c<u,v> F<u,v>F<u,v> c<v,u>c<v,u> Network Flow

48. 47 u v F<u,v>/c<u,v>F<u,v>/c<u,v> 0/c Flow Graph u v Augmentation Graph F<u,v>-wF<u,v>-w c<u,v>-F<u,v>c<u,v>-F<u,v> F<u,v>+c<v,u>F<u,v>+c<v,u> c<u,v>c<u,v> F<u,v>F<u,v> c<u,v>c<u,v> w Walking Network Flow

49. 48 Network Flow Given Flow F Construct Augmenting Graph G F Find path P Old New Where we got stuck before

50. 49 Network Flow Given Flow F Construct Augmenting Graph G F Find path P Let w be the max amount flow can increase along path P. Increase flow along path P by w. i.e newF = oldF + w × P +w -w ?

51. 50 Network Flow Given Flow F Construct Augmenting Graph G F Find path P using BFS, DFS, or generic search algorithm No path Stop.

52. 51 Network Flow Previous Input Previous Output Same Input Worse Output Same Output

53. 52

54. 53 An Application: Matching SamMary BobBeth JohnSue FredAnn Who loves whom. Who should be matched with whom so as many as possible matched and nobody matched twice? 3 matches Can we do better? 4 matches

55. 54 An Application: Matching st Flow st Augmentation Graph Augmentation Path Alternates adding edge removing edge adding edge removing edge adding edge Extra edge added st New Flow

56. 55 An Application: Matching SamMary BobBeth JohnSue FredAnn Who loves whom. Who should be matched with whom so as many as possible matched and nobody matched twice? 3 matches Can we do better? 4 matches

57. 56 Network Flow Can our Network Flow Algorithm get stuck in local max? Need to prove for every input network for every choice of augmenting paths Maximum Flow is found! No! How?

58. 57 Primal-Dual Hill Climbing Mars settlement has hilly landscape and many layers of roofs.

59. 58 Primal-Dual Hill Climbing Primal Problem: Exponential # of locations to stand. Find a highest one. Dual problem: Exponential # of roofs. Find a lowest one.

60. 59 Primal-Dual Hill Climbing Prove: Every roof is above every location to stand.  R  L height(R)  height(L)  height(R min )  height(L max ) Is there a gap?

61. 60 Primal-Dual Hill Climbing Prove: For every location to stand either: the alg takes a step up or the alg gives a reason that explains why not by giving a ceiling of equal height. i.e.  L [  L’ height(L’)  height(L) or  R height(R) = height(L)] or But  R  L height(R)  height(L) No Gap

62. 61 Primal-Dual Hill Climbing Prove: For every location to stand either: the alg takes a step up or the alg gives a reason that explains why not by giving a ceiling of equal height. i.e.  L [  L’ height(L’)  height(L) or  R height(R) = height(L)] or Cant go up from this location and no matching ceiling. Cant happen! ?

63. 62 Primal-Dual Hill Climbing Prove: For every location to stand either: the alg takes a step up or the alg gives a reason that explains why not by giving a ceiling of equal height. i.e.  L [  L’ height(L’)  height(L) or  R height(R) = height(L)] or No local maximum!

64. 63 Primal-Dual Hill Climbing Claim: Primal and dual have the same optimal value. height(R min ) = height(L max ) Proved:  R  L, height(R)  height(L) Proved: Alg runs until it provides L alg and R alg height(R alg ) = height(L alg ) No Gap height(R min )  height(R alg )  height(L alg )  height(L max ) height(R min )  height(L max ) L alg witness that height(L max ) is no smaller. R alg witness that height(L max ) is no bigger. Exit

65. 64 A network with its edge capacities Network Flow What is the maximum that can flow from s to t?

66. 65 A network with its edge capacities Network Flow The max total rate of the flow is 1+2-0 = 3. flow/capacity = 2/5 Prove that total can not be higher.

67. 66 No more flow can be pushed along the top path because the edge is at capacity. Similarly, the edge. No flow is pushed along the bottom path because this would decrease the total from s to t. Network Flow

68. 67 is a minimum cut Its capacity is the sum of the capacities crossing the cut = 1+2 = 3. is not included in because it is going in the wrong direction. Network Flow

69. 68 The edges crossing forward across the cut are at capacity those crossing backwards have zero flow. This is always true. Network Flow

70. 69 The maximum flow is 1+2=3 The minimum cut is 1+2=3. These are always equal. Network Flow

71. 70 Primal-Dual Network Flow Primal Problem: Max Flow Dual Problem: Min Cut What are the roofs to the flows?

72. 71 Instance: A Network is a directed graph G Special nodes s and t. Edges represent pipes that carry flow Each edge has a maximum capacity c Min Cut s t

73. 72 Instance: A Network is a directed graph G Special nodes s and t. Edges represent pipes that carry flow Each edge has a maximum capacity c Min Cut

74. 73 Solution: C = partition of nodes with s  U, t  V. Min Cut s t U V York UC Berkeley = Canada = USA

75. 74 Min Cut s t York UC Berkeley Ontario Solution: C = partition of nodes with s  U, t  V.

76. 75 Min Cut s t York UC Berkeley Toronto Solution: C = partition of nodes with s  U, t  V.

77. 76 Min Cut s t York UC Berkeley York Solution: C = partition of nodes with s  U, t  V.

78. 77 Min Cut s t York UC Berkeley UCB Solution: C = partition of nodes with s  U, t  V.

79. 78 Min Cut s t York UC Berkeley Berkeley Solution: C = partition of nodes with s  U, t  V.

80. 79 Min Cut s t York UC Berkeley U V Solution: C = partition of nodes with s  U, t  V.

81. 80 Value Solution C= : cap(C) = how much can flow from U to V =  u  U,v  V c Min Cut s t U V u v Goal: Min Cut

82. 81 Theorem: For all Networks Max F rate(F) = Min C cap(C) Prove:  F,C rate(F)  cap(C) Max Flow = Min Cut U V u v Prove:  flow F, alg either finds a better flow F or finds cut C such that rate(F) = cap(C) Alg stops with an F and C for which rate(F) = cap(C) F witnesses that the optimal flow cant be less C witnesses that it cant be more. Exit

83. 82 cap(C) = how much can flow from U to V =  u  U,v  V c rate(F,C) = Flow from U to V minus flow V to U =  u  U,v  V F - F Lemma: rate(F) = rate(F,C)  cap(C) Max Flow = Min Cut Prove:  flow F, cut C rate(F)  cap(C) s t U V York UC Berkeley = Canada = USA + - F  c & F  0 No leaks, no extra flow.

84. 83 Lemma:  F,C rate(F,C) = rate(F) Proof: By induction on the size of U. Base case: C = rate(F) = Flow from s into the network minus flow from the network back into s  =  u F -  v F  = rate(F,C) where C = Max Flow = Min Cut s t York UC Berkeley + -

85. 84 Lemma:  F,C rate(F,C) = rate(F) Proof: By induction on the size of C. Base case: C = Inductive step: Move nodes across C one at a time. Max Flow = Min Cut Flow into x = Flow out. Hence, flow across cut does not change. rate(F) = rate(F,C)  cap(C)

86. 85 Theorem: For all Networks Max F rate(F) = Min C cap(C) Prove:  F,C rate(F)  cap(C) Max Flow = Min Cut U V u v Prove:  flow F, alg either finds a better flow F or finds cut C such that rate(F) = cap(C)

87. 86 Given Flow F Construct Augmenting Graph G F Find path P Let w be the max amount flow can increase along path P. Increase flow along path P by w. i.e newF = oldF + w × P +w -w Max Flow = Min Cut

88. 87 Given Flow F Construct Augmenting Graph G F Find path P using BFS, DFS, or generic search algorithm No path Stop Max Flow = Min Cut

89. 88 Let F alg be this final flow. Let cut C alg =, where U are the nodes reachable from s in the augmented graph and V not. Claim: rate(F alg ) = cap(C alg ) Max Flow = Min Cut Bad example because U might contain more than s.

90. 89 Max Flow = Min Cut Prove: rate(F alg ) = cap(C alg ) Prove: rate(F alg,C alg ) = cap(C alg ) =

91. 90 cap(C) = how much can flow from U to V =  u  U,v  V c rate(F,C) = Flow from U to V minus flow V to U =  u  U,v  V F - F Max Flow = Min Cut Prove: rate(F alg,C alg ) = cap(C alg ) + F /c u v F<u,v>/c<u,v>F<u,v>/c<u,v> Flow Graph U V v’u’ -

92. 91 Max Flow = Min Cut F /c u v F<u,v>/c<u,v>F<u,v>/c<u,v> Flow Graph U V v’u’ need equal need zero + - cap(C) = how much can flow from U to V =  u  U,v  V c rate(F,C) = Flow from U to V minus flow V to U =  u  U,v  V F - F Prove: rate(F alg,C alg ) = cap(C alg )

93. 92 Max Flow = Min Cut u v F<u,v>/c<u,v>F<u,v>/c<u,v> Flow Graph c<u,v>-F<u,v>c<u,v>-F<u,v> F +c U V v’u’ F /c u Augmentation Graph U V v’ v u’ reachable from s = = not not edges =0 need equal + need zero - cap(C) = how much can flow from U to V =  u  U,v  V c rate(F,C) = Flow from U to V minus flow V to U =  u  U,v  V F - F Prove: rate(F alg,C alg ) = cap(C alg )

94. 93 Theorem: For all Networks Max F rate(F) = Min C cap(C) Prove:  F,C rate(F)  cap(C) Max Flow = Min Cut U V u v Prove:  flow F, alg either finds a better flow F or finds cut C such that rate(F) = cap(C) Alg stops with an F and C for which rate(F) = cap(C) F witnesses that the optimal flow cant be less C witnesses that it cant be more.

95. 94 Hill Climbing Problems: Can our Network Flow Algorithm get stuck in a local maximum? Local Max Global Max No!

96. 95 Hill Climbing Problems: Running time? If you take small step, could be exponential time.

97. 96 Network Flow

98. 97 Network Flow Add flow 1

99. 98 Network Flow Add flow 1

100. 99 Hill Climbing Problems: Running time? If each iteration you take the biggest step possible, Alg is poly time in number of nodes and number of bits in capacities. If each iteration you take path with the fewest edges Alg is poly time in number of nodes

101. 100 Taking the biggest step possible

102. 101 Linear Programming

103. 102 A Hotdog A combination of pork, grain, and sawdust, …

104. 103 Constraints: Amount of moistureAmount of moisture Amount of protein,Amount of protein, …

105. 104 The Hotdog Problem Given today’s prices, what is a fast algorithm to find the cheapest hotdog?

106. 105 Abstract Out Essential Details Cost: 29, 8, 1, 2 Amount to add: x 1, x 2, x 3, x 4 pork grainwater sawdust 3x 1 + 4x 2 – 7x 3 + 8x 4 ³ 12 2x 1 - 8x 2 + 4x 3 - 3x 4 ³ 24 -8x 1 + 2x 2 – 3x 3 - 9x 4 ³ 8 x 1 + 2x 2 + 9x 3 - 3x 4 ³ 31 Constraints: moisturemoisture protean,protean, … 29x 1 + 8x 2 + 1x 3 + 2x 4 Cost of Hotdog:

107. 106 3x 1 + 4x 2 – 7x 3 + 8x 4 ³ 12 2x 1 - 8x 2 + 4x 3 - 3x 4 ³ 24 -8x 1 + 2x 2 – 3x 3 - 9x 4 ³ 8 x 1 + 2x 2 + 9x 3 - 3x 4 ³ 31 29x 1 + 8x 2 + 1x 3 + 2x 4 Subject to: Minimize: Abstract Out Essential Details

108. 107 A Fast Algorithm For decades people thought that there was no fast algorithm. Then one was found! Theoretical Computer Science finds new algorithms every day. 3x 1 + 4x 2 – 7x 3 + 8x 4 ³ 12 2x 1 - 8x 2 + 4x 3 - 3x 4 ³ 24 -8x 1 + 2x 2 – 3x 3 - 9x 4 ³ 8 x 1 + 2x 2 + 9x 3 - 3x 4 ³ 31 29x 1 + 8x 2 + 1x 3 + 2x 4 Subject to: Minimize: 

109. 108 Network Flow as a Linear Program Given an instance of Network Flow: > express it as a Linear Program: The variables: Maximize: Subject to: Flows f for each edge.  : F  c. (Flow cant exceed capacity)  v:  u F =  w F (flow in = flow out) rate(F) =  u F -  v F

110. 109

111. 110 Primal Dual

112. 111 End

113. 112 Network Flow Locally in each edge, we see how much we can change the flow. Allowed Flow ≠ Allowed Change in Flow People find this hard. This is my first attempt to explain it.

114. 113 Network Flow u v 0/75 0/10 Old Flow Graph Walking 10 u v 10/75 0/10 New Flow Graph

115. 114 Network Flow u v 0/75 0/10 Old Flow Graph Walking 10 u v 0/75 10/10 New Flow Graph

116. 115 Network Flow u v 0/75 0/10 Old Flow Graph Walking -10 u v -10/75 0/10 New Flow Graph Same Walking 10 u v 0/75 10/10 New Flow Graph Not allowed

117. 116 Network Flow Walking -10 u v -10/75 0/10 New Flow Graph Same Walking 10 u v 0/75 10/10 New Flow Graph Not allowed u v F  [-10,75] Eqv Flow Graph F = -10

118. 117 Network Flow 10+Δ ≤ 75 F +Δ ≤ c Δ ≤ c -F Walking Δ u v 10+ Δ /75 0/10 New Flow Graph u v F  [-10,75] Eqv Flow Graph F = 10

119. 118 Network Flow 10-Δ ≥ -10 F -Δ ≥ -c Δ ≤ F + c u v 10- Δ /75 0/10 New Flow Graph Walking Δ u v F  [-10,75] Eqv Flow Graph F = 10

120. 119 Network Flow Δ ≤ F + c u v F  [-10,75] Eqv Flow Graph F = 10 Δ ≤ c -F u v Augmentation Graph

121. 120 Network Flow Locally in each edge, we see how much we can change the flow. Allowed Flow ≠ Allowed Change in Flow People find this hard. This is my second attempt to explain it.

122. 121 Network Flow u v 0/75 0/10 Flow Graph u v 75 10 Augmentation Graph

123. 122 Network Flow 75-21=54 u v 21/75 0/10 Flow Graph u v Augmentation Graph 2 21+2=23 Walking

124. 123 Network Flow 75-21=54 u v 21/75 0/10 Flow Graph u v Augmentation Graph 10 21-10=11 10 Edge shouldn’t have flow in both directions. 0 10 ? Walking 10 Walking -10

125. 124 Network Flow 75-21=54 u v 11/75 0/10 Flow Graph u v Augmentation Graph 10 ? Capacity of 10 not met

126. 125 Network Flow 75-21=54 u v 21/75 0/10 Flow Graph u v Augmentation Graph 21 10 ? Capacity of 10 exceeded 21

127. 126 Network Flow 75-21=54 u v 21/75 0/10 Flow Graph u v Augmentation Graph 21 21-21=0 10 ? Capacity of 10 not met Walking 21 Walking -21

128. 127 Network Flow 75-21=54 u v 21/75 0/10 Flow Graph u v Augmentation Graph 21+10 21-21=0 10 ? 21+10=31 Capacity of 10 met Walking 21 and another 10

129. 128 Network Flow 75-21=54 21+10=31 u v 21/75 0/10 Flow Graph u v Augmentation Graph 25 21-25=-4 4 0 Walking 25

130. 129 u v F<u,v>/c<u,v>F<u,v>/c<u,v> 0/c Flow Graph u v Augmentation Graph F<u,v>+wF<u,v>+w w Walking c<u,v>-F<u,v>c<u,v>-F<u,v> F<u,v>+c<v,u>F<u,v>+c<v,u> c<u,v>c<u,v> F<u,v>F<u,v> c<v,u>c<v,u> Network Flow

131. 130 u v F<u,v>/c<u,v>F<u,v>/c<u,v> 0/c Flow Graph u v Augmentation Graph F<u,v>-wF<u,v>-w c<u,v>-F<u,v>c<u,v>-F<u,v> F<u,v>+c<v,u>F<u,v>+c<v,u> c<u,v>c<u,v> F<u,v>F<u,v> c<u,v>c<u,v> w Walking Network Flow