1. Greedy algorithms David Kauchak cs161 Summer 2009

2. Administrative Thank your TAs

3. Kruskal’s and Prim’s Make greedy decision about the next edge to add

4. Greedy algorithm structure A locally optimal decision can be made which results in a subproblem that does not rely on the local decision

5. Interval scheduling Given n activities A = [a 1,a 2,.., a n ] where each activity has start time s i and a finish time f i. Schedule as many as possible of these activities such that they don’t conflict.

6. Interval scheduling Given n activities A = [a 1,a 2,.., a n ] where each activity has start time s i and a finish time f i. Schedule as many as possible of these activities such that they don’t conflict. Which activities conflict?

7. Interval scheduling Given n activities A = [a 1,a 2,.., a n ] where each activity has start time s i and a finish time f i. Schedule as many as possible of these activities such that they don’t conflict. Which activities conflict?

8. Simple recursive solution Enumerate all possible solutions and find which schedules the most activities

9. Simple recursive solution Is it correct? max{all possible solutions} Running time? O(n!)

10. Can we do better? Dynamic programming O(n 2 ) Greedy solution – Is there a way to make a local decision?

11. Overview of a greedy approach Greedily pick an activity to schedule Add that activity to the answer Remove that activity and all conflicting activities. Call this A’. Repeat on A’ until A’ is empty

12. Greedy options Select the activity that starts the earliest, i.e. argmin{s 1, s 2, s 3, …, s n }?

13. Greedy options Select the activity that starts the earliest? non-optimal

14. Greedy options Select the shortest activity, i.e. argmin{f 1 -s 1, f 2 -s 2, f 3 -s 3, …, f n -s n }

15. Greedy options Select the shortest activity, i.e. argmin{f 1 -s 1, f 2 -s 2, f 3 -s 3, …, f n -s n } non-optimal

16. Greedy options Select the activity with the smallest number of conflicts

17. Greedy options Select the activity with the smallest number of conflicts

18. Greedy options Select the activity with the smallest number of conflicts

19. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

20. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

21. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

22. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

23. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

24. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

25. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

26. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

27. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

28. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }? Multiple optimal solutions

29. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

30. Greedy options Select the activity that ends the earliest, i.e. argmin{f 1, f 2, f 3, …, f n }?

31. Efficient greedy algorithm Once you’ve identified a reasonable greedy heuristic: Prove that it always gives the correct answer Develop an efficient solution

32. Is our greedy approach correct? “Stays ahead” argument: show that no matter what other solution someone provides you, the solution provided by your algorithm always “stays ahead”, in that no other choice could do better

33. Is our greedy approach correct? “Stays ahead” argument Let r 1, r 2, r 3, …, r k be the solution found by our approach Let o 1, o 2, o 3, …, o k of another optimal solution Show our approach “stays ahead” of any other solution … r1r1 r2r2 r3r3 rkrk o1o1 o2o2 o3o3 okok …

34. Stays ahead … r1r1 r2r2 r3r3 rkrk o1o1 o2o2 o3o3 okok … Compare first activities of each solution

35. Stays ahead … r1r1 r2r2 r3r3 rkrk o1o1 o2o2 o3o3 okok … finish(r 1 ) ≤ finish(o 1 )

36. Stays ahead … r2r2 r3r3 rkrk o2o2 o3o3 okok … We have at least as much time as any other solution to schedule the remaining 2…k tasks

37. An efficient solution

38. Running time? Θ(n log n) Θ(n) Overall: Θ(n log n) Better than: O(n!) O(n 2 )

39. Scheduling all intervals Given n activities, we need to schedule all activities. Minimize the number of resources required.

40. Greedy approach? The best we could ever do is the maximum number of conflicts for any time period

41. Calculating max conflicts efficiently 3

42. 1

43. 3

44. 1

45. …

46. Calculating max conflicts

47. Correctness? We can do no better then the max number of conflicts. This exactly counts the max number of conflicts.

48. Runtime? O(2n log 2n + n) = O(n log n)

49. Horn formulas Horn formulas are a particular form of boolean logic formulas They are one approach to allow a program to do logical reasoning Boolean variables: represent some event x = the murder took place in the kitchen y = the butler is innocent z = the colonel was asleep at 8 pm

50. Implications Left-hand side is an AND of any number of positive literals Right-hand side is a single literal x = the murder took place in the kitchen y = the butler is innocent z = the colonel was asleep at 8 pm If the colonel was asleep at 8 pm and the butler is innocent then the murder took place in the kitchen

51. Implications Left-hand side is an AND of any number of positive literals Right-hand side is a single literal x = the murder took place in the kitchen y = the butler is innocent z = the colonel was asleep at 8 pm the murder took place in the kitchen

52. Negative clauses An OR of any number of negative literals u = the constable is innocent t = the colonel is innocent y = the butler is innocent not every one is innocent

53. Goal Given a horn formula (i.e. set of implications and negative clauses), determine if the formula is satisfiable (i.e. an assignment of true/false that is consistent with all of the formula) u x y z 0 1 1 0

54. Goal Given a horn formula (i.e. set of implications and negative clauses), determine if the formula is satisfiable (i.e. an assignment of true/false that is consistent with all of the formula) u x y z not satifiable

55. Goal Given a horn formula (i.e. set of implications and negative clauses), determine if the formula is satisfiable (i.e. an assignment of true/false that is consistent with all of the formula) ?

56. Goal Given a horn formula (i.e. set of implications and negative clauses), determine if the formula is satisfiable (i.e. an assignment of true/false that is consistent with all of the formula) implications tell us to set some variables to true negative clauses encourage us make them false

57. A brute force solution Try each setting of the boolean variables and see if any of them satisfy the formula For n variables, how many settings are there? 2 n

58. A greedy solution? w 0 x 0 y 0 z 0

59. A greedy solution? w 0 x 1 y 0 z 0

60. A greedy solution? w 0 x 1 y 1 z 0

61. A greedy solution? w 1 x 1 y 1 z 0

62. A greedy solution? w 1 x 1 y 1 z 0 not satisfiable

63. A greedy solution

64. set all variables of the implications of the form “  x” to true

65. A greedy solution if the all variables of the lhs of an implication are true, then set the rhs variable to true

66. A greedy solution see if all of the negative clauses are satisfied

67. Correctness of greedy solution Two parts: If our algorithm returns an assignment, is it a valid assignment? If our algorithm does not return an assignment, does an assignment exist?

68. Correctness of greedy solution If our algorithm returns an assignment, is it a valid assignment?

69. Correctness of greedy solution If our algorithm returns an assignment, is it a valid assignment? explicitly check all negative clauses

70. Correctness of greedy solution If our algorithm returns an assignment, is it a valid assignment? don’t stop until all implications with all lhs elements true have rhs true

71. Correctness of greedy solution If our algorithm does not return an assignment, does an assignment exist? Our algorithm is “stingy”. It only sets those variables that have to be true. All others remain false.

72. Running time? ?

73. O(nm) n = number of variables m = number of formulas

74. Knapsack problems: Greedy or not? 0-1 Knapsack – A thief robbing a store finds n items worth v 1, v 2,.., v n dollars and weight w 1, w 2, …, w n pounds, where v i and w i are integers. The thief can carry at most W pounds in the knapsack. Which items should the thief take if he wants to maximize value. Fractional knapsack problem – Same as above, but the thief happens to be at the bulk section of the store and can carry fractional portions of the items. For example, the thief could take 20% of item i for a weight of 0.2w i and a value of 0.2v i.

75. Data compression Given a file containing some data of a fixed alphabet Σ (e.g. A, B, C, D), we would like to pick a binary character code that minimizes the number of bits required to represent the data. A C A D A A D B …0010100100100 … minimize the size of the encoded file

76. Compression algorithms http://en.wikipedia.org/wiki/Data_compression

77. Simplifying assumption over general compression? Given a file containing some data of a fixed alphabet Σ (e.g. A, B, C, D), we would like to pick a binary character code that minimizes the number of bits required to represent the data. A C A D A A D B …0010100100100 … minimize the size of the encoded file

78. Frequency only The problem formulation makes the simplifying assumption that we only have character frequency information for a file A C A D A A D B … = SymbolFrequency ABCDABCD 70 3 20 37

79. Fixed length code Use ceil(log 2 |Σ|) bits for each character A = B = C = D =

80. Fixed length code Use ceil(log 2 |Σ|) bits for each character A = 00 B = 01 C = 10 D = 11 SymbolFrequency ABCDABCD 70 3 20 37 How many bits to encode the file? 2 x 70 + 2 x 3 + 2 x 20 + 2 x 37 = 260 bits

81. Fixed length code Use ceil(log 2 |Σ|) bits for each character A = 00 B = 01 C = 10 D = 11 SymbolFrequency ABCDABCD 70 3 20 37 Can we do better? 2 x 70 + 2 x 3 + 2 x 20 + 2 x 37 = 260 bits

82. Variable length code What about: A = 0 B = 01 C = 10 D = 1 SymbolFrequency ABCDABCD 70 3 20 37 1 x 70 + 2 x 3 + 2 x 20 + 1 x 37 = 173 bits

83. Decoding a file A = 0 B = 01 C = 10 D = 1 010100011010 What characters does this sequence represent?

84. Decoding a file A = 0 B = 01 C = 10 D = 1 010100011010 What characters does this sequence represent? A D or B?

85. Variable length code What about: A = 0 B = 100 C = 101 D = 11 SymbolFrequency ABCDABCD 70 3 20 37 How many bits to encode the file? 1 x 70 + 3 x 3 + 3 x 20 + 2 x 37 = 213 bits (18% reduction)

86. Prefix codes A prefix code is a set of codes where no codeword is a prefix of some other codeword A = 0 B = 100 C = 101 D = 11 A = 0 B = 01 C = 10 D = 1

87. Prefix tree We can encode a prefix code using a full binary tree where each child represents an encoding of a symbol A = 0 B = 100 C = 101 D = 11 A BC D 0 1

88. Decoding using a prefix tree To decode, we traverse the graph until a leaf node is reached and output the symbol A = 0 B = 100 C = 101 D = 11 A BC D 0 1

89. Decoding using a prefix tree Traverse the graph until a leaf node is reached and output the symbol A BC D 0 1 1000111010100

90. Decoding using a prefix tree Traverse the graph until a leaf node is reached and output the symbol A BC D 0 1 1000111010100 B

91. Decoding using a prefix tree Traverse the graph until a leaf node is reached and output the symbol A BC D 0 1 1000111010100 B A

92. Decoding using a prefix tree Traverse the graph until a leaf node is reached and output the symbol A BC D 0 1 1000111010100 B A D

93. Decoding using a prefix tree Traverse the graph until a leaf node is reached and output the symbol A BC D 0 1 1000111010100 B A D C

94. Decoding using a prefix tree Traverse the graph until a leaf node is reached and output the symbol A BC D 0 1 1000111010100 B A D C A

95. Decoding using a prefix tree Traverse the graph until a leaf node is reached and output the symbol A BC D 0 1 1000111010100 B A D C A B

96. Determining the cost of a file A BC D 0 1 SymbolFrequency ABCDABCD 70 3 20 37

97. Determining the cost of a file A BC D 0 1 SymbolFrequency ABCDABCD 70 3 20 37 70 320 37

98. Determining the cost of a file A BC D 0 1 SymbolFrequency ABCDABCD 70 3 20 37 70 320 37 23 60 What if we label the internal nodes with the sum of the children?

99. Determining the cost of a file A BC D 0 1 SymbolFrequency ABCDABCD 70 3 20 37 70 320 37 23 60 Cost is equal to the sum of the internal nodes and the leaf nodes

100. Determining the cost of a file A BC D 0 1 70 320 37 23 60 60 times we see a prefix that starts with a 1 of those, 37 times we see an additional 1 the remaining 23 times we see an additional 0 70 times we see a 0 by itself of these, 20 times we see a last 1 and 3 times a last 0 As we move down the tree, one bit gets read for every nonroot node

101. A greedy algorithm? Given file frequencies, can we come up with a prefix-free encoding (i.e. build a prefix tree) that minimizes the number of bits?

102. SymbolFrequency ABCDABCD 70 3 20 37 Heap

103. SymbolFrequency ABCDABCD 70 3 20 37 Heap B3 C 20 D37 A70

104. SymbolFrequency ABCDABCD 70 3 20 37 Heap BC23 D37 A70 BC 320 23 merging with this node will incur an additional cost of 23

105. SymbolFrequency ABCDABCD 70 3 20 37 Heap BCD 60 A 70 BC 320 23 D 37 60

106. SymbolFrequency ABCDABCD 70 3 20 37 Heap ABCD 130 BC 320 23 D 37 60 A 70

107. Is it correct? The algorithm selects the symbols with the two smallest frequencies first (call them f 1 and f 2 ) Consider a tree that did not do this: f1f1 fifi f2f2

108. Is it correct? The algorithm selects the symbols with the two smallest frequencies first (call them f 1 and f 2 ) Consider a tree that did not do this: f1f1 fifi f2f2 fifi f1f1 f2f2 - frequencies don’t change - cost will decrease since f 1 < f i contradiction

109. Runtime? 1 call to MakeHeap 2(n-1) calls ExtractMin n-1 calls Insert O(n log n)

110. Non-optimal greedy algorithms All the greedy algorithms we’ve looked at today give the optimal answer Some of the most common greedy algorithms generate good, but non-optimal solutions set cover clustering hill-climbing relaxation