2. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 1 http://www.cs.cityu.edu.hk/~helena Greedy Algorithms

3. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 2 http://www.cs.cityu.edu.hk/~helena Greedy Algorithms Coming up Casual Introduction: Two Knapsack Problems An Activity-Selection Problem Greedy Algorithm Design Huffman Codes (Chap 16.1-16.3)

4. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 3 http://www.cs.cityu.edu.hk/~helena 2 Knapsack Problems A thief robbing a store finds n items. i th item: worth v i dollars w i pounds W, w i, v i are integers. He can carry at most W pounds. 1. 0-1 Knapsack Problem: Which items should I take?

5. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 4 http://www.cs.cityu.edu.hk/~helena 2 Knapsack Problems A thief robbing a store finds n items. i th item: worth v i dollars w i pounds W, w i, v i are integers. He can carry at most W pounds. He can take fractions of items. 2. Fractional Knapsack Problem: ?

6. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 5 http://www.cs.cityu.edu.hk/~helena 2 Knapsack Problems Dynamic Programming Solution If j th item is removed from his load, Both problems exhibit the optimal-substructure property: Consider the most valuable load that weighs at most W pounds. the remaining load must be the most valuable load weighting at most W-w j that he can take from the n-1 original items excluding j. => Can be solved by dynamic programming

7. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 6 http://www.cs.cityu.edu.hk/~helena 2 Knapsack Problems Dynamic Programming Solution Example: 0-1 Knapsack Problem Suppose there are n=100 ingots: 30 Gold ingots: each $10000, 8 pounds (most expensive) 20 Silver ingots: each $2000, 3 pound per piece 50 Copper ingots: each $500, 5 pound per piece Then, the most valuable load for to fill W pounds = The most valuable way among the followings: (1) take 1 gold ingot + the most valuable way to fill W-8 pounds from 29 gold ingots, 20 silver ingots and 50 copper ingots (2) take 1 silver ingot + the most valuable way to fill W-3 pounds from 30 gold ingots, 19 silver ingots and 50 copper ingots (3) take 1 copper ingot + the most valuable way to fill W-5 pounds from 30 gold ingots, 20 silver ingots and 49 copper ingots

8. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 7 http://www.cs.cityu.edu.hk/~helena 2 Knapsack Problems Dynamic Programming Solution Example: Fractional Knapsack Problem Suppose there are totally n = 100 pounds of metal dust: 30 pounds Gold dust: each pound $10000 (most expensive) 20 pounds Silver dust: each pound $2000 50 pounds Copper dust: each pound $500 Then, the most valuable way to fill a capacity of W pounds = The most valuable way among the followings: (1) take 1 pound of gold + the most valuable way to fill W-1 pounds from 29 pounds of gold, 20 pounds of silver, 50 pounds of copper (2) take 1 pound of silver + the most valuable way to fill W-1 pounds from 30 pounds of gold, 19 pounds of silver, 50 pounds of copper (3) take 1 pound copper + the most valuable way to fill W-1 pounds from 30 pounds of gold, 20 pounds of silver, 49 pounds of copper

9. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 8 http://www.cs.cityu.edu.hk/~helena 2 Knapsack Problems By Greedy Strategy Both problems are similar. But Fractional Knapsack Problem can be solved in a greedy strategy. Step 1.Compute the value per pound for each item Eg. gold dust: $10000 per pound (most expensive) Silver dust: $2000 per pound Copper dust: $500 per pound Step 2.Take as much as possible of the most expensive (ie. Gold dust) Step 3. If the supply of that item is exhausted (ie. no more gold) and he can still carry more, he takes as much as possible of the item that is next most expensive and so forth until he can’t carry any more.

10. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 9 http://www.cs.cityu.edu.hk/~helena Knapsack Problems By Greedy Strategy We can solve the Fractional Knapsack Problem by a greedy algorithm: Always makes the choice that looks best at the moment. ie. A locally optimal Choice To see why we can’t solve 0-1 Knapsack Problem by greedy strategy, read Chp 16.2.

11. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 10 http://www.cs.cityu.edu.hk/~helena Greedy Algorithms 2 techniques for solving optimization problems: 1. Dynamic Programming 2. Greedy Algorithms (“Greedy Strategy”) Greedy Approach can solve these problems: For the optimization problems: Dynamic Programming can solve these problems: For some optimization problems, Dynamic Programming is “overkill” Greedy Strategy is simpler and more efficient.

12. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 11 http://www.cs.cityu.edu.hk/~helena Activity-Selection Problem For a set of proposed activities that wish to use a lecture hall, select a maximum-size subset of “compatible activities”. Set of activities: S={a 1,a 2,…a n } Duration of activity a i :[start_time i, finish_time i ) Activities sorted in increasing order of finish time: i1234567891011 start_time i 130535688212 finish_time i 4567891011121314

13. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 12 http://www.cs.cityu.edu.hk/~helena Activity-Selection Problem i1234567891011 start_time i 130535688212 finish_time i 4567891011121314 Compatible activities: {a 3, a 9, a 11 }, {a 1,a 4,a 8,a 11 }, {a 2,a 4,a 9,a 11 }

14. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 13 http://www.cs.cityu.edu.hk/~helena Activity-Selection Problem Dynamic Programming Solution (Step 1) Step 1. Characterize the structure of an optimal solution. S: i1234567891011(=n) start_time i 130535688212 finish_time i 4567891011121314 eg Definition: S ij ={a k  S: finish_time i  start_time k <finish_time k  start_time j } Let S i,j be the set of activities that start after a i finishes and finish before a j starts. eg. S 2,11 =

15. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 14 http://www.cs.cityu.edu.hk/~helena Activity-Selection Problem Dynamic Programming Solution (Step 1) S: i1234567891011(=n) start_time i 130535688212 finish_time i 4567891011121314 Add fictitious activities: a 0 and a n+1 : S: i01234567891011 12 start_time i 130535688212  finish_time i 04567891011121314 ie. S 0,n+1 ={a 1,a 2,a 3,a 4,a 5,a 6,a 7,a 8,a 9,a 10,a 11 } = S Note: If i>=j then S i,j =Ø

16. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 15 http://www.cs.cityu.edu.hk/~helena Substructure: Activity-Selection Problem Dynamic Programming Solution (Step 1) Suppose a solution to S i,j includes activity a k, then,2 subproblems are generated: S i,k, S k,j The problem: For a set of proposed activities that wish to use a lecture hall, select a maximum- size subset of “compatible activities Select a maximum-size subset of compatible activities from S 0,n+1. = The maximum-size subset A i,j of compatible activities is: A i,j =A i,k U {a k } U A k,j Suppose a solution to S 0,n+1 contains a 7, then, 2 subproblems are generated: S 0,7 and S 7,n+1

17. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 16 http://www.cs.cityu.edu.hk/~helena Activity-Selection Problem Dynamic Programming Solution (Step 2) Step 2. Recursively define an optimal solution Let c[i,j] = number of activities in a maximum-size subset of compatible activities in S i,j. If i>=j, then S i,j =Ø, ie. c[i,j]=0. 0if S i,j =Ø Max i<k<j {c[i,k] + c[k,j] + 1} if S i,j  Ø c(i,j) = Step 3. Compute the value of an optimal solution in a bottom-up fashion Step 4. Construct an optimal solution from computed information.

18. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 17 http://www.cs.cityu.edu.hk/~helena Activity-Selection Problem Greedy Strategy Solution Consider any nonempty subproblem S i,j, and let a m be the activity in S i,j with the earliest finish time. eg. S 2,11 ={a 4,a 6,a 7,a 8,a 9 } Among {a 4,a 6,a 7,a 8,a 9 }, a 4 will finish earliest 1. A 4 is used in the solution 2. After choosing A 4, there are 2 subproblems: S 2,4 and S 4,11. But S 2,4 is empty. Only S 4, 11 remains as a subproblem. Then, 1. A m is used in some maximum- size subset of compatible activities of S i,j. 2.The subproblem S i,m is empty, so that choosing a m leaves the subproblem S m,j as the only one that may be nonempty. 0if S i,j =Ø Max i<k<j {c[i,k]+c[k,j]+1}if S i,j  Ø c(i,j) =

19. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 18 http://www.cs.cityu.edu.hk/~helena Activity-Selection Problem Greedy Strategy Solution That is, To solve S 0,12, we select a 1 that will finish earliest, and solve for S 1,12. To solve S 1,12, we select a 4 that will finish earliest, and solve for S 4,12. To solve S 4,12, we select a 8 that will finish earliest, and solve for S 8,12. … Greedy Choices (Locally optimal choice) To leave as much opportunity as possible for the remaining activities to be scheduled. Solve the problem in a top-down fashion Hence, to solve the S i,j : 1. Choose the activity a m with the earliest finish time. 2.Solution of S i,j = {a m } U Solution of subproblem S m,j

20. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 19 http://www.cs.cityu.edu.hk/~helena Recursive-Activity-Selector(i,j) 1m = i+1 // Find first activity in S i,j 2while m < j and start_time m < finish_time i 3do m = m + 1 4if m < j 5then return {a m } U Recursive-Activity-Selector(m,j) 6else return Ø Activity-Selection Problem Greedy Strategy Solution Order of calls: Recursive-Activity-Selector(0,12) Recursive-Activity-Selector(1,12) Recursive-Activity-Selector(4,12) Recursive-Activity-Selector(8,12) Recursive-Activity-Selector(11,12) m=2 Okay m=3 Okay m=4 break the loop Ø {11} {8,11} 4 {4,8,11} 4 {1,4,8,11}

21. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 20 http://www.cs.cityu.edu.hk/~helena Iterative-Activity-Selector() 1Answer = {a 1 } 2last_selected=1 3for m = 2 to n 4if start_time m >=finish_time last_selected 5then Answer = Answer U {a m } 6last_selected = m 7return Answer Activity-Selection Problem Greedy Strategy Solution

22. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 21 http://www.cs.cityu.edu.hk/~helena Activity-Selection Problem Greedy Strategy Solution For both Recursive-Activity-Selector and Iterative-Activity-Selector, Running times are  (n) Reason: each a m are examined once.

23. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 22 http://www.cs.cityu.edu.hk/~helena Greedy Algorithm Design Steps of Greedy Algorithm Design: 1. Formulate the optimization problem in the form: we make a choice and we are left with one subproblem to solve. 2.Show that the greedy choice can lead to an optimal solution, so that the greedy choice is always safe. 3.Demonstrate that an optimal solution to original problem = greedy choice + an optimal solution to the subproblem Optimal Substructure Property Greedy- Choice Property A good clue that that a greedy strategy will solve the problem.

24. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 23 http://www.cs.cityu.edu.hk/~helena Greedy Algorithm Design Comparison: Dynamic ProgrammingGreedy Algorithms At each step, the choice is determined based on solutions of subproblems. At each step, we quickly make a choice that currently looks best. --A local optimal (greedy) choice. Bottom-up approachTop-down approach Sub-problems are solved first.Greedy choice can be made first before solving further sub- problems. Can be slower, more complexUsually faster, simpler

25. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 24 http://www.cs.cityu.edu.hk/~helena Huffman Codes For compressing data (sequence of characters) Widely used Very efficient (saving 20-90%) Use a table to keep frequencies of occurrence of characters. Output binary string. “Today’s weather is nice” “001 0110 0 0 100 1000 1110”

26. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 25 http://www.cs.cityu.edu.hk/~helena Huffman Codes FrequencyFixed-lengthVariable-lengthcodeword ‘a’450000000 ‘b’13000001101 ‘c’12000010100 ‘d’16000011111 ‘e’90001001101 ‘f’50001011100 Example: A file of 100,000 characters. Containing only ‘a’ to ‘e’ 300,000 bits 1*45000 + 3*13000 + 3*12000 + 3*16000 + 4*9000 + 4*5000 = 224,000 bits eg. “abc” = “000001010” eg. “abc” = “0101100”

27. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 26 http://www.cs.cityu.edu.hk/~helena 01 a:45b:13c:12d:16e:9 f:5 0 0 0 0 0 1 1 1 1 Huffman Codes The coding schemes can be represented by trees: FrequencyFixed-length (in thousands)codeword ‘a’45000 ‘b’13001 ‘c’12010 ‘d’16011 ‘e’9100 ‘f’5101 100 86 14 01 58 28 a:45b:13c:12d:16e:9 f:5 0 0 0 0 0 1 1 1 1 FrequencyVariable-length (in thousands)codeword ‘a’450 ‘b’13101 ‘c’12100 ‘d’16111 ‘e’91101 ‘f’51100 100 55 01 25 30 0 0 0 1 1 1 a:45 14 e:9 f:5 0 1 d:16 b:13c:12 01 a:45b:13c:12d:16e:9 f:5 0 0 0 0 0 1 1 1 1 14 01 58 28 a:45b:13c:12d:16e:9 f:5 0 0 0 0 0 1 1 1 1 86 14 01 58 28 a:45b:13c:12d:16e:9 f:5 0 0 0 0 0 1 1 1 1 Not a full binary tree A full binary tree every nonleaf node has 2 children A file of 100,000 characters.

28. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 27 http://www.cs.cityu.edu.hk/~helena Huffman Codes Frequency Codeword ‘a’450000 ‘b’13000101 ‘c’12000100 ‘d’16000111 ‘e’90001101 ‘f’50001100 100 55 01 25 30 0 0 0 1 1 1 a:45 14 e:9 f:5 0 1 d:16 b:13c:12 To find an optimal code for a file: 1. The coding must be unambiguous. Consider codes in which no codeword is also a prefix of other codeword. => Prefix Codes Prefix Codes are unambiguous. Once the codewords are decided, it is easy to compress (encode) and decompress (decode). 2. File size must be smallest. => Can be represented by a full binary tree. => Usually less frequent characters are at bottom Let C be the alphabet (eg. C={‘a’,’b’,’c’,’d’,’e’,’f’}) For each character c, no. of bits to encode all c’s occurrences = freq c *depth c File size B(T) =  c  C freq c *depth c Eg. “abc” is coded as “0101100”

29. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 28 http://www.cs.cityu.edu.hk/~helena Huffman Codes Huffman code (1952) was invented to solve it. A Greedy Approach. Q: A min-priority queue f:5e:9c:12b:13d:16 a:45 100 55 25 30 a:45 14 e:9 f:5 d:16 b:13c:12 b:13d:16 a:45 14 f:5 e:9 d:16 a:45 14 25 c:12 b:13 30 f:5 e:9 a:45 d:16 14 25 c:12 b:13 30 55 f:5 e:9 d:16 a:45 14 25 c:12 b:13 f:5 e:9 How do we find the optimal prefix code?

30. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 29 http://www.cs.cityu.edu.hk/~helena Huffman Codes HUFFMAN(C) 1 Build Q from C 2 For i = 1 to |C|-1 3 Allocate a new node z 4z.left = x = EXTRACT_MIN(Q) 5z.right = y = EXTRACT_MIN(Q) 6z.freq = x.freq + y.freq 7Insert z into Q in correct position. 8 Return EXTRACT_MIN(Q) Q: A min-priority queue f:5e:9c:12b:13d:16 a:45 c:12b:13d:16 a:45 14 f:5 e:9 d:16 a:45 14 25 c:12 b:13 f:5 e:9 …. If Q is implemented as a binary min-heap, “Build Q from C” is O(n) “ EXTRACT_MIN (Q)” is O(lg n) “Insert z into Q” is O(lg n) Huffman(C) is O(n lg n) How is it “greedy”?

31. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 5. Greedy Algorithms - 30 http://www.cs.cityu.edu.hk/~helena Greedy Algorithms Summary Casual Introduction: Two Knapsack Problems An Activity-Selection Problem Greedy Algorithm Design Steps of Greedy Algorithm Design Optimal Substructure Property Greedy-Choice Property Comparison with Dynamic Programming Huffman Codes