1. Greedy Algorithms

2. A short list of categories
Algorithm types we will consider include:
Simple recursive algorithms
Backtracking algorithms
Divide and conquer algorithms
Dynamic programming algorithms
Greedy algorithms
Branch and bound algorithms
Brute force algorithms
Randomized algorithms

3. Optimization problems
An optimization problem is one in which you want to find, not just a solution, but the best solution
A “greedy algorithm” sometimes works well for optimization problems
A greedy algorithm works in phases. At each phase:
You take the best you can get right now, without regard for future consequences
You hope that by choosing a local optimum at each step, you will end up at a global optimum 2

4. Example: Counting money
Suppose you want to count out a certain amount of money, using the fewest possible bills and coins
A greedy algorithm would do this would be: At each step, take the largest possible bill or coin that does not overshoot
Example: To make $6.39, you can choose:
a $5 bill
a $1 bill, to make $6
a 25¢ coin, to make $6.25
A 10¢ coin, to make $6.35
four 1¢ coins, to make $6.39
For US money, the greedy algorithm always gives the optimum solution 3

5. A failure of the greedy algorithm
In some (fictional) monetary system, “krons” come in 1 kron, 7 kron, and 10 kron coins
Using a greedy algorithm to count out 15 krons, you would get
A 10 kron piece
Five 1 kron pieces, for a total of 15 krons
This requires six coins
A better solution would be to use two 7 kron pieces and one 1 kron piece
This only requires three coins
The greedy algorithm results in a solution, but not in an optimal solution 4

6. A scheduling problem
You have to run nine jobs, with running times of 3, 5, 6, 10, 11, 14, 15, 18, and 20 minutes
You have three processors on which you can run these jobs
You decide to do the longest-running jobs first, on whatever processor is available P1 P2 P3 20 10 3 18 11 6 15 14 5
Time to completion: = 35 minutes
This solution isn’t bad, but we might be able to do better 5

7. Another approach
What would be the result if you ran the shortest job first?
Again, the running times are 3, 5, 6, 10, 11, 14, 15, 18, and 20 minutes P1 P2 P3 3 10 15 5 11 18 6 14 20
That wasn’t such a good idea; time to completion is now = 40 minutes
Note, however, that the greedy algorithm itself is fast
All we had to do at each stage was pick the minimum or maximum 6

8. An optimum solution Better solutions do exist: 20 18 15 14 11 10 6 5 3
This solution is clearly optimal (why?)
Clearly, there are other optimal solutions (why?)
How do we find such a solution?
One way: Try all possible assignments of jobs to processors
Unfortunately, this approach can take exponential time 7

9. Huffman Compression
Compression: We want to store a text file using the least number of bits possible
How?
Fixed Width: encode every character with exactly b bits.
Develop a unique bit encoding for each character based on the frequency of each character
Common characters have shorter encoding….

10. Huffman Compression
Compression: We want to store a text file using the least number of bits possible
Fixed Width: encode every character with exactly b bits.
For a message of n characters over an alphabet of size k: total length will be nlog2k bits

11. Sample: Fixed Width String: abacadae encoding: a: 101 b: 110 c: 010

12. Huffman Compression
Compression: We want to store a text file using the least number of bits possible
How to do better?
Develop a unique bit encoding for each character based on the frequency of each character
Common characters have shorter encoding….
Harder to decode
Better than nlog2k

13. Sample: Variable Width
This is an ambiguous code
String: abacadae
encoding:
a: 1
b: 10
c: 01
d: 00
e: 11
eccbce

14. Prefix Codes
A prefix code is a variable length coding such that no character code is a prefix for any other.
In other words: no code begins with another code.

15. Not A Prefix Code String: abacadae encoding: a: 1 b: 10 c: 01

16. Prefix Code String: abacadae Unambiguous encoding: a: 1 b: 010 c: 001

17. Prefix Code String: abacadae encoding: a: 1 b: 010 c: 001 d: 000
Optimal Fixed width scheme:
8(3) = 24 bits
This variable width scheme:
4(1) + 4(3) = 16 bits

18. Tree Representation 1 A 1
A: 1
B: 011
C: 010
D: 001
E: 000 1 1 E D C B

19. Character Frequency
Given a file, we can measure the frequency of a character x as the fraction of characters in the file that are equal to x -- denoted f(x).
File: adbacaedcd
= {a,b,c,d,e}
x
f(x) a
3/10 = 0.3 b
1/10 = 0.1 c
2/10 = 0.2 d e
Notice: We can assume f(x) > 0

20. Code Length
We can see the length of the encoding by the depth of the node
(x)= node assigned to symbol x
d(w) = depth of node w E A
x
d((x)) a 2 b 4 c d 3 e 1 D B C

21. Compression Length E A D B C File: adbacaedcd Total length:
x
d((x))
f(x) a 2
0.3 b 4
0.1 c
0.2 d 3 e 1 E A
Total length:
10(0.3)*2 +
10(0.1)*4 +
10(0.2)*4 +
10(0.3)*3 +
10(0.1)*1 = 28 bits D B C

22. Compression Length A E D B C File: adbacaedcd Total length:
x
d((x))
f(x) a 1
0.3 b 4
0.1 c
0.2 d 3 e 2 A E
Total length:
10(0.3)*1 +
10(0.1)*4 +
10(0.2)*4 +
10(0.3)*3 +
10(0.1)*2 = 26 bits D B C

23. Compression Length A C D B E File: adbacaedcd Total length:
x
d((x))
f(x) a 2
0.3 b 3
0.1 c
0.2 d e A C D
Total length:
10(0.3)*2 +
10(0.1)*3 +
10(0.2)*3 +
10(0.1)*3 = 24 bits B E

24. ABL(T) Given: A message M of n characters
Goal: Pick a tree to minimize the encoding size:
ES(T) = xnf(x)d((x))
Equivalent goal: Minimize the average bit length of the encodings:
ABL(T) = ES(T) / n
= 1/n (xnf(x)d((x)))
= xf(x)d((x))

25. Compression Length A C D B E File: adbacaedcd Average Bit Length:
x
d((x))
f(x) a 2
0.3 b 3
0.1 c
0.2 d e A C D
Average Bit Length:
(0.3)*2 +
(0.1)*3 +
(0.2)*3 +
(0.1)*3 = 2.4 bits B E

26. Optimal Prefix Code Consider a sequence M such that:
M contains characters from an alphabet 
For x  , f(x) is the number of characters in M that are x
An optimal prefix code is one that corresponds to a prefix tree T such that ABL(T) is as small as possible (minimized)
Or: the average bit length of encoding M is minimized
Or: the compressed length of M is minimized

27. Compression: Problem Statement
Input: A message M over alpha , where for a  . Output: A binary tree T of m=|| leaves, with each leaf assigned a unique character from . Goal: That the value: ABL(T)= af(a)d(T(a)) is minimized over all possible T.

28. Problem History Problem falls into the field of Information Theory
Worked on by Shannon and Fano
Fathers of information theory
Their algorithm didn’t always work
1951: Fano assigns term paper
Ph.D. student David Huffman tackles problem
Produces Huffman coding
Earns A+ for course

29. A greedy approach to creating an optimal tree
I have a message, and want to create the optimal tree
I find the two characters used the least
I create two nodes for them and make them siblings
I know that there is an optimal tree that has these nodes as siblings
Thus I have not yet screwed up – I can build around this

30. Algorithm
x
f(x) a
0.25 b c
0.2 d
0.15 e
1.0 1 a b c d e
0.55 1
0.45 1
0.3 1 a
0.25 b
0.25 c
0.2 d
0.15 e
0.15 00 10 11
010
011

31. Algorithm Description
Node HuffmanTree(vector Sigma)
PriorityQueue PQ
for i  0 to |Sigma|-1
PQ.push(createNode(f(Sigma[i]), NIL, NIL)
while PQ.size() > 1
(fu, u)  PQ.ExtractMin()
(fv, v)  PQ.ExtractMin()
PQ.push(createNode(fu + fv, u, v))
return PQ.ExtractMin()

32. Final Word
Huffman code is optimal for symbol-by-symbol coding with a known data distribution
Cannot adapt to changing statistics
Cannot consider multi-character
relationships
Does not realize that “the” is more common
than “teh"

33. Huffman Coding Proof of optimality to come
What is the runtime analysis
Of creating the code?
Of encoding a string?
Of decoding a string?

34. Huffman encoding The Huffman encoding algorithm is a greedy algorithm
You always pick the two smallest numbers to combine
Average bits/char: 0.22* * * * * *4 = 2.42
The Huffman algorithm finds an optimal solution
100 54
A=00 B=100 C=01 D=1010 E=11 F=1011 27 46 15
A B C D E F 8

35. Minimum spanning tree
A minimum spanning tree is a least-cost subset of the edges of a graph that connects all the nodes
Start by picking any node and adding it to the tree
Repeatedly: Pick any least-cost edge from a node in the tree to a node not in the tree, and add the edge and new node to the tree
Stop when all nodes have been added to the tree 6
The result is a least-cost ( =12) spanning tree
If you think some other edge should be in the spanning tree:
Try adding that edge
Note that the edge is part of a cycle
To break the cycle, you must remove the edge with the greatest cost
This will be the edge you just added 3 2 4 1 5 3 2 4 9

36. Traveling salesman E A B C D
A salesman must visit every city (starting from city A), and wants to cover the least possible distance
He can revisit a city (and reuse a road) if necessary
He does this by using a greedy algorithm: He goes to the next nearest city from wherever he is
From A he goes to B
From B he goes to D
This is not going to result in a shortest path!
The best result he can get now will be ABDBCE, at a cost of 16
An actual least-cost path from A is ADBCE, at a cost of 14 E A B C D 2 3 4 10

37. Analysis
A greedy algorithm typically makes (approximately) n choices for a problem of size n
(The first or last choice may be forced)
Hence the expected running time is: O(n * O(choice(n))), where choice(n) is making a choice among n objects
Counting: Must find largest useable coin from among k sizes of coin (k is a constant), an O(k)=O(1) operation;
Therefore, coin counting is (n)
Huffman: Must sort n values before making n choices
Therefore, Huffman is O(n log n) + O(n) = O(n log n)
Minimum spanning tree: At each new node, must include new edges and keep them sorted, which is O(n log n) overall
Therefore, MST is O(n log n) + O(n) = O(n log n) 11

38. Other greedy algorithms
Dijkstra’s algorithm for finding the shortest path in a graph
Always takes the shortest edge connecting a known node to an unknown node
Kruskal’s algorithm for finding a minimum-cost spanning tree
Always tries the lowest-cost remaining edge
Prim’s algorithm for finding a minimum-cost spanning tree
Always takes the lowest-cost edge between nodes in the spanning tree and nodes not yet in the spanning tree

39. Dijkstra’s shortest-path algorithm
Dijkstra’s algorithm finds the shortest paths from a given node to all other nodes in a graph
Initially,
Mark the given node as known (path length is zero)
For each out-edge, set the distance in each neighboring node equal to the cost (length) of the out-edge, and set its predecessor to the initially given node
Repeatedly (until all nodes are known),
Find an unknown node containing the smallest distance
Mark the new node as known
For each node adjacent to the new node, examine its neighbors to see whether their estimated distance can be reduced (distance to known node plus cost of out-edge)
If so, also reset the predecessor of the new node

40. Analysis of Dijkstra’s algorithm I
Assume that the average out-degree of a node is some constant k
Initially,
Mark the given node as known (path length is zero)
This takes O(1) (constant) time
For each out-edge, set the distance in each neighboring node equal to the cost (length) of the out-edge, and set its predecessor to the initially given node
If each node refers to a list of k adjacent node/edge pairs, this takes O(k) = O(1) time, that is, constant time
Notice that this operation takes longer if we have to extract a list of names from a hash table

41. Analysis of Dijkstra’s algorithm II
Repeatedly (until all nodes are known), (n times)
Find an unknown node containing the smallest distance
Probably the best way to do this is to put the unknown nodes into a priority queue; this takes k * O(log n) time each time a new node is marked “known” (and this happens n times)
Mark the new node as known -- O(1) time
For each node adjacent to the new node, examine its neighbors to see whether their estimated distance can be reduced (distance to known node plus cost of out-edge)
If so, also reset the predecessor of the new node
There are k adjacent nodes (on average), operation requires constant time at each, therefore O(k) (constant) time
Combining all the parts, we get: O(1) + n*(k*O(log n)+O(k)), that is, O(nk log n) time

42. Connecting wires
There are n white dots and n black dots, equally spaced, in a line
You want to connect each white dot with some one black dot, with a minimum total length of “wire”
Example:
Total wire length above is = 8
Do you see a greedy algorithm for doing this?
Does the algorithm guarantee an optimal solution?
Can you prove it?
Can you find a counterexample?

43. Collecting coins A checkerboard has a certain number of coins on it
A robot starts in the upper-left corner, and walks to the bottom left-hand corner
The robot can only move in two directions: right and down
The robot collects coins as it goes
You want to collect all the coins using the minimum number of robots
Example:
Do you see a greedy algorithm for doing this?
Does the algorithm guarantee an optimal solution?
Can you prove it?
Can you find a counterexample?

44. The End 12