1. Huffman Codes Juan A. Rodriguez CS 326 5/13/2003

2. Presentation Content Introduction Encoding Huffman’s algorithm Huffman’s code Dynamic Huffman encoding Quiz

3. Introduction Suppose we have to encode a text that compromises of n characters. Huffman code is a coding scheme that yields a shorter bit string by applying the idea of assigning shorter codes words to more frequent characters and longer code words to less frequent characters. Same idea that was used in the mid-19 th century by Samuel Morse, where frequent letters such as e(.) and a(._) are assigned short sequence of dots and dashes while infrequent letters such as q (_ _. _) and z (_ _..) have longer ones.

4. Encoding Fixed Length encoding assigns to each character a bit string of the same length. That is what the standard seven-bit ASCII codes does

5. Encoding Variable length encoding assigns code words a different lengths to different characters Huffman codes is an example of variable length encoding

6. Encoding Prefix codes No code word is a prefix of another code word Simply scan a bit string until the first group of bits that is a code word for some character and repeat this operation until the bit string’s end is reached. Simplifies encoding and decoding.

7. Huffman’s Algorithm Suppose you have a 1000 character data file with the following properties:

8. Huffman’s Algorithm Step 1: Initialize n one-node trees and label them with the character of the alphabet I_UEJLROS D

9. Huffman’s Algorithm Step 2: Record the frequency of each character in its tree’s root to indicate the tree’s weight. D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04

10. Huffman’s Algorithm Step 3: Find 2 trees with the smallest weight and make them left and right sub-tree of a new tree and record the sum of their weights in the root of the new tree as it’s weight D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04

11. Huffman’s Algorithm Step 3 continued D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04

12. Huffman’s Algorithm Step 3 continued D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04.08

13. Huffman’s Algorithm Step 3 continued D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04.08.14

14. Huffman’s Algorithm Step 3 continued D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04.08.14.2

15. Huffman’s Algorithm Step 3 continued D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04.08.14.2.27

16. Huffman’s Algorithm Step 3 continued D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04.08.14.2.27.33

17. Huffman’s Algorithm Step 3 continued D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04.08.14.2.27.33.40

18. Huffman’s Algorithm Step 3 continued D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04.08.14.2.27.33.40.60

19. Huffman’s Algorithm Step 3 cont D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04.08.14.2.27.33.40.60 1

20. Huffman’s Algorithm Step 4: We take the convention that going left down the binary tree means adding a 0, and going right down the binary tree means adding a 1.

21. Huffman’s Algorithm Step 4 cont D.1 I.2 _.01 U.13 E.18 J.1 L.06 R.03 O.15 S.04.08.14.2.27.33.40.60 1 0 0 1 0 0 0 0 0 0 0 1 1 1 1 1 11 1

22. Huffman’s Algorithm The algorithm is greedy, which means that it makes choices that are locally optimal and hopes it yields a globally optimal solution. Notice that this is a full binary tree: every non-leaf node has two children. This is true of all optimal codes.

23. Huffman’s Algorithm The operation that we need to perform repeatedly is the extraction of the two sub-trees with the smallest frequencies. This can be implemented using a priority queue.

24. Example of Huffman’s algorithm code implementation ML. The operation that we need to perform repeatedly is the extraction of the two sub-trees with the smallest frequencies. This can be implemented using a priority queue. Building the initial queue takes time O(n log n) since each enqueue operations takes O(log n). Then we perform n-1 merges, each of which takes O(log n). Thus this implementation of Huffman’s algorithm takes O(n long n). datatype HTree = Leaf of char * int | Branch of HTree * int * Htree fun huffmanTree(alpha : (char * int) list) : HTree = let val alphasize = length(alpha) fun freq(node:HTree):int = case node of Leaf(_,i) => i | Branch(_,i,_) => i val q = new_heap (fn (x,y) => Int.compare(freq x, freq y)) alphasize fun merge(i:int):HTree = if i = 0 then extract_min(q) else let val x = extract_min(q) val y = extract_min(q) in insert q (Branch(x, freq(x)+freq(y), y)); merge(i-1) end in app (fn (c:char,i:int):unit => insert q (Leaf(c,i))) alpha; merge(alphasize-1) end

25. Huffman’s Code The output of Huffman’s algorithms is Huffman’s code:

26. Huffman’s Code Encoding of LORI: 1010 110 101101 00 Given the probabilities and codeword length the expected bits per character in the code is 2.66. Had we used a fixed- length encoding for the same alphabet, we would use at least 4 bits per character

27. Huffman’s Code This code achieves the compression ratio, a standard measure of the compression algorithm’s effectiveness, of 35.5%. Huffman’s encoding of this text will use 35.5% less memory than it’s fixed length encoding. Extensive experiments with Huffman’s codes have shown the compression ratio for this scheme typically falls between 20% and 80%.

28. Dynamic Huffman Encoding Huffman’s encoding yields an optimal (minimal length) encoding providing the probabilities of characters occurrences are know in advance. Draw back: a preliminary scanning of a given text to count the frequencies of the character occurrences in it. We use the algorithm to compute the an optimal prefix tree, and we scan the text a SECOND time, writing the out the code words of each character of the text.

29. Dynamic Huffman Encoding Dynamic (Adaptive) Huffman coding builds a tree incrementally in such a way that the coding always is optimal for the sequence of characters already seen.

30. Huffman’s Code Quiz Given the following bit string, what does it decode into? 0111111011110010 1111011000010111 1011001010110101 101010