1. Ch4. Zero-Error Data Compression Yuan Luo

2. Content  Ch4. Zero-Error Data Compression  4.1 The Entropy Bound  4.2 Prefix Codes  4.2.1 Definition and Existence  4.2.2 Huffman Codes 4.3 Redundancy of Prefix Codes

3. 4.1 The Entropy Bound

4.  Definition 4.2 A code C is uniquely decodable if for any finite source sequence, the sequence of code symbols corresponding to this source sequence is different from the sequence of code symbols corresponding to any other (finite) source sequence. 4.1 The Entropy Bound

10.  Definition 4.8. The redundancy R of a D-ary uniquely decodable code is the difference between the expected length of the code and the entropy of the source. 4.1 The Entropy Bound

11.  4.2.1 Definition and Existence Definition 4.9. A code is called a prefix-free code if no codeword is a prefix of any other codeword. For brevity, a prefix-free code will be referred to as a prefix code. 4.2 Prefix Codes

12. 4.2.1 Definition and Existence

15. achieve

16. Shannon Code If the codeword lengths satisfy Then Kraft inequality is satisfied: Then prefix code exists which we call Shannon code.

17. Shannon-Fano-Elias Code and Arithmetic code Shannon-Fano-Elias code is not efficient comparing with Shannon code, but considers accumulative probability distribution which is used to construct codewords one by one in Arithmetic code. Example. 60% prob.: symbol NEUTRAL 20% prob.: symbol POSITIVE 10% prob.: symbol NEGATIVE 10% prob.: symbol END.

19. 4.2.2 Huffman Codes As we have mentioned, the efficiency of a uniquely decodable code is measured by its expected length. Thus for a given source X, we are naturally interested in prefix codes which have the minimum expected length. Such codes, called optimal codes, can be constructed by the Huffman procedure, and these codes are referred to as Huffman codes.

20. 4.2.2 Huffman Codes

21.  Lemma 4.15. In an optimal code, shorter codewords are assigned to larger probabilities.  Lemma 4.16. There exists an optimal code in which the codewords assigned to the two smallest probabilities are siblings, i.e., the two codewords have the same length and they differ only in the last symbol. 4.2.2 Huffman Codes

22. Theorem The Huffman procedure produces an optimal prefix code. 4.2.2 Huffman Codes

24. 4.3 Redundancy of Prefix Codes

31. Thank you!