1. 資訊理論 授課老師 : 陳建源 Email:cychen07@nuk.edu.tw 研究室 : 法 401 網站 http://www.csie.nuk.edu.tw/~cychen/ Ch3: Coding Theory

2. Discrete and memoryless 3. 1 Memoryless sources source Letter: 彼此獨立 傳輸形式 : code Ex: Morse code E. Q --.- B -… Variable-length codes There are N n possible source sequences of length n 探討 code word 與 source sequence 關係

3. Ch3: Coding Theory 3. 2 Fixed-length codes source 傳輸形式 : code 每個 code 長度都相同 The number of letters in the code alphabet: M The length of a code word: m M m code words 探討 code word 與 source sequence 關係 Mm ≧ NnMm ≧ Nn 為了能夠解回原 sequence ，必須滿足 N n possible source sequences

4. Ch3: Coding Theory 3. 2 Fixed-length codes Ex 3.2 Each letter of the English alphabet is converted to binary representation by fixed-length coding. What is the minimum length of a code word? N=26 ； n=1 M=2 Mm ≧ NnMm ≧ Nn m ≧ log26 ≧ 4.7 the minimum length of a code word: 5

5. Ch3: Coding Theory 3. 2 Fixed-length codes Given ε>0 and δ>0, the source sequences can be divided into two groups when n is sufficient large. The probability P(u) of the occurrence of a sequence u in the first group satisfies The sum of the probabilities of all sequences in the second group is less than ε. Source sequence partition theorem G 1 : contains every sequence u in which G 2 : consists of all other sequences

6. Ch3: Coding Theory 3. 3 Variable-length codes A code is unique decodable if, for each source sequence of finite length, the sequence of code letters does not coincide with the sequence of code letters for any other source sequence. Uniquely decodable A prefix condition code is one in which no code word is the prefix of any other code word. Prefix condition code

7. Ch3: Coding Theory 3. 3 Variable-length codes Source letter P(a k ) Code A Code B Code C Code D a 1 0.5 0 0 0 0 a 2 0.25 0 1 10 01 a 3 0.125 1 00 110 011 a 4 0.125 10 11 111 0111 Prefix condition code Uniquely decodable undecodable

8. Ch3: Coding Theory 3. 3 Variable-length codes A prefix condition code exists for code words of length L 1, L 2, …, L N if, and only if, Kraft’s theorem

9. Ch3: Coding Theory 3. 3 Variable-length codes If a code is uniquely decodable with code words of length L 1, L 2, …, L N then the inequality of Kraft’s theorem is satisfied Theorem 3.3 Any uniquely decodable code can be replaced by a prefix condition code without changing any of the lengths of the code words. Corollary 3.3

10. Ch3: Coding theory 假設 code word w 1, w 2, …,w N, 其長度 L 1, L 2, …,L N 平均長度 3. 4 Average length of a code word 機率 p 1, p 2, …,p N A source sequence of n letters would then be expected to initiate a code sequence of about letters.

11. Ch3: Coding theory Variable-length coding theorem 3. 4 Average length of a code word For any uniquely decodable code Code words can always be chosen to satisfy the prefix condition and

12. Ch3: Coding theory According to variable-length coding theorem 3. 4 Average length of a code word A prefix condition code is uniquely decodable. For any given source, a code can be discovered which is uniquely decodable and whose average length is fixed to within one code letter. The source entropy s a fair estimate of the average length of code word. -log p k is a guide to the number of digits in the code word corresponding to a k. The higher the information associated with a k, the longer is the code word attached to it.

13. Ch3: Coding theory 3. 5 Optimal coding Optimal coding can be regarded as achieved if any other set of code words has an which is at least as large. For a given source there is an optimal (uniquely decodable) binary code in which the least likely code words w N-1 and w N have the same lengths and differ only in the last digit, w N-1 ending in 0 and w N in 1. 假設 code word w 1, w 2, …,w N, 其機率 p 1 ≧ p 2 ≧ … ≧ p N Theorem 3.5a

14. Ch3: Coding theory 3. 5 Optimal coding If a prefix condition code is optimal for U’, it is also optimal for U. Theorem 3.5b U: 假設 code word w 1, w 2, …,w N, 其機率 p 1 ≧ p 2 ≧ … ≧ p N U’: 假設 code word w’ 1, w’ 2, …,w’ N-1, 其中

15. Ch3: Coding theory 3. 5 Optimal coding Huffman coding Source letter P(a k ) Code A a 1 0.3 a 2 0.25 a 3 0.25 a 4 0.1 a 5 0.1 a5a5 a4a4 0.2 a3a3 0.45 a2a2 0.55 a1a1 1 0 0 0 1 1 1 0 1 11 10 01 001 000 =2*0.3+2*0.25+2*0.25+3*0.1+3*0.1=2.2