1. §3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem

2. 1. Source coder §3.1 Source coding Source coder Source alphabet Channel input alphabet Code Extended source coder  Example 3.1

3. §3.1 Source coding 2. Examples 1) ASCII source coder ASCII coder {0,1} {English symbol, command} {binary code, 7 bits}

4. 2) Morse source coder Source coder (1) Source coder (2) {0,1}{., —} {A,B,…,Z}Binary code 2. Examples §3.1 Source coding

5. 3) Chinese telegraph coder “中”“中” “0022” “01101 01101 11001 11001” 2. Examples §3.1 Source coding

6. Constant-length codes Variable-length codes Distortionless codes Distortion codes 2. Classification of the source coding Uniquely decodable (UD) codes Non-UD codes §3.1 Source coding The code C is called uniquely decodable (UD) if each string in each C k arises in only one way as a concatenation of codewords. This means that if say and each of the τ’s and σ’s is a codeword, then Thus every string in C k can be uniquely decoded into a concatenation of codewords.

7. 2. Classification of the source coding  Example 3.2 §3.1 Source coding

8. 3. Parameters about source coding 1) Average length of coding For extended source coding: (code/sig) code/m-sigs Length of codeword §3.1 Source coding (code/sig)

9. 2) Information rate of coding (bit/code) 3. Parameters about source coding §3.1 Source coding

10. 3) Coding efficiency Actual rate Maximum rate 3. Parameters about source coding §3.1 Source coding For extended source coding:

11. §3.2 Distortionless source coding theorem §3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem

12.  Example 3.3 The binary DMS has the probability space: §3.2 Distortionless source coding theorem 1) “0” a1a1, “1” a2a2 2) a 1 a 1 : 0 a 1 a 2 : 10 a 2 a 1 : 110 a 2 a 2 : 111

13. Average length of coding: Code efficiency: §3.2 Distortionless source coding theorem “0” a1a1, “1” a2a2 Rate:  Example 3.3

14. Extended source coding code Length of codeword a1a1a1a1 01 a1a2a1a2 102 a2a1a2a1 1103 a2a2a2a2 1113 Average length of coding : Code efficiency: Rate: §3.2 Distortionless source coding theorem  Example 3.3

15. m times extended source coding m = 3: R 3 = 0.985 (bit/code) m = 4: R 4 = 0.991 (bit/code) m §3.2 Distortionless source coding theorem  Example 3.3

16. §3.2 Distortionless source coding theorem  Distortionless source coding theorem Theorem 3.1 If the code C is UD, its average length must exceed the s-ary entropy of the source, that is, (Theorem 11.3 in textbook)

17. §3.2 Distortionless source coding theorem  Distortionless source coding theorem Theorem 3.2 (Theorem 11.4 in textbook)

18. §3.2 Distortionless source coding theorem Theorem 3.3 (Theorem 11.5 in textbook)  Distortionless source coding theorem The source can indeed be represented faithfully using s-ary symbols per source symbol.

19. §3.2 Distortionless source coding theorem  Distortionless source coding theorem corollary The efficient UD codes are achievable if rate R ≤ C. (C is the capacity of s-ary lossless channel )

20. Review KeyWords: Source coder Variable-length codes distortionless codes Uniquely decodable codes Average length of coding Information rate of coding Coding efficiency Shannon’s TH1

21. Homework 1. p344: 11.12 2. p345:11.20

22. §3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem

23. §3.3 The rate-distortion function 1. Introduction Review  Distortionless source coding theorem (corollary) The efficient UD codes are achievable if rate R ≤ C. (C is the capacity of s-ary lossless channel ) Conversely, any sequence of (2 nR, n) codes with must have R ≤ C.  The channel coding theorem (Statement 2 ): All rates below capacity C are achievable. Specifically, for every rate R ≤ C, there exists a sequence of (2 nR,n) codes with maximum probability of error.

24. §3.3 The rate-distortion function 1. Introduction Review For distortionless coding:R≤C - (P E →0, R→C - ) But actually…… Given a source distribution and a distortion measure, what is the minimum expected distortion achievable at a particular rate? what is the minimum rate description required to achieve a particular distortion?

25. §3.3 The rate-distortion function 2. Distortion measure coding channel uiui vjvj A U ={u 1,u 2,…,u r }A V ={v 1,v 2,…,v s } Source symbol Destination symbol

26. 2. Distortion measure Average distortion measure: Let the input and output of the channel be U=(U 1,U 2,…,U k ) and V=(V 1,V 2,…,V k ) respectively where, §3.3 The rate-distortion function

27.  Example 3.3.1 A U = A V = {0,1};source statistics p(0) = p, p(1) = q = 1-p, where p ½; and distortion matrix 2. Distortion measure §3.3 The rate-distortion function

28.  Example 3.3.2 A U = {-1,0,+1}, A V = {-1/2, +1/2};source statistics(1/3,1/3,1/3) and distortion matrix 2. Distortion measure §3.3 The rate-distortion function

29. 2. Distortion measure Fidelity criterion: §3.3 The rate-distortion function Test channel: Let the source statistics p(u) and distortion measure d(u,v) are fixed.

30. 3. Rate-distortion function 1) Definition The function is a function of the source statistics (p(u)),the distortion matrix D, and the real number. §3.3 The rate-distortion function The information rate distortion function R k (δ) for a source U with distortion measure d(U, V) is defined as The information rate distortion function R k (δ) for a source U with distortion measure d(U, V) is defined as

31. 3. Rate-distortion function ③ If, then §3.3 The rate-distortion function ② The minimum possible value of is,where R(δ) and C ① The function I(U;V) actually achieves its minimum value on the region of ;

32. 3. Rate-distortion function 2) Properties Theorem 3.4 is a convex function of. (Theorem 3.1 in textbook) §3.3 The rate-distortion function R(0)=H(U)

33. 3. Rate-distortion function 2) Properties Theorem 3.4 is a convex function of. Theorem 3.5 For a DMS, for all k and. (Theorem 3.1 in textbook) (Theorem 3.2 in textbook) §3.3 The rate-distortion function

34. 3. Rate-distortion function  Example 3.3.1 (continued) A U = A V = {0,1};source statistics p(0) = p, p(1) = q = 1-p, where p ½; and distortion matrix §3.3 The rate-distortion function 2) Properties

35. §3.3 The rate-distortion function A U = A V = {0,1,…,r-1}, P{U=u}=1/r Distortions are given by: 2) Properties  Example 3.3.3 3. Rate-distortion function

36. §3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem

37. 1. Distortion source coding theorem (modified on Theorem 3.4 in textbook) §3.4 Distortion source coding theorem Theorem 3.6 (Shannon’s source coding theorem with a fidelity criterion) If, there exists a source code C of length k with M codewords, where: If,no such codes exist. A source symbol can be compressed into R(δ) bits if a distortion δ is allowable.

38. 2. Relation of shannon’s theorems §3.4 Distortion source coding theorem Source Distortion source coder Distortionless source coder Sink Distortion source decoder Distortionless source decoder channel Channel coder Channel decoder A general communication system

39. Review KeyWords: Distortion measure Average distortion measure Fidelity criterion Test channel Rate-distortion function Shannon’s TH3

40.  thinking Source X has the alphabet set {a 1,a 2,…,a 2n },P{X = a i }=1/2n, i = 1,2,…,2n. The distortion measure is Hamming distortion measure,that is Design a source coding method with δ=1/2. §3.4 Distortion source coding theorem