§3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem

1. Source coder §3.1 Source coding Source coder Source alphabet Channel input alphabet Code Extended source coder  Example 3.1

§3.1 Source coding 2. Examples 1) ASCII source coder ASCII coder {0,1} {English symbol, command} {binary code, 7 bits}

2) Morse source coder Source coder (1) Source coder (2) {0,1}{., —} {A,B,…,Z}Binary code 2. Examples §3.1 Source coding

3) Chinese telegraph coder “中”“中” “0022” “ ” 2. Examples §3.1 Source coding

Constant-length codes Variable-length codes Distortionless codes Distortion codes 2. Classification of the source coding Uniquely decodable (UD) codes Non-UD codes §3.1 Source coding The code C is called uniquely decodable (UD) if each string in each C k arises in only one way as a concatenation of codewords. This means that if say and each of the τ’s and σ’s is a codeword, then Thus every string in C k can be uniquely decoded into a concatenation of codewords.

2. Classification of the source coding  Example 3.2 §3.1 Source coding

3. Parameters about source coding 1) Average length of coding For extended source coding: (code/sig) code/m-sigs Length of codeword §3.1 Source coding (code/sig)

2) Information rate of coding (bit/code) 3. Parameters about source coding §3.1 Source coding

3) Coding efficiency Actual rate Maximum rate 3. Parameters about source coding §3.1 Source coding For extended source coding:

§3.2 Distortionless source coding theorem §3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem

 Example 3.3 The binary DMS has the probability space: §3.2 Distortionless source coding theorem 1) “0” a1a1, “1” a2a2 2) a 1 a 1 : 0 a 1 a 2 : 10 a 2 a 1 : 110 a 2 a 2 : 111

Average length of coding: Code efficiency: §3.2 Distortionless source coding theorem “0” a1a1, “1” a2a2 Rate:  Example 3.3

Extended source coding code Length of codeword a1a1a1a1 01 a1a2a1a2 102 a2a1a2a a2a2a2a Average length of coding : Code efficiency: Rate: §3.2 Distortionless source coding theorem  Example 3.3

m times extended source coding m = 3: R 3 = (bit/code) m = 4: R 4 = (bit/code) m §3.2 Distortionless source coding theorem  Example 3.3

§3.2 Distortionless source coding theorem  Distortionless source coding theorem Theorem 3.1 If the code C is UD, its average length must exceed the s-ary entropy of the source, that is, (Theorem 11.3 in textbook)

§3.2 Distortionless source coding theorem  Distortionless source coding theorem Theorem 3.2 (Theorem 11.4 in textbook)

§3.2 Distortionless source coding theorem Theorem 3.3 (Theorem 11.5 in textbook)  Distortionless source coding theorem The source can indeed be represented faithfully using s-ary symbols per source symbol.

§3.2 Distortionless source coding theorem  Distortionless source coding theorem corollary The efficient UD codes are achievable if rate R ≤ C. (C is the capacity of s-ary lossless channel )

Review KeyWords: Source coder Variable-length codes distortionless codes Uniquely decodable codes Average length of coding Information rate of coding Coding efficiency Shannon’s TH1

Homework 1. p344: p345:11.20

§3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem

§3.3 The rate-distortion function 1. Introduction Review  Distortionless source coding theorem (corollary) The efficient UD codes are achievable if rate R ≤ C. (C is the capacity of s-ary lossless channel ) Conversely, any sequence of (2 nR, n) codes with must have R ≤ C.  The channel coding theorem (Statement 2 ): All rates below capacity C are achievable. Specifically, for every rate R ≤ C, there exists a sequence of (2 nR,n) codes with maximum probability of error.

§3.3 The rate-distortion function 1. Introduction Review For distortionless coding:R≤C - (P E →0, R→C - ) But actually…… Given a source distribution and a distortion measure, what is the minimum expected distortion achievable at a particular rate? what is the minimum rate description required to achieve a particular distortion?

§3.3 The rate-distortion function 2. Distortion measure coding channel uiui vjvj A U ={u 1,u 2,…,u r }A V ={v 1,v 2,…,v s } Source symbol Destination symbol

2. Distortion measure Average distortion measure: Let the input and output of the channel be U=(U 1,U 2,…,U k ) and V=(V 1,V 2,…,V k ) respectively where, §3.3 The rate-distortion function

 Example A U = A V = {0,1};source statistics p(0) = p, p(1) = q = 1-p, where p ½; and distortion matrix 2. Distortion measure §3.3 The rate-distortion function

 Example A U = {-1,0,+1}, A V = {-1/2, +1/2};source statistics(1/3,1/3,1/3) and distortion matrix 2. Distortion measure §3.3 The rate-distortion function

2. Distortion measure Fidelity criterion: §3.3 The rate-distortion function Test channel: Let the source statistics p(u) and distortion measure d(u,v) are fixed.

3. Rate-distortion function 1) Definition The function is a function of the source statistics (p(u)),the distortion matrix D, and the real number. §3.3 The rate-distortion function The information rate distortion function R k (δ) for a source U with distortion measure d(U, V) is defined as The information rate distortion function R k (δ) for a source U with distortion measure d(U, V) is defined as

3. Rate-distortion function ③ If, then §3.3 The rate-distortion function ② The minimum possible value of is,where R(δ) and C ① The function I(U;V) actually achieves its minimum value on the region of ;

3. Rate-distortion function 2) Properties Theorem 3.4 is a convex function of. (Theorem 3.1 in textbook) §3.3 The rate-distortion function R(0)=H(U)

3. Rate-distortion function 2) Properties Theorem 3.4 is a convex function of. Theorem 3.5 For a DMS, for all k and. (Theorem 3.1 in textbook) (Theorem 3.2 in textbook) §3.3 The rate-distortion function

3. Rate-distortion function  Example (continued) A U = A V = {0,1};source statistics p(0) = p, p(1) = q = 1-p, where p ½; and distortion matrix §3.3 The rate-distortion function 2) Properties

§3.3 The rate-distortion function A U = A V = {0,1,…,r-1}, P{U=u}=1/r Distortions are given by: 2) Properties  Example Rate-distortion function

§3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2 Distortionless source coding theorem §3.3 The rate-distortion function §3.4 Distortion source coding theorem

1. Distortion source coding theorem (modified on Theorem 3.4 in textbook) §3.4 Distortion source coding theorem Theorem 3.6 (Shannon’s source coding theorem with a fidelity criterion) If, there exists a source code C of length k with M codewords, where: If,no such codes exist. A source symbol can be compressed into R(δ) bits if a distortion δ is allowable.

2. Relation of shannon’s theorems §3.4 Distortion source coding theorem Source Distortion source coder Distortionless source coder Sink Distortion source decoder Distortionless source decoder channel Channel coder Channel decoder A general communication system

Review KeyWords: Distortion measure Average distortion measure Fidelity criterion Test channel Rate-distortion function Shannon’s TH3

 thinking Source X has the alphabet set {a 1,a 2,…,a 2n },P{X = a i }=1/2n, i = 1,2,…,2n. The distortion measure is Hamming distortion measure,that is Design a source coding method with δ=1/2. §3.4 Distortion source coding theorem