1. Increasing Information per Bit
Information in a source
Mathematical Models of Sources
Information Measures
Compressing information
Huffman encoding
Optimal Compression for DMS?
Lempel-Ziv-Welch Algorithm
For Stationary Sources?
Practical Compression
Quantization of analog data
Scalar Quantization
Vector Quantization
Model Based Coding
Practical Quantization
m-law encoding
Delta Modulation
Linear Predictor Coding (LPC)

2. Huffman encoding Variable length binary code for DMS
finite alphabet, fixed probabilities
Code satisfies the Prefix Condition
Codes are instantaneously and unambiguously decodable as they arrive e.g, 0,10,110,111 is OK ,01,011,111 is not OK 01 = 0,111, or 01

3. Huffman encoding
Use Probabilities to order coding priorities of letters
Low probability get codes first (more bits)
This smoothes out the information per bit

4. Huffman encoding D0 D1 D2 D3 D4 Use a code tree to make the code
Combine the symbols with lowest probability to make a new block symbol
Assign a 1 to one of the old symbols code word and 0 to the other symbol
Now reorder and combine the two lowest probability symbols of the new set
Each time the synthesized block symbol has lowest probability the code words get shorter
D0 D1 D2 D D4

5. Huffman encoding Result: Self Information or Entropy
H(X) = (The best possible average number of bits)
Average number of bits per letter
nk = number of bits per symbol
So the efficiency =

6. Huffman encoding
Lets compare to simple 3 bit code

7. Huffman encoding
Another example
D0 D D2 D3

8. Huffman encoding Multi-symbol Block codes
Use symbols made of original symbols
Symbols
Can show the new codes information per bit satisfies:
So large enough block code gets you as close to H(X) as you want

9. Huffman encoding
Lets consider a J=2 block code example

10. Encoding Stationary Sources
Now there are joint probabilities of blocks of symbols that depend on previous bits
Unless DMS
Can show the joint entropy is:
Which means less bits than a symbol by symbol code can be used

11. Encoding Stationary Sources
H(X | Y) is the conditional entropy
Joint (total) probability of xi
Information in xi given yj
Can show:

12. Conditional Entropy
Plotting this for n = m = 2 we see that when Y depends strongly on X then H(X|Y) is low
P(Y=0|X=0)
P(Y=1|X=1)

13. Conditional Entropy To see how P(X|Y) and P(Y|X) relate consider:
They are very simlar when P(X=0) ~ 0.5
P(X=0|Y=0)
P(Y=0|X=0)

14. Optimal Codes for Stationary Sources
Can show that for large blocks of symbols Huffman encoding is efficient
Define
Then Huffman code gives:
Now if
Get:
i.e., Huffman is optimal

15. Lempel-Ziv-Welch Code
Huffman encoding is efficient but need to know joint probabilities of large blocks of symbols
Finding joint probabilities is hard
LZW is independent of source statistics
Is a universal source code algorithm
Is not optimal

16. Input String = /WED/WE/WEE/WEB/WET
Lempel-Ziv-Welch
Build a table from strings not already in the table
Output table location for strings in the table
Build the table again to decode
Input String = /WED/WE/WEE/WEB/WET
Characters Input
Code Output
New code value
New String /
256 /W W
257 WE E
258 ED D
259 D/
260
/WE
261 E/
262
/WEE
263
E/W
264
WEB B
265 B/
266
/WET T
Source:

17. Lempel-Ziv-Welch Decode Input Codes: / W E D 256 E 260 261 257 B 260 T
NEW_CODE
OLD_CODE
STRING/
Output
CHARACTER
New table entry / W
256 = /W E
257 = WE D
258 = ED
256 /W
259 = D/
260 = /WE
260
/WE
261 = E/
261 E/
262 = /WEE
257 WE
263 = E/W B
264 = WEB
265 = B/ T
266 = /WET

18. Lempel-Ziv-Welch
Typically takes hundreds of entries to table before compression occurs
Some nasty patents make licensing an issue