1. CSI-447: Multimedia Systems
Chapter 8: Data Compression (b)

2. Outline Entropy Encoding Predictive Coding Arithmetic Coding
Lossless Predictive Coding
Differential Coding
Lossy Predictive Coding
Differential Pulse Code Modulation Coding (DPCM)
Delta Modulation (DM)

3. Entropy Encoding: Arithmetic Coding
Initial idea introduced in 1948 by Shannon
Many researchers worked on this idea
Modern arithmetic coding can be attributed to Pasco (1976) and Rissanen and Langdon (1979)
Arithmetic coding treats the whole message as one unit
In practice, the input data is usually broken up into chunks to avoid error propagation

4. Entropy Encoding: Arithmetic Coding
A message is represented by a half-open interval [a,b), where a,b.
General idea of encoding
Map the message into an open interval [a,b)
Find a binary fractional number with minimum length that belongs to the above interval. This will be the encoded message
Initially, [a,b) = [0,1)
When the message becomes longer, the length of the interval shortens and the # of bits needed to represent the interval increases

5. Entropy Encoding: Arithmetic Coding
Coding Algorithm
Algorithm
// Input:
ArithmeticCoding
symbol: Input stream of the message terminator: terminator symbol
// : Low[] and High[]: all symbols’ //
ranges the message
Output: binary fractional code of
low = 0; high = 1; range = 1; do {
get (symbol);
high = low + low = low +
range * High(symbol); range * Low(symbol);
– low;
range = high
} while (symbol != terminator)
return
CodeWord(low,high);

6. Entropy Encoding: Arithmetic Coding
Binary code generation
Algorithm CodeWord
// Input: low and high
// Output: binary fractional code = 0;
k = 1;
while (value(code) < low) { assign 1 to the kth binary if (value(code) > high)
code
fraction
bit;
replace the kth bit by 0; k++; }

7. Entropy Encoding: Arithmetic Coding
Example: Assume S = {A,B,C,D,E,F,$}, where $ is the terminator symbol. In addition, assume the following probabilities for each character:
– Pr (A) = 0.2
– Pr(B) = 0.1
– Pr(C) = 0.2
– Pr(D) = 0.05
– Pr(E) = 0.3
– Pr(F) = 0.05
– Pr($) = 0.1
Generate the fractional binary code of the message CAEE

8. Entropy Encoding: Arithmetic Coding
It can be proven that ⎡log (1/ Pi)⎤ is the upper bound on the number of bits needed to encode a message
In our case, the maximum is equal to 12.
When the length of the message increases, the range decreases and the upper bound value
Generally, arithmetic coding outperforms Huffman coding
Treats the whole message as one unit vs. an integral number of bits to code each character in Huffman coding
Redo the previous example CAEE$ using Huffman coding and notice how many bits are required to code this message.

9. Entropy Encoding: Arithmetic Coding
Decoding Algorithm
ArithmeticDecoding
// Input: code: binary code
// : Low[] and High[]: all //
Algorithm
symbols’ ranges
Output: The decoded message
value = Do {
find a
convert2decimal(code);
symbol s so that
Low(s) <= value < High(s); s;
output
low = Low(s); high = High(s); range = high –
low;
value = (value – low) / range;
} while s is not the terminator symbol;

10. Entropy Encoding: Arithmetic Coding
Example

11. Predictive Coding
Predictive coding simply means transmitting differences
Predict the next sample as being equal to the current sample
More complex prediction schemes can be used
Instead of sending the current sample, send the error involved in the previous assumption

12. Predictive Coding: Why?
The idea of forming differences is to make the histogram of sample values more peaked.
In this case, what happens to the entropy?
As a result, which is better to compress?

13. Predictive Coding: Why?

14. Lossless Predictive Coding
Formally, define the integer signal as the set of values fn. Then, we predict values f^n and compute the error en as follows: t ˆ
  ank fnk fn
k 1 ˆ
en  fn  fn
when t = 1, we get ...
Usually, t is between 2 and 4 (in this case it is called a linear predictor)
We might need to have a truncating or rounding operation following the prediction computation

15. Lossless Predictive Coding

16. Lossless Predictive Coding: Example
Consider the following predictor:
⎢ 1 ⎥ ˆ
fn  ⎢
 fn1  fn2 ⎥
⎣ 2 ⎦ ˆ
en  fn  fn
Show how to code the following sequence
f1 , f 2 , f3 , f 4 , f5  21, 22 , 27 , 25, 22 .

17. Lossless Predictive Coding
Examples in the Image Compression Domain
Differential Coding
Lossless JPEG

18. Lossy Predictive Coding: DPCM
DPCM = Differential Pulse Code Modulation
Form the prediction f ^ n
Form an error en
Quantize the error

19. Lossy Predictive Coding: DPCM
The distortion is the average squared error
To illustrate the quality of a compression scheme, diagrams of distortion vs. the number of bit levels used are usually shown
Quantization used
Uniform
Lloyd-Max
– Does better than “Uniform”

20. Lossy Predictive Coding: DPCM

21. Lossy Predictive Coding: DPCM
Example
⎢ 1 ~
f  f  ~ ⎥ ˆ
f 
⎢ 2 ⎥ n
n1
n2 ⎣ ⎦ ˆ
en  fn  fn
e~  Q[e ]  16 * ⎢ 255  en ⎥  256  8
⎢ ⎥
n n ⎣ 16 ⎦ ~ ˆ ~
fn  fn  en
Show how to code the following sequence
f1 , f 2 , f 3 , f 4 , f 5  130 , 150 , 140 , 200 , 230

22. Lossy Predictive Coding
DM (Delta Modulation) is a simplified version of DPCM that is used as a quick analog-to-digital converter.
– Note that the prediction simply involves a delay