1. BAB 3
HUFFMAN CODE
Universitas Telkom
INFORMATION THEORY

2. Content Huffman Code Extended Huffman Code (Block Huffman Code)
Non Binary Huffman Code
Adaptive Huffman Code

3. Huffman Code Algorithm
Statistical Encoding
Assign fewer bits to symbols that occur more frequently and more bits to symbols appear less often.
There’s no unique Huffman code and every Huffman code has the same average code length.

4. Huffman Code Algorithm
To determine Huffman code, it is useful to construct a binary tree.
Leaves are characters to be encoded
Nodes carry occurrence probabilities of the characters belonging to the subtree

5. Huffman Coding Algorithm
Find (or assume) the probability of each values occurrence.
Order the values in a row of a table according to their probability.
Take the two symbols with the lowest probability, and place them as leaves on a binary tree.
Form a new row in the table replacing the these two symbols symbols with a new symbol. This new symbol forms a branch node in the tree. Draw it in the tree with branches to its leaf (component) symbols

6. Huffman Coding Algorithm
Assign the new symbol a probability equal to the sum of the component symbol’s probability.
Repeat the above until there is only one symbol left. This is the root of the tree.
Nominally assign 1’s to the right hand branches and 0’s to the left hand branches at each node.
Read the code for each symbol from the root of the tree.

7. Huffman Coding Stepping
Example of Huffman Design Steps
1. Label each node w/ one of the source symbol probabilities
2. Merge the nodes labeled by the two smallest probabilities into a parent node
3. Label the parent node w/ the sum of the two children’s probabilities
4. Among the elements in reduced alphabet, merge two with smallest probs.
5. Label the parent node w/ the sum of the two children probabilities.
6. Repeat steps 4 & 5 until only a single super symbol remains

8. Huffman Coding Example

9. Huffman Coding Example

10. Huffman Coding Example

11. Huffman Coding Example
Encoding
Decoding

12. Properties of Huffman Codes
No Huffman code is the prefix of any other Huffman codes so decoding is unambiguous
The Huffman coding technique is optimal (but we must know the probabilities of each symbol for this to be true)
Symbols that occur more frequently have shorter Huffman codes

13. Example of Huffman coding

14. Example of Huffman coding

15. Huffman Coding : Another Presentation
Algorithm:
Make a leaf node for each code symbol
Add the generation probability of each symbol to the leaf node
Take the two leaf nodes with the smallest probability and connect them into a new node
Add 1 or 0 to each of the two branches
The probability of the new node is the sum of the probabilities of the two connecting nodes
If there is only one node left, the code construction is completed. If not, go back to (2)

16. Huffman Coding Example
Example: How does a Huffman code look like for symbols with statistical symbol occurrence probabilities:
P(A) = 0.16
P(B) = 0.51
P(C ) = 0.09
P(D) = 0.13
P(E) = 0.11

17. Huffman Coding Example
Step 1 : Sort all Symbols according to their probabilities
(left to right) from Smallest to largest
these are the leaves of the Huffman tree
P(B) = 0.51
P(C) = 0.09
P(E) = 0.11
P(D) = 0.13
P(A)=0.16

18. Huffman Coding Example
Step 2: Build a binary tree from left to
Right
Policy: always connect two smaller nodes
together (e.g., P(CE) and P(DA) had both
Probabilities that were smaller than P(B),
Hence those two did connect first
P(CEDAB) = 1
P(B) = 0.51
P(CEDA) = 0.49
P(CE) = 0.20
P(DA) = 0.29
P(C) = 0.09
P(E) = 0.11
P(D) = 0.13
P(A)=0.16 18

19. Huffman Coding Example
Step 3: label left branches of the tree
With 0 and right branches of the tree
With 1
P(CEDAB) = 1 1
P(B) = 0.51
P(CEDA) = 0.49 1
P(CE) = 0.20
P(DA) = 0.29 1 1
P(C) = 0.09
P(E) = 0.11
P(D) = 0.13
P(A)=0.16

20. Huffman Coding Example
Step 4: Create Huffman Code
Symbol A = 011
Symbol B = 1
Symbol C = 000
Symbol D = 010
Symbol E = 001
P(CEDAB) = 1 1
P(B) = 0.51
P(CEDA) = 0.49 1
P(CE) = 0.20
P(DA) = 0.29 1 1
P(C) = 0.09
P(E) = 0.11
P(D) = 0.13
P(A)=0.16

21. Limits of Huffman coding (worst case)
When all the probabilities are equal.
That is there is no statistical bias.
Example
A(1/8), B(1/8), C(1/8), D(1/8)
E(1/8), F(1/8), G(1/8). H(1/8)
Figures in brackets are probabilities
Construct Huffman tree:
A(1/8), B(1/8), C(1/8), D(1/8) E(1/8), F(1/8), G(1/8). H(1/8)
AB(1/4), C(1/8), D(1/8) E(1/8), F(1/8), G(1/8). H(1/8)
AB(1/4), CD(1/4), E(1/8), F(1/8), G(1/8). H(1/8)
AB(1/4), CD(1/4), E(1/8), F(1/8), G(1/8). H(1/8)
AB(1/4), CD(1/4), EF(1/4), G(1/8), H(1/8)
AB(1/4), CD(1/4), EF(1/4), GH(1/4)
ABCD(1/2), EFGH(1/2)
ABCDEFGH(1)

22. Limits of Huffman coding (worst case)
Reading the codes
A E 011
B F 010
C G 001
D H 000

23. Limits of Huffman Coding (Best Case)
When all the probabilities change in powers of 2.
That is there is optimum statistical bias.
Example
A(1/128), B(1/128), C(1/64), D(1/32)
E(1/16), F(1/8), G(1/4). H(1/2)
Figures in brackets are probabilities
Construct Huffman tree:
A(1/128), B(1/128), C(1/64), D(1/32), E(1/16), F(1/8), G(1/4). H(1/2)
AB(1/64), C(1/64), D(1/32), E(1/16), F(1/8), G(1/4). H(1/2)
ABC(1/32), D(1/32), E(1/16), F(1/8), G(1/4). H(1/2)
ABCD(1/16), E(1/16), F(1/8), G(1/4). H(1/2)
ABCDE(1/8), F(1/8), G(1/4). H(1/2)
ABCDEF(1/4), G(1/4). H(1/2)
ABCDEFG(1/2). H(1/2)
ABCDEFGH(1)

24. Limits of Huffman Coding (Best Case)
Reading the codes
A E 1110
B F 110
C G 10
D H 0

25. PROPERTIES The Huffman code results in unambiguous code.
The code is reversible without loss.
The table for the translation of the code has to be stored together with the coded image.
The Huffman code does not take correlation between adjacent pixels into consideration.

26. Huffman Coding Variants
extended Huffman coding
we group the symbols into k symbols giving an extended alphabet of nk symbols
This leads to somewhat better compression
adaptive Huffman coding
we don’t assume that we know the exact probabilities
Start with an estimate and update the tree as we encode/decode

27. Exercise Symbol Probability A 0.30 B 0.25 C 0.2 D 0.12 E 0.08 F 0.05
For 6 symbols given below :
a. calculate the entropy,
b. build a Huffman code
c. Calculate the average code size,
d. Calculate the percentage of difference between average code size compare to the entropy
e. Calculate the eficiency of the code compare to the the code that using equal length bit
Symbol
Probability A
0.30 B
0.25 C
0.2 D
0.12 E
0.08 F
0.05

28. Extended Huffman Code (Block Huffman Code)
Universitas Telkom
INFORMATION THEORY

29. Extended Huffman Code
Useful when Huffman not effective due to large Pmax, so the deviation from the entropy is quite big
Example: Consider IID (Independent, identically distributed) Source with P(a1) = 0.8 P(a2) = 0.02 P(a3) = 0.18
The entropy is bits/ symbol, the average length code is 1.2 bits/ symbol
So, The Huffman code just gives 47% more bits than the entropy!!
Letter
Codeword a1 a2 11 a3 10

30. Extended Huffman Code
Block codes allow better performance Because they allow noninteger # bits/symbol
Group into n-symbol blocks 􀃎
–> map between original alphabet & a new “extended” alphabet
Need mn codewords… use Huffman procedure on probs of blocks

31. Extended Huffman Code Example
Letter
Probabilty
Code
a1a1
0.64
a1a2
0.016
10101
a1a3
0.144 11
a2a1
101000
a2a2
0.0004
a2a3
0.0036
a3a1
0.1440
100
a3a2
a3a3
0.0324
1011
2 bit is read at a time  There are 23 = 9 symbol
The average codeword length for this extended code is bits/ symbol
Therefore, in terms of the original alphabet, the average codeword length is /2 = bits/ symbol, only about 5.5% of the entropy

32. Non Binary Huffman Code
Universitas Telkom
INFORMATION THEORY

33. Non Binary Huffman Code
Huffman codes can be applied to n-ary code space.
For example, codewords composed of {0, 1, 2}, we have ternary Huffman code
Let A = {a1, …, a5}, P(ai) = {0.25, 0.25, 0.2, 0.15, 0.15}.

34. Non Binary Huffman Code
For example, codewords composed of {0, 1, 2}, we have ternary Huffman code
Let A = {a1, …, a5}, P(ai) = {0.25, 0.25, 0.2, 0.15, 0.15}.
Symbol
Codeword a1 1 a2 2 a3 00 a4 01 a5 02
P(a1 .. a5) = 1 1 2
P(a1) = 0.25
P(a2) = 0.25
P(a3..a5) = 0.50 2 1
P(a3) = 0.20
P(a4) = 0.15
P(a5)=0.15

35. Adaptive Huffman Code
Universitas Telkom
INFORMATION THEORY

36. ADAPTIVE HUFFMAN CODE
Adaptive Huffman coding (also called Dynamic Huffman coding) is an adaptive coding technique based on Huffman coding.
It permits building the code as the symbols are being transmitted, having no initial knowledge of source distribution, that allows one-pass encoding and adaptation to changing conditions in data.
The benefit of one-pass procedure is that the source can be encoded in real time, though it becomes more sensitive to transmission errors, since just a single loss ruins the whole code.

37. Why Adaptive Huffman Coding?
Huffman coding suffers from the fact that the uncompresser need have some knowledge of the probabilities of the symbols in the compressed files
this can need more bit to encode the file
if this information is unavailable compressing the file requires two passes
first pass: find the frequency of each symbol and construct the huffman tree
second pass: compress the file

38. The key idea
The key idea is to build a Huffman tree that is optimal for the part of the message already seen, and to reorganize it when needed, to maintain its optimality

39. Some history
Adaptive Huffman coding was first conceived independently by Faller (1973) and Gallager (1978)
Knuth contributed improvements to the original algorithm (1985) and the resulting algorithm is referred to as algorithm FGK
A more recent version of adaptive Huffman coding is described by Vitter (1987) and called algorithm V

40. Algorithm FGK
The basis for algorithm FGK is the Sibling Property (Gallager 1978)
A binary code tree with nonnegative weights has the sibling property if each node (except the root) has a sibling and if the nodes can be numbered in order of nondecreasing weight with each node adjacent to its sibling. Moreover the parent of a node is higher in the numbering
A binary prefix code is a Huffman code if and only if the code tree has the sibling property

41. ADAPTIVE HUFFMAN CODE
Adaptive Huffman coding maintains a dynamic code tree. The tree will be updated synchronously on both transmitter-side and receiver-side. If the alphabet size is m, the total number of nodes £ 2m – 1.

42. Initial Codewords
Before transmission of any symbols, all symbols in the source alphabet {a1, a2, …, am} belongs to the NYT (not yet transmitted) list.
Each symbol in the alphabet has an initial codeword using either log2m or log2m+1 bits fixed-length binary code.
When a symbol ai is transmitted for the first time, the code for NYT is transmitted, followed by the fixed code for ai. A new node is created for ai and ai is taken out of the NYT list.
From this point on, we follow the update procedure to maintain the Huffman code tree.

43. Update Procedure

44. Encoding Procedure

45. Decoding Procedure

46. Algorithm FGK 32 11 11 f 21 10 9 10 11 7 8 5 c 5 d 6 e 5 4 5 6 3 2 a 3 b 1 2
Note that node numbering corresponds to the order in which the nodes are combined by Huffman’s algorithm, first nodes 1 and 2, then nodes 3 and 4 ...

47. Algorithm FGK
In algorithm FGK, both encoder and decoder maintain dynamically changing Huffman code trees. For each symbol the encoder sends the codeword for that symbol in current tree and then update the tree
The problem is to change quickly the tree optimal after t symbols (not necessarily distinct) into the tree optimal for t+1 symbols
If we simply increment the weight of the t+1-th symbols and of all its ancestors, the sibling property may no longer be valid  we must rebuild the tree

48. Algorithm FGK b Suppose next symbol is “b” if we update the weigths...33 32 11
Suppose next symbol is “b”
if we update the weigths...
... sibling property is violated!!
This is no more a Huffman tree 11 f 22 21 10 9 11 10 11 7 8 5 c 5 d 6 e 5 6 4 5 6 3 2 a 3 b 4 1 2
no more ordered by nondecreasing weight

49. Algorithm FGK The solution can be described as a two-phase process
first phase: original tree is transformed in another valid Huffman tree for the first t symbols, that has the property that simple increment process can be applied succesfully
second phase: increment process, as described previously

50. Algorithm FGK The first phase starts at the leaf of the t+1-th symbol
We swap this node and all its subtree, but not its numbering, with the highest numbered node of the same weight
New current node is the parent of this latter node
The process is repeated until we reach the root

51. Algorithm FGK First phase b Second phase Node 2: nothing to be done
Node 4: to be swapped with node 5
Node 8: to be swapped with node 9
Root reached: stop!
Second phase 33 32 11 b 11 f 12 21 10 9 6 10 11 7 8 4 5 c 5 d 6 e 5 4 5 6 3 2 a 3 b 1 2

52. Algorithm FGK - summary
The algorithm starts with only one leaf node, the 0-node. As the symbols arrive, new leaves are created and each time the tree is recomputed
Each symbol is coded with its codeword in the current tree, and then the tree is updated
Unseen symbols are coded with 0-node codeword and some other bits are needed to specify the symbol
Algorithm FGK compares favourably with static Huffman code, if we consider also overhead costs (it is used in the Unix utility compact)

53. Exercise
Construct the static Huffman tree and the FGK tree for the message e eae de eabe eae dcf and evaluate the number of bits needed for the coding with both the algorithms, ignoring the overhead for Huffman
SOL. FGK  60 bits, Huffman  52 bits
FGK is obtained using the minimum number of bits for the element in the list of the unseen symbols

54. Algorithm FGK
if T=“total number of bits transmitted by algorithm FGK for a message of length t containing n distinct symbols“, then
where S is the performance of the static Huffman (Vitter 1987)
So the performance of algorithm FGK is never much worse than twice optimal

55. Algorithm V
Vitter in his work of the 1987 introduces two improvements over algorithm FGK, calling the new scheme algorithm
As a tribute to his work, the algorithm is become famous... with the letter flipped upside-down... algorithm

56. Algorithm V : The key ideas
swapping of nodes during encoding and decoding is onerous
In FGK algorithm the number of swapping (considering a double cost for the updates that move a swapped node two levels higher) is bounded by , where is the length of the added symbol in the old tree (this bound require some effort to be proved and is due to the work of Vitter)
In algorithm V, the number of swapping is bounded by 1

57. Algorithm V : The key ideas
Moreover algorithm V, not only minimize as Huffman and FGK, but also , i.e. the height of the tree, and , i.e. is better suited to code next symbol, given it could be represented by any of the leaves of the tree
This two objectives are reached through a new numbering scheme, called implicit numbering

58. Algorithm V : Implicit Numbering
The nodes of the tree are numbered in increasing order by level; nodes on one level are numbered lower than the nodes on the next higher level
Nodes on the same level are numbered in increasing order from left to right
If this numbering is satisfied (and in FGK it is not always satisfied), certain types of updates cannot occur

59. An invariant
The key to minimize the other kind of interchanges is to maintain the following invariant
for each weight w, all leaves of weight w precede (in the implicit numbering) all internal nodes of weight w
The interchanges, in the algorithm V, are designed to restore implicit numbering, when a new symbol is read, and to preserve the invariant

60. Algorithm V
if T=“total number of bits transmitted by algorithm V for a message of length t containing n distinct symbols“, then
At worst then, Vitter's adaptive method may transmit one more bit per codeword than the static Huffman method
Empirically, algorithm V slightly outperforms algorithm FGK

61. Vitter Algorithm
Code is represented as a tree structure in which every node has a corresponding weight and a unique number.
Numbers go down, and from right to left.
Weights must satisfy the sibling property, which is that nodes can be listed in order of nonincreasing weight with each node adjacent to its sibling. Thus if A is parent node of B and node C is child of B, then W(A) > W(B) > W(C).
The weight is merely the count of symbols transmitted which codes are associated with children of that node.
A set of nodes with same weights make a block.
To get the code for every node, in case of binary tree we could just traverse all the path from the root to the node, writing down (for example) "1" if we go to the right and "0" if we go to the left.
We need some general and straightforward method to transmit symbols which are not yet transmitted (NYT), we could use, for example, transmission of binary numbers for every symbol in alphabet.
Encoder and decoder start with only the root node, which has the maximum number. In the beginning it is our initial NYT node.

62. Vitter Algorithm
Code is represented as a tree structure in which every node has a corresponding weight and a unique number.
When we transmit an NYT symbol we have to transmit code for the NYT node, then for its generic code.
For every symbol which is already in the tree we only have to transmit code for its leaf node.
For every symbol transmitted on both sides we must execute update procedure:
If current symbol is NYT, add two child nodes to NYT node, one will be a new NYT node the other is leaf node for our symbol, increase weight for new leaf node and old NYT, go to step 4, else go to symbol's leaf node.
If this node does not have the highest number in a block swap it with which has the highest number
Increase weight for current node
If this is not the root node go to parent node, go to step 2, else end.
Note: swapping nodes means swapping weights and corresponding symbols, but not the numbers.

63. Assignment
Given the eight symbols A,B,C,D,E,F,G, and H with probabilties 1/30, 1/30, 1/30, 2/30, 3/30, 5/30, and 12/30. Determine the huffman code and calculate the average code size
Make a paper describing Shanon – Fano code complete with algorithm and example (in Bahasa Indonesia)