1. Lecture 6 Instantaneous Codes and Kraft’s Theorem (Section 1.4)
Theory of Information Lecture 6
Theory of Information
Lecture 6 Instantaneous Codes and Kraft’s Theorem (Section 1.4)

2. Theory of Information Lecture 6
Instantaneous Codes
Theory of Information Lecture 6
DEFINITION A code is said to be instantaneous if, whenever any
sequence of codewords is transmitted, each codeword can be interpreted
as soon as it is received.
Example.
{0, 01}
{0, 10}

3. Prefix and Suffix Properties
Theory of Information Lecture 6
DEFINITION
1) A code is said to have the prefix property if no codeword is a
prefix of any other codeword.
2) A code is said to have the suffix property if no code word is a
suffix of any other codeword.
Fact: Having the suffix or prefix property is sufficient to be uniquely
Decipherable. But not vice versa.
Does {0,01} have the prefix property?
suffix property?
Is it uniquely decipherable?
Does {00, , , 11} have the prefix property?
suffix property?
Is it uniquely decipherable?

4. Prefix Property = Instantaneousity
Theory of Information Lecture 6
THEOREM A code is instantaneous if and only if it has the
prefix property.
Comma code: {0, 10, 110, 1110} has the prefix property and is
instantaneous.
How about {0, 01, 011, 0111}:
Prefix property?
Instantaneous?
Suffix property?
Uniquely decipherable?

5. Theory of Information Lecture 6
Kraft’s Theorem
Theory of Information Lecture 6
THEOREM 1.4.2
There exists an instantaneous r-ary code with codeword lengths
a1,a2,…,aq if and only if Kraft’s inequality is satisfied:
1/ra1 + … + 1/raq  1.
Let C be an instantaneous r-ary code. Then C is maximal
instantaneous, i.e. C is not contained in any strictly larger instantaneous
code, if and only if equality holds in Kraft’s inequality.
Suppose that C is an instantaneous code with maximum codeword
Length m. If C is not maximal, then it is possible to add a word of length
m to C without destroying its property of being instantaneous.

6. Theory of Information Lecture 6
Kraft’s Theorem
Theory of Information Lecture 6
THEOREM 1.4.2
There exists an instantaneous r-ary code with codeword lengths
a1,a2,…,aq if and only if Kraft’s inequality is satisfied:
1/ra1 + … + 1/raq  1.
Note: That a given code C satisfies Kraft’s inequality does not necessarily
mean that C is instantaneous; maybe another code D --- with the same
codeword lengths --- is instantaneous rather than C itself.
Example: C={0, 11, 100, 110}
Does C satisfy Kraft’s inequality?
Is C instantaneous?
However, D={0,11,101,100} has the same codeword lengths and
Is instantaneous.

7. The Utility of Kraft’s Theorem
Theory of Information Lecture 6
Kraft’s theorem allows us to find an instantaneous code (if such exists)
with given codelengths a1a2… aq as follows:
For i=1 to i=q do: Pick any codeword of length ai such that no earlier
codeword is its prefix
Example: Construct a ternary code {c1,c2,c3,c4,c5,c6} with codelengths
1, 1, 2, 4, 4, 5
Kraft’s test to see if such a code exists:
c1=
c2=
c3=
c4=
c5=
c6=

8. Reasonable Uniquely Decipherable Codes Are Instantaneous
Theory of Information Lecture 6
Reasonable Uniquely Decipherable Codes Are Instantaneous
THEOREM If a uniquely decipherable code exists with codeword
lengths l1,l2,…,ln, then an instantaneous code must also exist with these
same codeword lengths.
Proof: Suppose the above uniquely decipherable code exists. Then, by
McMillan’s Theorem, Kraft’s inequality should be satisfied for those
lengths. But then, by Kraft’s Theorem, there must also exist an
Instantaneous code with the same codelengths.

9. Theory of Information Lecture 6
Homework
Theory of Information Lecture 6
Exercises 2,3,4,5,6,7,8,9,10,11,12,13,14 of Section 1.4.