Lecture 6 Instantaneous Codes and Kraft’s Theorem (Section 1.4) Theory of Information Lecture 6 Theory of Information Lecture 6 Instantaneous Codes and Kraft’s Theorem (Section 1.4)

Theory of Information Lecture 6 Instantaneous Codes Theory of Information Lecture 6 DEFINITION A code is said to be instantaneous if, whenever any sequence of codewords is transmitted, each codeword can be interpreted as soon as it is received. Example. {0, 01} {0, 10}

Prefix and Suffix Properties Theory of Information Lecture 6 DEFINITION 1) A code is said to have the prefix property if no codeword is a prefix of any other codeword. 2) A code is said to have the suffix property if no code word is a suffix of any other codeword. Fact: Having the suffix or prefix property is sufficient to be uniquely Decipherable. But not vice versa. Does {0,01} have the prefix property? suffix property? Is it uniquely decipherable? Does {00, 001000, 001011, 11} have the prefix property? suffix property? Is it uniquely decipherable?

Prefix Property = Instantaneousity Theory of Information Lecture 6 THEOREM 1.4.1 A code is instantaneous if and only if it has the prefix property. Comma code: {0, 10, 110, 1110} has the prefix property and is instantaneous. How about {0, 01, 011, 0111}: Prefix property? Instantaneous? Suffix property? Uniquely decipherable?

Theory of Information Lecture 6 Kraft’s Theorem Theory of Information Lecture 6 THEOREM 1.4.2 There exists an instantaneous r-ary code with codeword lengths a1,a2,…,aq if and only if Kraft’s inequality is satisfied: 1/ra1 + … + 1/raq  1. Let C be an instantaneous r-ary code. Then C is maximal instantaneous, i.e. C is not contained in any strictly larger instantaneous code, if and only if equality holds in Kraft’s inequality. Suppose that C is an instantaneous code with maximum codeword Length m. If C is not maximal, then it is possible to add a word of length m to C without destroying its property of being instantaneous.

Theory of Information Lecture 6 Kraft’s Theorem Theory of Information Lecture 6 THEOREM 1.4.2 There exists an instantaneous r-ary code with codeword lengths a1,a2,…,aq if and only if Kraft’s inequality is satisfied: 1/ra1 + … + 1/raq  1. Note: That a given code C satisfies Kraft’s inequality does not necessarily mean that C is instantaneous; maybe another code D --- with the same codeword lengths --- is instantaneous rather than C itself. Example: C={0, 11, 100, 110} Does C satisfy Kraft’s inequality? Is C instantaneous? However, D={0,11,101,100} has the same codeword lengths and Is instantaneous.

The Utility of Kraft’s Theorem Theory of Information Lecture 6 Kraft’s theorem allows us to find an instantaneous code (if such exists) with given codelengths a1a2… aq as follows: For i=1 to i=q do: Pick any codeword of length ai such that no earlier codeword is its prefix Example: Construct a ternary code {c1,c2,c3,c4,c5,c6} with codelengths 1, 1, 2, 4, 4, 5 Kraft’s test to see if such a code exists: c1= c2= c3= c4= c5= c6=

Reasonable Uniquely Decipherable Codes Are Instantaneous Theory of Information Lecture 6 Reasonable Uniquely Decipherable Codes Are Instantaneous THEOREM 1.4.3. If a uniquely decipherable code exists with codeword lengths l1,l2,…,ln, then an instantaneous code must also exist with these same codeword lengths. Proof: Suppose the above uniquely decipherable code exists. Then, by McMillan’s Theorem, Kraft’s inequality should be satisfied for those lengths. But then, by Kraft’s Theorem, there must also exist an Instantaneous code with the same codelengths.

Theory of Information Lecture 6 Homework Theory of Information Lecture 6 Exercises 2,3,4,5,6,7,8,9,10,11,12,13,14 of Section 1.4.