1. Coding Theory Efficient and Reliable Transfer of Information
Information Theory
Algebraic Coding Theory

2. Information Source
Encoder
Communication Channel
Decoder
Information Sink
Noise

3. What is Information

4. Information Theory Information is a decrease in uncertainty
What is uncertainty?

5. Uncertainty Number of Possibilities 2 ways 3 ways Together: 6 ways
Prefer additive measure
take logarithm of number of symbols M

6. Symbols with unequal probability

7. Average uncertainty per symbol: Shannon Entropy

9. Mutual Information Change in uncertainty
Change in uncertainty in one random variable from knowledge of another

10. Molecular Information Theory
Applied to DNA Sequences
Pattern of information, or label, to ribosome to start translation at given site

11. Channel Capacity
Maximum rate at which information can be sent and recovered with vanishing probability of error
Channel Coding Theorem

12. Source Coding Can’t Compress information below entropy
Optimal Prefix-Free Codes for Representation
Code Lengths are greater than or equal to entropy of source

13. Rate Distortion Encoding bits of information per symbol bit
How many given a certain distortion (error) p 1
1-p

14. Algorithmic Information Theory
Kolmogorov
Descriptive complexity of an object is the length of the shortest computer program that describes the object
If equal to length of object then it is random Pi