1. The Cutoff Rate and Other Limits: Passing the Impassable Richard E. Blahut University of Illinois UIUC 5/4/20151

2. Shannon’s Ideal Channel Example: Binary Memoryless Channel Stationary Discrete Memoryless 5/4/20152

3. 0 1 0 0 1 1 0 1 0 1 0 … 1 1 1 0 0 1 0 1 1 0 1 … 0 1 1 1 0 0 1 0 1 1 1 … 1 0 1 ….. 1 1 1 1 1 0 0 0 1 0 0 … … 0 0 0 1 1 0 0 …. 1 1 … … 0 0 0 … 0 0 1 … 0 1 0 … 0 1 1 … 1 1 1 A Large Code 5/4/20153

4. + + + A convolutional encoder 5/4/20154

5. Information theory asserts existence of good codes Coding theory wants practical codes and decoders There are binary codes 5/4/20155

6. Brief History of Codes Algebraic Block Codes 1948 Reed-Solomon codes (1960) Convolutional Codes 1954 Sequential decoding (1951) Viterbi algorithm (1967) Euclidean Trellis Codes 1982 Turbo Codes 1993 Gallager (LDPC) codes (1960) 5/4/20156

7. Decoders Maximum Likelihood Maximum Block Posterior Maximum Symbol Posterior Typical Sequence Iterative Posterior Minimum Distance Bounded Distance 5/4/20157

8. My View 1)Channel Capacity 2)Cutoff Rate 3)Critical Rate Distance -based codes Likelihood -based codes Posterior -based codes Polar codes 5/4/20158

9. For any fixed there is a sequence of codes for which exponentially in blocklength. This sequence does not approach Channel Error Exponent Fact #2 Every code satisfies Fact #1 Codes exist such that 5/4/20159

10. E(R) 5/4/201510

11. A sequence of codes drawn from a set of ensembles 5/4/201511

12. Channel Capacity Channel Critical Rate Channel Cutoff Rate 5/4/201512

13. Binary Hypotheses Testing Type 1 Error Type 2 Error 5/4/201513

14. Binary Hypotheses Testing Change Notation 5/4/201514

15. Bounds onand Upper Bounds on Sphere Packing Bound Minimum Distance Bound Lower Bounds on Random Coding Bound Expurgated Bound 5/4/201515

16. Bhattacharyya Distance 5/4/201516

17. 175/4/2015

18. is quadratic near Let be a sequence with Then with so A Code Sequence Approaching Capacity with if 5/4/201518

19. Capacity: C Shannon(1948) Cutoff Rate: Jacobs & Berlekamp(1968) Massey(1981) Arikan(1985/1988) Error Exponent: Gallager(1965) Forney(1968) Blahut(1972) 5/4/201519

20. Gallager (1965) Forney (1968) Blahut (1972) where is the Kullback divergence 5/4/201520

21. Forney’s List Decoding Likelihood Function Likelihood Ratio 5/4/201521

22. Sequential Decoding Exponential waiting time Work exponential in time Pareto Distribution with Work unbounded if Sequential decoding fails if Is maximum likelihood decoding sequential decoding? 5/4/201522

23. Two Pareto parameters and Pareto Distribution 5/4/201523

24. Start with an exponential distribution If is exponential, then is a Pareto distribution The Origin of a Pareto Distribution 5/4/201524

25. The Origins of Graph-Based Codes Brillouin deBrogle Shannon Battail (1987) Hagenauer (1989) Berrou et al (1993) 5/4/201525

26. Coding Beyond the Cutoff Rate Parallel – Pinsker Hybrid – Jelinek Turbo – Berrou/Glavieux LDPC – Gallager/Tanner/Wiberg Polar - Arikan 5/4/201526

27. The Massey Distraction (1981) QECBEC QEC 2 BEC 5/4/201527

28. Performance Measures Bit Error Rate vs. Message Error Rate 5/4/201528

29. The Arikan Retraction 5/4/201529

30. 5/4/201530

31. 5/4/201531

32. The Arkan Redistraction* *Rhetorical 5/4/201532