Probabilistic verification Mario Szegedy, Rutgers www/cs.rutgers.edu/~szegedy/07540 Lecture 4

Codes I (outline) Madhu’s notes : It all started with Shannon: binary symmetric (noisy) channels Malicious (rather than random) errors Error correcting codes and their parameters Papadimitriou’s notes: Reed-Solomon (RS) codes Berlekamp-Welsh (BW) decoding algorithm for RS codes Madhu’s notes: A generalization of the BW decoding algorithm by Sudan My notes: Generalized Reed-Solomon codes and its parameters Self-correcting properties of the generalized RS codes (local decodability)

Claude Shannon 1916 – 2001 In 1948 Shannon published A Mathematical Theory of Communication article in two parts in the July and October issues of the Bell System Technical Journal.A Mathematical Theory of CommunicationBell System Technical Journal This work focuses on the problem of how best to encode the information a sender wants to transmit. In this fundamental work he used tools in probability theory, developed by Norbert Wiener, which were in their nascent stages of being applied to communication theory at that time.informationNorbert Wiener Shannon developed information entropy as a measure for the uncertainty in a message while essentially inventing the field of information theory.information entropyinformation theory

some channel models Input X P(y|x)output Y transition probabilities memoryless: - output at time i depends only on input at time i - input and output alphabet finite

Example: binary symmetric channel (BSC) Error Source + E X Output Input E is the binary error sequence s.t. P(1) = 1-P(0) = p X is the binary information sequence Y is the binary output sequence 1-p 0 p 1 1-p

Other models (light on) 1 (light off) p 1-p X Y P(X=0) = P E10E1 1-e e 1-e P(X=0) = P 0 Z-channel (optical) Erasure channel (MAC)

Erasure with errors E10E1 p p e e 1-p-e

General noisy channel is an arbitrary bipartite graph p ij is the probability that upon sending i the receiver receives j. (p ij ) is a stochastic matrix. p 22 p 35 p 11

Original message + redundancy Encoded messageReceived message - decoding Original message

Encoded message m E(m) Rate of encoding R = |m| / |E(m)| For a fixed channel is there a fixed rate that allows almost certain recovery? (m→ infinity)

Madhu’s notes : 4.pdf

THEOREM: For binary symmetric channel with error probablity p we can find code with rate: 1-H(p), where H(p) = p log 2 p + (1-p) log 2 (1-p) (Entropy function) DEFINITIONS: Hamming distance of two binary strings is Δ(x,y) Hamming ball with radius r around binary string x is B(x,r) BASIC: For r=pn we have |B(x,r)| ≈ 2 H(p)n x r

ENCODING: (R = 1-H(p) ) E: {0,1} Rn → {0,1} n, random DECODING: Given y = E(m) + error, find the unique m’ such that Δ(y,E(m’)) ≤ pn, if such m’ exists, otherwise decode arbitrarily. ANALYSIS: Pr[decoding error] ≤ Pr[ Δ(y,E(m) ≥ pn ] + Pr[ many codewords in B(y,pn)] ≤ small + 2 H(p)n 2 Rn 2 -n y pn E(m’) A B Pr[ A, B intersect ]= |A||B|/|X| (A is random) x

No better rate is possible Transmit random messages. Decoding error is large for any code with rate > 1-H(p). PROOF: The decoding procedure D partitions {0,1} n into 2 Rn parts. {0,1} n Distribution of E(m) + error Decodes to m

2 n / 2 Rn versus 2 H(p)n R ≤ 1-H(p)

C= (n,k,d) q codes n = Block length k = Information length d = Distance k/n = Information rate d/n = Distance rate q = Alphabet size

Linear Codes C = {x | x Є L} (L is a linear subspace of F q n ) Δ(x,y) = Δ(x-z,y-z) min Δ(0,x) = min Δ(x,y) x,y Є L k = dimension of the message space = dim L n = dimension of the whole space (in which the code words lie) Generator matrix: {xG | x Є ∑ k } “Parity” check matrix: {y Є ∑ n | yH = 0}

Reed-Solomon codes The Reed–Solomon code is an error-correcting code that works by oversampling a polynomial constructed from the data.error-correcting code oversamplingpolynomial C is a [n, k, n-k+1] code; in other words, it is a linear code of length n (over F) with dimension k and minimum distance n-k+1.linear codedimension