1. CS774. Markov Random Field : Theory and Application Lecture 10 Kyomin Jung KAIST Oct 06 2009

2. Review: Hamming Code For each message block of size 4, we will produce a 7 bit codeword by: where the operation is over

3. Review: Hamming Code Claim: If then and differ in least 3 coordinates. Hence we can correct any one bit error. But how? Now, let H be Then,

4. Hamming Code Claim: If y has only 1 or 2 non-zero entry, then Hence, any two codewords have distant at least 3. (minimum distance is 3)

5. Decoding of Hamming Code We can check if an error happened by varifying whether. If we have an error, we need to determine which bits were flipped.

6. LDPC(Low Density Parity Check) Code Discovered by Gallager(1963), rediscovered later by Neal & Mackay (MN codes) and by Sipser & Spielman (Expander codes) State of the art codes that exhibit Near Shannon limit performance. Practical - Simple decoding algorithms based on Message- Passing decoding:  low decoding complexity  allow parallel implementation – enabling high data rates Flexibility in choice of parameters make it possible to design appropriate LDPC codes for many communication scenarios.

7. LDPC Code H= m n 111111 111111 111111 111111 Parity-Check Matrix A (c,d)-LDPC code is a linear block code represented by a sparse parity-check matrix H.  Each row of H contains at most c many 1’s  Each column of H contains at most d many 1’s. Why Sparse matrix is useful?  It gives a lot of information about where the error happened.

8. LDPC Code R Parity Check Nodes d=6 L Variable nodes c=3 n m The code can also be represented by a bipartite graph B. The left side nodes (variable nodes) represent the codeword bits. The right side nodes (check nodes) represent the parity- check constraints on the codeword bits. c is the maximum degree of the left vertices. d is the maximum degree of the right vertices. m n 111111 111111 111111 111111

9. Encoding / Decoding of LDPC Codes Encoding is a matrix operation.(multiplication by G)  Send the encoded codeword. Decoding is to find the most probable vector x such that xH mod 2 = 0 w.r.t. Hamming distance. How to decode?  By a flipping algorithm, whose implementation is similar to BP.

10. Expander graph When the codes have large minimum distance?  When B is a expander graph.  For a set let be the set of nodes adjacent to at least one node in S and let T1(S) be the set of nodes adjacent to exactly one node in S.  A bipartite graph B is called an (r,s)-expander if every set with has.

11. Expander graph A bipartite graph B=(L,R,E) is (c,d) regular if every vertex in L has degree at most c and every vertex in R has degree at most d. If B=(L,R,E) is a (c,d)-regular and a (r,s)-expander, then  for all such that, (Lemma 1)  Moreover, if r>c/2, then the code corresponding to B has minimum distance at least sn. (Lemma 2)

12. Decoding algorithm Flip Algorithm [Sipser-Spielman ‘96]  While there exists a left vertex v with more violated neighbors than unviolated neighbors, flip v. At each iteration # of violated right vertices decreases. If there are k initial violated constraints, Flip Algorithm terminates within k many iterations.

13. Decoding algorithm Termination possibilities: 1. Terminate with the sent codeword y. 2. Terminate with a wrong codeword y’. 3. Terminate with a non-codeword z. We show that 2, 3 does not happen if number of errors is smaller than sn/2c and.

14. Correctness proof 2 does not happen:  Let y be the transmitted codeword  r be the received vector  z be the assignment to the left side of B when the Flip Algorithm terminates.  Hence z cannot be a wrong codeword since the code has distance at least sn by Lemma 2.

15. Correctness proof 3 does not happen:  Let x=y+z  Since y is a codeword, x has the same assignment as z on the right side.  Let S be the set of non-zero left side vertices of x.  Then  By lemma 1,  Hence there must be a vertex in S that has more violated constraints than unviolated constraints, which contradicts the stopping rule of the Flip Algorithm.