Erasure Correcting Codes In The Real World Udi Wieder Incorporates presentations made by Michael Luby and Michael Mitzenmacher.

Based On.. Practical Loss-Resilient Codes Michael Luby, Amin Shokrollahi, Dan Spielman, Bolker Stemann STOC ’97 Analysis of Random Processes Using And-Or Tree Evolution Michael Luby, Amin Shokrollahi SODA ’98 LT Codes Michael Luby STOC 2002 Online Codes Petar Maymounkov

Probabilistic Channels p p ? p p 1 1 1 1 1-p 1-p The binary erasure channel The binary symmetric channel

Erasure Codes Content Encoding Encoding Transmission Received Decoding cn Encoding Transmission ≥n Received Decoding n Content

Performance Measures Time Overhead Reception Efficiency The time to encode and decode expressed as a multiple of the encoding length. Reception Efficiency Ratio of packets in message to packets needed to decode. Optimal is 1.

Known Codes Random Linear Codes (Elias) Reed-Solomon Codes A linear code of minimum distance d is capable of correcting any pattern of d-1 or less erasures. Achieves capacity of the channel with high probability, i.e. can be used to transmit over erasure channel at any rate R<1-p. Decoding time O(n3). Unacceptable. Reed-Solomon Codes Optimal reception efficiency with probability 1. Decoding and Encoding in Quadratic time. (About one minute to encode 1MB).

Tornado Codes Practical Loss-Resilient Codes Michael Luby, Amin Shokrollahi, Dan Spielman, Bolker Stemann (1997) Analysis of Random Processes Using And-Or Tree Evolution Michael Luby, Amin Shokrollahi (1998)

Low Density Parity Check Codes Introduced in the early 60’s by Gallager and were reinvented many times. Check bits Message bits a b c d e f g h i j k l The time to encode is proportional to the number of edges.

Encoding Process. Bipartite Graph Bipartite Graph Standard Loss-Resilient Code. Bipartite Graph Bipartite Graph Length of message: k Check bits: Rate: 1-

Decoding Rule Given the value of a check bit and all but one of the message bits on which it depends, set the missing message bit to be the XOR of the check bit and its known message bits. XOR the message bit with all its neighbors. Delete from the graph the message bit and all edges to which it belongs. Decoding ends (successfully) when all edges are deleted.

Decoding Process a ? c d ? f ? ?

Decoding Process ? ? ? ?

Regular Graphs Random Permutation of the Edges Degree 3 Degree 6

3-6 Regular Graph Analysis left right left Pr[ not recovered] =  ¢ (1-(1-x)5)2 Pr[ all recovered] = (1-x)5 x = Pr[ not recovered ]

Decoding to Completion (sketch) Most message bits are roots of trees. Concentration results (edge exposure martingale) proves that all but a small fraction of message bits are decoded with high probability. The remaining bits are decoded do to expansion. (Original graph is a good expander on small sets). If a set of size s and average degree a has more than as/2 neighbors then a unique neighbor exists and decoding continues.

Efficiency Encoding time (sec), 1k packets size Reed-Solomon Tornado 4.6 0.06 500k 19 0.12 1 MB 93 0.26 2 MB 442 0.53 4 MB 1717 1.06 9 MB 6994 2.13 16 MB 30802 4.33 Decoding time (sec), 1k packets size Reed-Solomon Tornado 250k 2.06 0.06 500k 8.4 0.09 1 MB 40.5 0.14 2 MB 199 0.19 4 MB 800 0.40 9 MB 3166 0.87 16 MB 13829 1.75 Rate = 0.5 Erasure probability = 0.5 Implementation = ?

LT Codes LT Codes Michael Luby (2002)

‘Rateless’ Codes A different model of transmition. Sender sends an infinite sequence of encoding symbols. Time complexity: Average time for encoding a symbol. Erasures are independent of content. Receiver may decode when received enough symbols. Reception efficiency. ‘Digital Fountain’ approach.

Applications Unreliable Channels. Multi-source download. In Tornado codes small rate implies big graphs and therefore a lot of memory (proportional to the size of the encoding). Multi-source download. Downloading from different servers requires no coordination. Efficient exchange of data between users requires small rate of the source. Multi-cast without feedback (say over the internet). Rateless codes are the natural notion.

Trivial Examples - Repetition Each time unit send a random symbol of the code. Advantage: Encoding complexity O(1). Disadvantage: Need k’ = k ln(k/) code symbols to cover all k content symbols with failure probability at most . Example: k = 100,000,  =10-6 Reception overhead = 2400% (terrible)

Trivial Examples – Reed Solomon Each time unit send an evaluation of the polynomial on a random point. Advantage: Decoding possible when k symbols received. Disadvantage: Large time complexity for encoding and decoding.

Parameters of LT Codes Encoding time complexity O(ln n) per symbol. Decoding time complexity O(n ln n). Reception efficiency: Asymptotically zero (unlike Tornado codes). Failure probability: very small (smaller than Tornado).

LT encoding Content Degree Dist. Choose 2 random content symbols Choose degree XOR content symbols 2 Insert header, and send Degree Prob 1 0.055 0.0004 0.3 2 0.1 3 0.08 4 100000 Degree Dist.

LT encoding Content Degree Dist. Choose 1 random content symbol Choose degree Copy content symbol 1 Insert header, and send Degree Prob 1 0.055 0.0004 0.3 2 0.1 3 0.08 4 100000 Degree Dist.

LT encoding Content Degree Dist. Choose 4 random content symbols Choose degree XOR content symbols 4 Insert header, and send Degree Prob 1 0.055 0.0004 0.3 2 0.1 3 0.08 4 100000 Degree Dist.

LT encoding properties Encoding symbols generated independently of each other Any number of encoding symbols can be generated on the fly Reception overhead independent of loss patterns The success of the decoding process depends only on the degree distribution of received encoding symbols. The degree distribution on received encoding symbols is the same as the degree distribution on generated encoding symbols.

LT decoding Content (unknown) Collect enough encoding symbols and set up graph between encoding symbols and content symbols to be recovered Collect enough encoding symbols and set up graph between encoding symbols and content symbols to be recovered Identify encoding symbol of degree 1. STOP if none exists. Identify encoding symbol of degree 1. STOP if none exists. 3. Copy value of encoding symbol into unique neighbor, XOR value of newly recovered content symbol into encoding symbol neighbors and delete edges emanating from content symbol. 3. Copy value of encoding symbol into unique neighbor, XOR value of newly recovered content symbol into encoding symbol neighbors and delete edges emanating from content symbol. 4. Go to Step 2. 4. Go to Step 2.

Releasing an encoding symbol xth recovered content symbol releases encoding symbol x-1 recovered content symbols x-1 x k-x unrecovered content symbols content symbol can be recovered by encoding symbol i-2 encoding symbol of degree i

The Ripple Definition: At each decoding step, the ripple is the set of encoding symbols that have been released at any previous decoding step but their one remaining content symbol has not yet been recovered. x recovered content symbols x k-x unrecovered content symbols collision encoding symbols in the ripple

Successful Decoding Decoding succeeds iff the ripple never becomes empty Ripple small Small chance of encoding symbol collisions  small reception overhead Risk of ripple becoming empty due to random fluctuations is large Ripple large Large chance of encoding symbol collisions  large reception overhead Risk of ripple becoming empty due to random fluctuations is small

LT codes idea Control the release of encoding symbols over the entire decoding process so that ripple is never empty but never too large Very few encoding symbol collisions Very little reception overhead

Release probability Definition: Release probability for degree i encoding symbols at decoding step x is q(i,x). Proposition: For i = 1: q(i,x) = 1 for x = 0, q(i,x) = 0 for all x > 1 For i > 1: for x = i -1, …, k-1,

Release probability x-1 x i-2 xth recovered content symbol releases encoding symbol x-1 recovered content symbols x-1 x k-x unrecovered content symbols content symbol can be recovered by encoding symbol i-2 encoding symbol is released at decoding step x

Release distributions for specific degrees k = 1000

Overall release probability Definition: At each decoding step x, r(x) is the overall probability that an encoding symbol is released at decoding step x with respect to specific degree distribution p(·) Proposition:

Uniform release question Question: Is there a degree distribution such that the overall release distribution is uniform over x? Why interesting? One encoding symbol released for each content symbol decoded Ripple will tend to stay small  minimize reception overhead Ripple will tend not to become empty  decoding will succeed

Uniform release answer: YES! Ideal Soliton Distribution:

Ideal Soliton Distribution k = 1000

A simple way to choose from Ideal SD Choose A uniformly from the interval [0,1) If then degree Else degree = 1. Degree 1/k 6 5 4 3 2 Value of A 1/6 1/4 1/3 1/2 1 1/k 1/5

Ideal SD theorem Ideal SD Theorem: The overall release distribution is exactly uniform, i.e., r(x) = 1/k for all x = 0,…,k-1.

Overall release distribution for Ideal SD k = 1000

In expected value … Optimal recovery with respect to Ideal SD Receive exactly k encoding symbols Exactly one encoding symbol released before any decoding steps, recovers one content symbol At each decoding step a content symbol is recovered, it releases exactly one new encoding symbol, which in turn recovers exactly one more content symbol Ripple size always exactly 1 Performance Analysis No reception overhead Average degree

When taking into account random fluctuations … Ideal Soliton Distribution fails miserably Expected behavior not equal to actual behavior because of variance Ripple very likely to become empty Fails with very very high probability (even with high reception overhead)

Robust Soliton Distribution design Need to ensure that the ripple never empties At the beginning of the decoding process ISD: ripple is not large enough to withstand random fluctuations RSD: boost p(1)=c/ sqrt{k} so that expected ripple size at beginning is c *sqrt{k} At the end of the decoding process ISD: expected rate of adding to the ripple not large enough to compensate for collisions towards the end of the decoding process when ripple is large relative to the number of unrecovered content symbols RSD: boost p(i) for higher degrees i so that expected ripple growth at the end of the decoding process is higher

LT Codes – Bottom line Using the Robust Soliton Distribution: Number of symbols needed to recover the data with probability  is: The average degree of an encoding symbol is:

Online Codes Petar Maymounkov We are out of time Online Codes Petar Maymounkov