AN IFORMATION THEORETIC APPROACH TO BIT STUFFING FOR NETWORK PROTOCOLS Jack Keil Wolf Center for Magnetic Recording Research University of California, San Diego La Jolla, CA DIMACS Workshop on Network Information Theory Rutgers University March 17-19, 2003

Acknowledgement Some of this work was done jointly with: Patrick Lee Paul Bender Sharon Aviran Jiangxin Chen Shirley Halevy (Technion) Paul Siegel Ron Roth (Technion)

Constrained Sequences and Network protocols In many protocols, specific data patterns are used as control signals. These prohibited patterns must be prevented from occurring in the data. In any coding scheme that prohibits certain patterns from occurring, the number of constrained (or coded) bits must exceed the number of data bits.

Relationship to Information Theory The average number of data bits per constrained bit is called the rate of the code. The Shannon capacity is the maximum rate of any code. Practical codes usually have rates strictly less than the Shannon capacity.

Bit Stuffing Bit stuffing is one coding technique for preventing patterns from occurring in data. The code rate for bit stuffing is always less than the Shannon capacity. Here we show how to make the code rate for bit stuffing equal to the Shannon capacity.

Bit Stuffing and Protocols Definition from Webopedia: ( “bit stuffing-The practice of adding bits to a stream of data. Bit stuffing is used by many network and communication protocols for the following reasons: To prevent data being interpreted as control information. For example, many frame- based protocols, such as X.25, signal the beginning and end of a frame with six consecutive 1’s. Therefore, if the actual data has six consecutive 1 bits in a row, a zero is inserted after the first 5 bits… Of course, on the receiving end, the stuffed bits must be discarded…” data: … This won’t work: transmit: … But this will: transmit: …

A Diversion: Binary (d,k) Constrained Sequences The X.25 constraint is a special case of a binary (d,k) constraint used in digital storage systems. Such binary sequences have at least d and at most k 0’s between adjacent 1’s. For d > 0 and finite k, the sequences are produced by the edge labels when taking tours of the graph: 01d-1dd+1k-1k … …

Binary (0, k ) Constrained Sequences For d=0 and finite k, allowable sequences are produced by the edge labels of the graph: 012k-1k

Binary (d, ∞ ) Constrained Sequences For infinite k, the sequences are produced by the edge labels when touring the graph: 012d-1d  0   

Back to Protocols By complementing the bits in a (0,5) code, we will produce sequences that have no more than 5 consecutive 1’s. Thus, after complementing the bits, any (0,5) code can be used in the X25 protocol.

Bit Stuffing In this talk we will investigate the code rates which can be achieved with bit stuffing and compare these rates with the Shannon capacity. We will use as our constraint, binary (d,k) codes, although our technique applies to a much wider class of codes. We will begin with plain vanilla bit stuffing which gives rates strictly less than capacity. Then we show how bit stuffing can be modified to yield rates equal to capacity for some values of d and k. Finally we show how bit stuffing can be further modified to yield rates equal to capacity for all values of d and k.

Bit Stuffing for (d,k) Codes For any value of d and k (0 < d < k), one can use bit stuffing to form sequences that satisfy the constraint. The bit stuffing encoding rule is: Step 1. If last bit is a 1, stuff d 0’s. Go to next step. (Skip this step if d=0.) Step 2. If last k bits are 0’s stuff a 1. Return to first step. (Skip this step if k=∞.) 01d-1dd+1k-1k … …

Rate for Bit Stuffing vs Shannon Capacity of (d,k) Codes The rate for bit stuffing is the average number of information bits per transmitted symbol. The rate here is computed for i.i.d. binary data with equally likely 0’s and 1’s. The Shannon capacity of a (d,k) constrained sequence is the maximum rate of any encoder-decoder that satisfies the constraint. Therefore the rate for bit stuffing is less than or equal to the Shannon capacity of the constraint.

Shannon Capacity of a (d,k) Constraint Define N(n) as the number of distinct binary sequences of length n that satisfy the constraint. Then, for every 0 < d < k, exists and is called the Shannon capacity of the code.

Shannon Capacity Shannon (1948) gave several methods for computing the capacity of (d,k) codes. For finite k, he showed that the following difference equation describes the growth of N(n) with n: N(n)=N(n-(d+1))+N(n-(d+2))+ … +N(n-(k+1)). By solving this difference equation, Shannon showed that the capacity, C = C(d,k), is equal to the base 2 logarithm of the largest real root of the equation: x k+2 - x k+1 - x k-d+1 +1 = 0.

Bit Stuffing and Shannon Capacity If one uses bit stuffing on uncoded data, except for ` the trivial case of (d=0, k=∞), the rate always is strictly less than the Shannon capacity. The rate here is computed for i.i.d. binary data with equally likely 0’s and 1’s. But by a modification to bit stuffing, using a distribution transformer, we can improve the rate and sometimes achieve capacity.

Slight Modification to Bit Stuffing A distribution transformer converts the binary data sequence into an i.i.d. binary sequence that is p- biased for 0 < p < 1. The probability of a 1 in this biased stream is equal to p. The distribution transformer can be implemented by a source decoder for a p-biased stream. The conversion occurs at a rate penalty h(p), where h(p) = -plog(p)-(1-p)log(1-p). We can choose p to maximize the code rate and sometimes achieve capacity.

Bit Stuffing with Distribution Transformer Distribution Transformer p-Bias … Bit Stuffer … Distribution Transformer p-Bias … … ½ 1-p p Inverse Distribution Transformer … … Bit Unstuffer

Slight Modification to Bit Stuffing As shown by Bender and Wolf, after optimizing p, the code rate can be made equal to the Shannon capacity for the cases of (d, d+1) and (d,∞), sequences for every d > 0. However, even after choosing the optimum value of p, the code rate is strictly less than the Shannon capacity for all other values of d and k.

Code Rate vs. p (B&W)

Code Rate/Capacity vs k for Fixed d: Optimum p (B&W)

Two Questions Why does this technique achieve capacity only for the cases: k = d+1 and k = ∞ ? Is it possible to achieve capacity for other cases? To answer these questions we make a slight diversion.

A Further Diversion: Bit Stuffing and 2-D Constraints Bit stuffing has been used to generate two dimensional constrained arrays. Details of this work are in a series of papers, the latest entitled “Improved Bit-Stuffing Bounds on Two-Dimensional Constraints” which has been submitted to the IEEE Transactions on Information Theory by: Shirley HalevyTechnion Jiangxin ChenUCSD Ron RothTechnion Paul SiegelUCSD MeUCSD

Two Dimensional Constraints Two dimensional constrained arrays can have applications in page oriented storage. These arrays could be defined on different lattices. Commonly used are the rectangular lattice and the hexagonal lattice. Example 1: Rectangular lattice with a (1, ∞) constraints on the rows and columns: Example 2: Hexagonal lattice with a (1, ∞) constraints in 3 directions:

Capacity and Two Dimensional Constrained Arrays Calculating the Shannon capacity for two dimensional constrained arrays is largely an open problem. The exact value of the capacity for the rectangular (1, ∞) constraint is not known. However, Baxter has obtained the exact value of the capacity of the hexagonal (1, ∞) constraint. In some cases where the two dimensional capacity is not known, we have used bit stuffing to obtain tight lower bounds to the capacity.

Two Dimensional Bit Stuffing: Rectangular Lattice with (1, ∞) Constraint A distribution transformer is used to produce a p- biased sequence. The p-biased sequence is written on diagonals Every time a p-biased 1 is written, a 0 is inserted (that is stuffed) to the right and below it. In writing the p-biased sequence on diagonals, the positions in the array containing stuffed 0’s are skipped.

Bit Stuffing and Two Dimensional Constrained Arrays Suppose we wish to write the p-biased sequence …

“Double Stuffing” with the (1, ∞) Constraint Sometimes a p-biased 1 results in only one stuffed 0 since there is already a stuffed 0 to the right of it. In writing the sequence …, 1 5 results in only a single stuffed 0, since 1 4 having been written above and to the right of it, already has written the other 0. That is, 0 4,5 is a “double” stuffed ,

Multiple p-Biased Transformers This suggests having two values for p: one, p 0, for the case where the bit above and to the right of it is a 0 and the other, p 1, when that bit is a 1. Doing this and optimizing we obtain: p 0 = p 1 = code rate= (which is within 0.1% of capacity) This suggests using multiple p’s in one dimension to improve the code rate.

Shannon Capacity and Edge Probabilities The maximum entropy (i.e., the capacity) of a constraint graph induces probabilities on the edges of the graph. For finite k, the Shannon capacity is achieved when the edges of the graph are assigned the probabilities as indicated below where C = log( ). 012d-1d -(d+1) d+1 (1- -(d+1) ) kd+2  -(d+2) / (1- -(d+1) )} -(d+2) / (1- -(d+1) )  

Shannon Capacity and Edge Probabilities And for the (d, ∞) constraint, the Shannon capacity is achieved when the edges of the graph are assigned the probabilities as indicated: 012d-1d -(d+1) 1- -(d+1)   

Why Bit Stuffing Sometimes Achieved Capacity for B&W The graphs for the two cases of constraints that achieved capacity are shown below : Note that for both graphs, only one state has two edges emanating from it. Thus, only one bias suffices and the optimum p for both cases is: p= -(d+1). For other values of d and k, there will be more than one state with two exiting edges. 1  -(d+1) 1 1 -(d+1) (d,d+1) Constraint (d, ∞ ) Constraint -(d+1) 11

Capacity Achieving Bit Stuffing This suggests a better scheme which achieves capacity for all values of d and k. For k finite, there are (k-d) states in the graph with two protruding edges. The binary data stream is converted into (k-d) data streams, each by a different distribution transformer. The p’s of each of the transformers are chosen to emulate the maxentropic edge probabilities for the the (k-d) states with two protruding edges.

Block Diagram of Encoder Distribution Transformer p d -Bias Distribution Transformer P d+1 -Bias Distribution Transformer p (k-1) -Bias Smart DeMux Bit Stuffer Smart Mux

Bit Stuffing with Average Rate Equal to the Shannon Capacity Example: (1,3) Code The maxentropic probabilities for the branches are: Thus, one distribution transformer should have p= and the second distribution transformer should have p= Run Length Probabilities LengthProbability

Bit Stuffing with Average Rate Equal to the Shannon Capacity Example: (2,4) Code The maxentropic probabilities for the branches are: Thus, one distribution transformer should have p= and the second distribution transformer should have p= But only one distribution transformer is needed. Why? Run Length Probabilities LengthProbability

Bit Flipping and Bit Stuffing For the (2,4) case, one can use one distribution transformer and bit flipping in conjunction with bit stuffing to achieve capacity. For k finite, we next examine such a system for arbitrary (d,k): Distribution Transformer p-Bias Bit Stuffer Controlled Bit Flipper

Questions What is the optimal bit flipping position? When can we improve the rate by bit flipping? Can we achieve capacity for more constraints, using bit flipping? If not, how far from capacity are we? 01dd+1kk-1 p 11 1-p ?? p... pp 1-p 1 p

Answers (Aviran) For all (d,k) with d≥1, d+2≤k<∞ and p<0.5: – the optimal flipping position is k-1. For all (d,k) with d≥1, d+2≤k<∞: – flipping improves the rate over original bit stuffing. Capacity is achieved only for the (2,4) case. 01dd+1kk-1 p 1 1-p... p 1-p 1 p 1 k-2 1-p p

Numerical Results

Topics Missing From This Talk A lot of interesting mathematics. Results for more general one and two dimensional constraints. A list of unsolved problems.