Capacity Upper Bounds for Deletion Channels Suhas Diggavi Michael Mitzenmacher Henry Pfister

The Most Basic Channels Binary erasure channel. –Each bit is replaced by a ? with probability p. Binary symmetric channel. –Each bit flipped with probability p. Binary deletion channel. –Each bit deleted with probability p.

The Most Basic Channels Binary erasure channel. –Each bit is replaced by a ? with probability p. –Very well understood. Binary symmetric channel. –Each bit flipped with probability p. –Very well understood. Binary deletion channel. –Each bit deleted with probability p. –We dont even know the capacity!!!

The Challenge Would like a single-letter characterization of capacity. –Or tight upper/lower bounds. –Or effective means of calculating capacity. Such characterizations are difficult. –Deletion channels are channels with memory.

Recent Progress Chain of results giving better lower bounds. –Shannon-style arguments. Diggavi/Grossglauser, Drinea/Mitzenmacher, Drinea/Kirsch. Global result: capacity is at least (1 – p)/9. But essentially no work on upper bounds. –Ullmans bound: zero-error decoding for worst-case synchronization errors. (Does not apply.) –Trivial bound of (1 – p). –Lower bound progress motivates need for progress in the other direction. –How close are we getting to capacity???

Our Results An upper bound approach using side information that gives numerical bounds. An upper bound approach using side information that gives asymptotic behavior as fraction of deletions p goes to 1, for a bound of c(1 – p).

Upper Bound via Run Lengths Input can be thought of as runs of maximally contiguous 0s/1s. Side information: Suppose an undeletable marker inserted every time a complete run is deleted. –Can only increase capacity. Equivalent to a memoryless channel where symbols are run lengths ….

Example …. Sent: Received: 0011*11000*00… …. Sent: Received: ….

Capacity Per Unit Cost Associate cost of x with run of length x at input. Capacity of binary channel with side info = Capacity per unit cost of run length channel. Latter can be upper bounded numerically using Kuhn-Tucker conditions. –Challenging because of infinite alphabet.

Upper Bound Statement For channel defined by p Y | X and any output distribution q Y let Then for any non-negative cost function c(x), the capacity per unit cost C satisfies [Abdel-Ghaffar 1993]

Upper Bound Calculation Problem: Optimization over infinite alphabet. Solution: Finitize the problem. –Find optimal input/output distribution for truncated alphabet. –Replace tail of finite output distribution with geometric distribution. To allow analytic bound on for large x. –Bound performance of resulting distribution. –Optimize over truncated alphabet.

Bounds Achieved pLower boundUpper bound

Asymptotic Result Motivation: –Previous upper bound approach breaks down for large p. Not surprising; large p implies more completely deleted runs, so more side information released. –Want to find limits of possible global results. The (1 – p)/9 lower bound argument seems tightest as p approaches 1. Can we obtain an asymptotic c(1 – p) upper bound? –Build upon insights from global lower bound.

Upper Bound via Markers Suppose an undeletable marker is inserted every 1/(1 – p) bits in the transmission. Channel now memoryless. –Input symbols = 1/(1 – p) bits. –Output symbols = random subsequence, with expected length 1. –Capacity should scale with (1 – p). –Capacity bound: How can we optimize over such a large dimensional space? –Symbols are big.

Upper Bound Calculation : Step 1, Output Problem: Optimization over all output symbols. –Potentially infinitely many bit strings. Solution: Finitize the problem. –At receiver, number of bits between markers converges to Poisson distribution. –For upper bounds, assume that if k > 6 bits received, then receiver obtained k bits of information. Only affects bounds by a few percent. –Only need to consider outputs of 6 or fewer bits.

Upper Bound Calculation : Step 2, Input Problem: Optimization over input strings. –Sequence of 1/(1 – p) bits. Potentially infinite alphabet. Solution: Finitize the problem. –Key Lemma: if only considering up to 6 bits at output, need only consider sequences of up to 6 runs at input. Same output distribution achieved by convex combination. –Upper bound achieved by optimizing over large number of finite-dimensional vectors representing up to 6 runs. Heuristic/computational approach.

Bounds Achieved As p goes to 1, cannot obtain capacity better than (1 – p). Gap between (asymptotic) upper/lower bounds now roughly (1 – p)/9 and 4(1 – p)/5. –Room for improvement, probably both sides.

Conclusions and Open Questions What are the limitations of such side information arguments? Are novel upper bound techniques required for these channels? Is there a more purely information theoretic approach? Can we characterize optimal input/output distributions? Heuristic/other approaches to guide theory? How tight can we make upper/lower bounds? What is the right answer? Extend upper bounds for insertion channels? Many, many others…