Lower Bounds & Models of Computation Jeff Edmonds York University COSC 3101 Lecture 8

Lower Bounds for Sorting using Information Theory

The Time Complexity of a Problem P Merge, Quick, and Heap Sort can sort N numbers using O(N log N) comparisons between the values. Theorem: No algorithm can sort faster. The Time Complexity of a Problem P: The minimum time needed by an algorithm to solve it.

The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it. Problem P is computable in time T upper (n) if there is an algorithm A which outputs the correct answer in this much time Eg: Sorting computable in T upper (n) = O(n 2 ) time.  A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|) Upper Bound:

Understand Quantifiers!!! One girl Could be a separate girl for each boy. SamMary BobBeth John Marilin Monro FredAnn SamMary BobBeth John Marilin Monro FredAnn Algorithm Input  A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|)

The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it.  A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|)  I,  A, A(I)=P(I) and Time(A,I)  T upper (|I|) What does this say? True for any problem P and time T upper. Given fixed I. Its output is P(I). Let A P(I) be the algorithm that outputs the string P(I).

The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it. Time T lower (n) is a lower bound for problem p if no algorithm solve the problem faster. There may be algorithms that give the correct answer or run quickly on some inputs instance. Lower Bound:

The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it. Lower Bound: Time T lower (n) is a lower bound for problem p if no algorithm solve the problem faster. Eg: No algorithm can sort N values in T lower = sqrt(N) time. But for every algorithm, there is at least one instance I for which either the algorithm gives the wrong answer or it runs in too much time.  A,  I, A(I)  P(I) or Time(A,I)  T lower (|I|)

Understand Quantifiers!!! One girl Could be a separate girl for each boy. SamMary BobBeth John Marilin Monro FredAnn SamMary BobBeth John Marilin Monro FredAnn Algorithm Input  A,  I, A(I)  P(I) or Time(A,I)  T lower (|I|)

 A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|)  A,  I, A(I) ≠ P(I) or Time(A,I)  T lower (|I|) Lower Bound: Upper Bound: The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it. “There is” and “there isn’t a faster algorithm” are almost negations of each other.

 A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|) Upper Bound: Prover-Adversary Game I have an algorithm A that I claim works and is fast. I win if A on input I gives the correct output in the allotted time. Oh yeah, I have an input I for which it does not. What we have been doing all along.

Lower Bound: Prover-Adversary Game I win if A on input I gives the wrong output or runs slow.  A,  I, [ A(I)  P(I) or Time(A,I)  T lower (|I|)] Proof by contradiction. I have an algorithm A that I claim works and is fast. Oh yeah, I have an input I for which it does not.

Lower Bound: Prover-Adversary Game  A,  I, [ A(I)  P(I) or Time(A,I)  T lower (|I|)] I have an algorithm A that I claim works and is fast. Lower bounds are very hard to prove, because I must consider every algorithm no matter how strange.

The Yes/No Questions Game I choose a number  [1..N]. I ask you yes/no questions. Time is the number of questions asked in the worst case. I answer. I determine the number.

The Yes/No Questions Game Upper Bound 6 Great!

The Yes/No Questions Game Upper Bound Time N leaves = # of questions = height = log 2 (N)

The Yes/No Questions Game Lower Bound? Time N leaves Is there a faster algorithm? Is the third bit 1? If different questions? = # of questions = height = log 2 (N)

The Yes/No Questions Game Lower Bound? N leaves Is there a faster algorithm? If it has a different structure? Best case Worst case Time = # of questions = height = log 2 (N)

The Yes/No Questions Game Lower Bound Theorem: For every question strategy A, with the worst case object I to ask about, log 2 N questions need to be asked to determine one of N objects.  A,  I, A(I)  P(I) or Time(A,I)  log 2 N Proof: Prover/Adversary Game

Lower Bound Proof Oh yeah, I have an input I for which it does not. I have an algorithm A that I claim works and is fast. Two cases: # output leaves < N # output leaves  N

I have an algorithm A that I claim works and is fast. Man your algorithm does not have N output leaves. I win if A on input I gives the wrong output or requires log N questions. Lower Bound Proof: case 1 I give a input I = 4 with missing output.

Good now your algorithm has all N outputs as leaves. It must have height  log N. I win if A on input I gives the wrong output or requires log N questions. I have an algorithm A that I claim works and is fast. I give a input I = 5 at a deep leaf. Lower Bound Proof: case 2

The Yes/No Questions Game Lower Bound Theorem: For every question strategy A, with the worst case object I to ask about, log 2 N questions need to be asked to determine one of N objects.  A,  I, A(I)  P(I) or Time(A,I)  log 2 N Proof: Prover/Adversary Game End of Proof.

Communication Complexity Or a obj  a set of N objs. Time is the number of bits sent in the worst case. I send you a stream of bits. I determine the object. I choose a number  [1..N].

Communication Complexity Upper Bound 6 Great!

Communication Complexity Lower Bound Theorem: For every communication strategy A, with the worst case object I to communicate, log 2 N bits need to transmitted to communicate one of N objects.  A,  I, A(I)  P(I) or Time(A,I)  log 2 N

The Sorting Game I choose a permutation of {1,2,3,…,N}. I ask you yes/no questions. Time is the number of questions asked in the worst case. I answer. I determine the permuation.

Sorting Upper Bound b,c,a Great!

Time N! leaves = # of questions = height = log 2 (N!) Sorting Upper Bound

N! = 1 × 2 × 3 × … × N / 2 × … × N N factors each at most N. N / 2 factors each at least N / 2.  N N N / 2  N/2N/2 Bounding log(N!) N / 2 log( N / 2 )  log(N!)  N log(N) =  (N log(N)).

Time N! leaves = # of questions = height = log 2 (N!) =  (N log(N)). Is there a faster algorithm? Is the third bit of c 1? If different questions? Sorting Lower Bound

class InsertionSortAlgorithm { for (int i = 1; i < a.length; i++) { int j = i; while ((j > 0) && (a[j-1] > a[i])) { a[j] = a[j-1]; j--; } a[j] = B; }} Is there a faster algorithm? If different model of computation? Sorting Lower Bound

Sorting Lower Bound Theorem: For every sorting algorithm A, with the worst case input instance I,  (N log 2 N) comparisons (or other bit operations) need to be executed to sort N objects.  A,  I, A(I)  P(I) or Time(A,I)  N log 2 N Proof: Prover/Adversary Game

Lower Bound Proof Oh yeah, I will find an input I for which it does not. I have an algorithm A that I claim works and is fast. class InsertionSortAlgorithm { for (int i = 1; i < a.length; i++) { int j = i; while ((j > 0) && (a[j-1] > a[i])) { a[j] = a[j-1]; j--; } a[j] = B; }}

Lower Bound Proof I have an algorithm A that I claim works and is fast. class InsertionSortAlgorithm { for (int i = 1; i < a.length; i++) { int j = i; while ((j > 0) && (a[j-1] > a[i])) { a[j] = a[j-1]; j--; } a[j] = B; }} But first let me translate your algorithm into a decision tree.

I choose a permutation of {1,2,3,…,N}. I answer. I determine the permuation. class InsertionSortAlgorithm { for (int i = 1; i < a.length; i++) { int j = i; while ((j > 0) && (a[j-1] > a[i])) { a[j] = a[j-1]; j--; } a[j] = B; }} I ask you this yes/no question. I run my algorithm until I need to know something about the input permutation. Lower Bound Proof

class InsertionSortAlgorithm { for (int i = 1; i < a.length; i++) { int j = i; while ((j > 0) && (a[j-1] > a[i])) { a[j] = a[j-1]; j--; } a[j] = B; }} Lower Bound Proof

Oh yeah, I have an input I for which it does not. I have an algorithm A that I claim works and is fast. Two cases: # output leaves < N! # output leaves  N!

I have an algorithm A that I claim works and is fast. Man your algorithm does not have N! output leaves. I win if A on input I gives the wrong output or requires log N questions. Lower Bound Proof: case 1 I give a input I = with missing output.

Good now your algorithm has all N! outputs as leaves. It must have height  log N!. I win if A on input I gives the wrong output or requires log N! questions. I have an algorithm A that I claim works and is fast. Lower Bound Proof: case 2 I give a input I = at a deep leaf. =  (N log(N)).

Sorting Lower Bound Theorem: For every sorting algorithm A, on the worst case input instance I,  (N log 2 N) comparisons (or other bit operations) need to be executed to sort N objects.  A,  I, A(I)  P(I) or Time(A,I)  N log 2 N Proof: Prover/Adversary Game End of Proof.

End