CSCE 411 Design and Analysis of Algorithms Set 10: Lower Bounds Prof. Jennifer Welch Fall 2014 CSCE 411, Fall 2014: Set 10

What is a Lower Bound? Provides information about the best possible efficiency of ANY algorithm for a problem Tells us whether we can improve an algorithm or not When lower bound is (asymptotically) the same as the best upper bound (provided by an algorithm), the bound is TIGHT If there is a gap, there might be room for improvement CSCE 411, Fall 2014: Set 10

Lower Bound Proof Technique #1: Trivial Use size of input or output Ex: Any algorithm to generate all permutations of n elements requires time Ω(n!) because there are n! outputs Ex: Computing the product of two n-by-n matrices requires time Ω(n2) because the output has n2 entries Ex: Any TSP solution for n cities requires Ω(n2) time because there are Θ(n2) inputs to be taken into account. CSCE 411, Fall 2014: Set 10

Lower Bound Proof Technique #2: Information-Theoretic Consider the amount of information that the solution must provide Show that each step of the algorithm can only produce so much information Uses mechanism of decision trees Example next… CSCE 411, Fall 2014: Set 10

Comparison-Based Sorting We’ve seen mergesort, insertion sort, quicksort, heapsort,… All these algorithms are comparison-based the behavior depends on relative values of keys, not exact values behavior on [1,3,2,4] is same as on [9,25,23,99] Fastest of these algorithms is O(nlog n). We will show that's the best you can get with comparison-based sorting. CSCE 411, Fall 2014: Set 10

Decision Tree Consider any comparison based sorting algorithm Represent its behavior on all inputs of a fixed size with a decision tree Each non-leaf tree node corresponds to the execution of a comparison Each non-leaf tree node has two children, one for the case when the comparison is true and one for false Each leaf tree-node represents correct sorted order for that path CSCE 411, Fall 2014: Set 10

Decision Tree Diagram first comparison: check if ai ≤ aj YES NO second comparison if ai ≤ aj : check if ak ≤ al second comparison if ai > aj : check if am ≤ ap YES NO YES NO third comparison if ai ≤ aj and ak ≤ al : check if ax ≤ ay NO YES CSCE 411, Fall 2014: Set 10

Insertion Sort comparison for j := 2 to n to key := a[j] i := j-1 while i > 0 and a[i] > key do a[i+1] := a[i] i := i -1 endwhile a[i+1] := key endfor comparison CSCE 411, Fall 2014: Set 10

Insertion Sort for n = 3 a1 ≤ a2 ? YES NO a2 ≤ a3 ? a1 ≤ a3 ? NO YES CSCE 411, Fall 2014: Set 10

Insertion Sort for n = 3 a1 ≤ a2 ? YES NO a2 ≤ a3 ? a1 ≤ a3 ? YES NO CSCE 411, Fall 2014: Set 10

How Many Leaves? Must be at least one leaf for each permutation of the input otherwise there would be a situation that was not correctly sorted Number of permutations of n keys is n!. Idea: since there must be a lot of leaves, but each decision tree node only has two children, tree cannot be too shallow depth of tree is a lower bound on running time CSCE 411, Fall 2014: Set 10

Key Lemma Height of a binary tree with n! leaves is (n log n). Proof: Maximum number of leaves in a binary tree with height h is 2h. h = 3, 23 leaves h = 2, 22 leaves h = 1, 21 leaves CSCE 411, Fall 2014: Set 10

Proof of Lemma Let h be height of decision tree. Number of leaves in decision tree, which is n!, is at most 2h. 2h ≥ n! h ≥ log(n!) = log(n(n-1)(n-2)…(2)(1)) ≥ (n/2)log(n/2) by algebra = (n log n) CSCE 411, Fall 2014: Set 10

Finishing Up Any binary tree with n! leaves has height (n log n). Decision tree for any comparison-based sorting algorithm on n keys has height (n log n). Any comparison-based sorting algorithm has at least one execution with (n log n) comparisons Any comparison-based sorting algorithm has (n log n) worst-case running time. CSCE 411, Fall 2014: Set 10

Lower Bound Proof Technique #3: Problem Reduction (Transformation) We have already seen how to use problem transformations in a “positive” way use a pre-existing algorithm for one problem Q to get an algorithm for another problem P So an upper bound for Q helps us get an upper bound for P We can also use this idea in a “negative” way use an already-known lower bound for P to get a lower bound for Q (note change in direction!) CSCE 411, Fall 2014: Set 10

Using a Transformation to Get an Algorithm for P from an Algorithm for Q Recall transforming problem P to problem Q: transform P-input to Q-input run (known) Q-algorithm on Q-input transform Q-output to P-output If we have an algorithm for Q already, and we can do steps 1 and 3, then we get an algorithm for P Ex: transform max-flow to LP CSCE 411, Fall 2014: Set 10

Using a Transformation to Get an Algorithm for P from an Algorithm for Q P-algorithm Q-input Q-output P-output P-input transform Q-algorithm transform Touttransform Tintransform TQ TP = TQ + (Tintransform + Touttransform) = TQ + Ttransform CSCE 411, Fall 2014: Set 10

Using a Transformation to get a Lower Bound for Q from a Lower Bound for P transform P-input to Q-input run any Q-algorithm on Q-input transform Q-output to P-output If we have a lower bound for P and if steps 1 and 2 can be done “quickly enough”, we get a lower bound for Q CSCE 411, Fall 2014: Set 10

Using a Transformation to Get a Lower Bound for Q from a Lower Bound for P P-algorithm Q-input Q-output P-output P-input transform Q-algorithm transform Touttransform Tintransform TQ TP = TQ + Ttransform If TP = Ω(X) and Ttransform = o(X), then TQ = Ω(X) CSCE 411, Fall 2014: Set 10

Use Ω(n log n) Sorting Lower Bound to Get Ω(n log n) Convex Hull Lower Bound Idea: If we could solve convex hull faster than Θ(n log n), then we could solve sorting faster than Θ(n log n) Need to figure out how to use a convex hull “subroutine” to sort numbers Key observation: given n numbers x1,…,xn, the points (x1,x12),…,(xn,xn2) are on a parabola and thus are corners of a convex region CSCE 411, Fall 2014: Set 10

Transform Sorting to Convex Hull Given input x1,…,xn, compute (x1,x12),…(xn,xn2) Run a convex hull algorithm on (x1,x12),…(xn,xn2), resulting in a sequence of points Y Return sequence of first coordinates in Y, starting at point with smallest first coordinate and wrapping around if necessary Note: the convex hull algorithm must be comparison-based and must produce its output points in counter-clockwise (or clockwise) order O(n) Ω(n log n) CSCE 411, Fall 2014: Set 10