CSC401: Analysis of Algorithms CSC401 – Analysis of Algorithms Chapter 5 – 2. Divide-and-Conquer Objectives: Introduce the Divide-and-conquer paradigm and variants: Decrease-and-Conquer and Transform-and- Conquer Review the Merge-sort algorithm Solve recurrence equations –Iterative substitution –Recursion trees –Guess-and-test –The master method Discuss integer and matrix multiplications

CSC401: Analysis of Algorithms5-2-2 Divide-and-Conquer Divide-and conquer is a general algorithm design paradigm: –Divide: divide the input data S in two or more disjoint subsets S 1, S 2, … –Recur: solve the subproblems recursively –Conquer: combine the solutions for S 1, S 2, …, into a solution for S The base case for the recursion are subproblems of constant size Analysis can be done using recurrence equations

CSC401: Analysis of Algorithms5-2-3 Merge-Sort Review Merge-sort on an input sequence S with n elements consists of three steps: –Divide: partition S into two sequences S 1 and S 2 of about n  2 elements each –Recur: recursively sort S 1 and S 2 –Conquer: merge S 1 and S 2 into a unique sorted sequence Algorithm mergeSort(S, C) Input sequence S with n elements, comparator C Output sequence S sorted according to C if S.size() > 1 (S 1, S 2 )  partition(S, n/2) mergeSort(S 1, C) mergeSort(S 2, C) S  merge(S 1, S 2 ) 7 2  9 4   2  2 79  4   72  29  94  4

CSC401: Analysis of Algorithms5-2-4 Recurrence Equation Analysis The conquer step of merge-sort consists of merging two sorted sequences, each with n  2 elements and implemented by means of a doubly linked list, takes at most bn steps, for some constant b. Likewise, the basis case ( n < 2) will take at b most steps. Therefore, if we let T(n) denote the running time of merge-sort: We can therefore analyze the running time of merge- sort by finding a closed form solution to the above equation. –That is, a solution that has T(n) only on the left-hand side.

CSC401: Analysis of Algorithms5-2-5 Iterative Substitution In the iterative substitution, or “plug-and-chug,” technique, we iteratively apply the recurrence equation to itself and see if we can find a pattern Note that base, T(n)=b, case occurs when 2 i =n. That is, i = log n. So, Thus, T(n) is O(n log n).

CSC401: Analysis of Algorithms5-2-6 The Recursion Tree Draw the recursion tree for the recurrence relation and look for a pattern: depthT’ssize01n 12 n2n2n2n2 i 2i2i2i2i n2in2in2in2i ……… timebn bn bn … Total time = bn + bn log n (last level plus all previous levels)

CSC401: Analysis of Algorithms5-2-7 Guess-and-Test Method In the guess-and-test method, we guess a closed form solution and then try to prove it is true by induction: Guess: T(n) < cnlogn. Wrong: we cannot make this last line be less than cn log n

CSC401: Analysis of Algorithms5-2-8 Guess-and-Test Method: Part 2 Recall the recurrence equation: Guess #2: T(n) < cn log 2 n. –if c > b. So, T(n) is O(n log 2 n). In general, to use this method, you need to have a good guess and you need to be good at induction proofs.

CSC401: Analysis of Algorithms5-2-9 Master Method Many divide-and-conquer recurrence equations have the form: The Master Theorem:

CSC401: Analysis of Algorithms Master Method, Examples The form: The Master Theorem: Example 1: –Solution: a=4, b=2, log b a=2, f(n)=n, so case 1 says T(n) is O(n2). Example 2: –Solution: a=b=2, log b a=1, f(n)=nlogn, so case 2 says T(n) is O(n log2 n).

CSC401: Analysis of Algorithms Master Method, Examples The form: The Master Theorem: Example 3: –Solution: a=1, b=3, log b a=0, f(n)=nlogn, so case 3 says T(n) is O(nlogn). Example 4: –Solution: a=2, b=8, log b a=3, f(n)=n 2, so case 1 says T(n) is O(n 3 ).

CSC401: Analysis of Algorithms Master Method, Examples The form: The Master Theorem: Example 5: –Solution: a=9, b=3, log b a=2, f(n)=n 3, so case 3 says T(n) is O(n 3 ). Example 6: (binary search) –Solution: a=1, b=2, log b a=0, f(n)=1, so case 2 says T(n) is O(log n). Example 7: (heap construction) –Solution: a=2, b=2, log b a=1, f(n)=logn, so case 1 says T(n) is O(n).

CSC401: Analysis of Algorithms Iterative “Proof” of the Master Theorem Using iterative substitution, let us see if we can find a pattern: We then distinguish the three cases as –The first term is dominant –Each part of the summation is equally dominant –The summation is a geometric series

CSC401: Analysis of Algorithms Integer Multiplication Algorithm: Multiply two n-bit integers I and J. –Divide step: Split I and J into high-order and low-order bits –We can then define I*J by multiplying the parts and adding: –So, T(n) = 4T(n/2) + n, which implies T(n) is O(n 2 ). –But that is no better than the algorithm we learned in grade school.

CSC401: Analysis of Algorithms An Improved Integer Multiplication Algorithm Algorithm: Multiply two n-bit integers I and J. –Divide step: Split I and J into high-order and low-order bits –Observe that there is a different way to multiply parts: –So, T(n) = 3T(n/2) + n, which implies T(n) is O(n log 2 3 ), by the Master Theorem. –Thus, T(n) is O(n ).

CSC401: Analysis of Algorithms Matrix Multiplication Algorithm Problem: Given n×n matrices X and Y, find their product Z=X×Y, General algorithm: O(n 3 ) Strassen’s Algorithm: –Rewrite the matrix product as: –Where I=AE+BG, J=AF+BH K=CE+DG, L=CF+DH –Define 7 submatrix products: S1=A(F-H), S2=(A+B)H, S3=(C+D)E, S4=D(G-E) S5=(A+D)(E+H), S6=(B-D)(G+H), S7=(A-C)(E+F) –Construct: I=S4+S5+S6-S2, J=S1+S2, K=S3+S4, L=S1-S7- S3-S5 –Running time: T(n) = 7T(n/2)+bn 2 for some constant b, which gives: O(n log7 ), better than O(n 3 )

CSC401: Analysis of Algorithms Decrease and Conquer bReduce problem instance to smaller instance of the same problem and extend solution bSolve smaller instance bExtend solution of smaller instance to obtain solution to original problem bAlso referred to as inductive or incremental approach

CSC401: Analysis of Algorithms Examples of Decrease and Conquer Decrease by one: –Insertion sort –Graph search algorithms: DFSBFS Topological sorting –Algorithms for generating permutations, subsets Decrease by a constant factor –Binary search –Fake-coin problems –multiplication à la russe –Josephus problem Variable-size decrease –Euclid’s algorithm –Selection by partition

CSC401: Analysis of Algorithms Decrease by One Generating all subsets –Given n elements in the set –Take one element from the set –Generate all subsets of remaining elements –Insert the element taken to the subsets generated –Base case: empty set only has a single subset – itself –Time efficiency? Generating permutations –Given n symbols –Take one symbol from the set –Generate all permutations of remaining symbols –Insert the symbol taken to the permutations generated –Base case: a single symbol has only one permutation -- itself –Time efficiency?

CSC401: Analysis of Algorithms Variable-size-decrease: Binary search trees Arrange keys in a binary tree with the binary search tree property: k <k>k What about repeated keys? What about repeated keys? Example 1: Example 1: 5, 10, 3, 1, 7, 12, 9 Example 2: Example 2: 4, 5, 7, 2, 1, 3, 6

CSC401: Analysis of Algorithms Searching and insertion in binary search trees Searching – straightforward Insertion – search for key, insert at leaf where search terminated All operations: worst case # key comparisons = h + 1 log n ≤ h ≤ n – 1 with average (random files) 1.41log n Thus all operations have: –worst case: Θ(n) –average case: Θ(lgn) Bonus: inorder traversal produces sorted list (treesort)

CSC401: Analysis of Algorithms Transform and Conquer Solve problem by transforming into: a more convenient instance of the same problem (instance simplification) –presorting –Gaussian elimination a different representation of the same instance (representation change) a different representation of the same instance (representation change) –balanced search trees –heaps and heapsort –polynomial evaluation by Horner’s rule –Fast Fourier Transform a different problem altogether (problem reduction) a different problem altogether (problem reduction) –reductions to graph problems –linear programming

CSC401: Analysis of Algorithms Instance simplification - Presorting Solve instance of problem by transforming into another simpler/easier instance of the same problem Presorting: Many problems involving lists are easier when list is sorted. searching computing the median (selection problem) computing the mode finding repeated elements

CSC401: Analysis of Algorithms Selection Problem Find the k th smallest element in A[1],…,A[n]. Special cases: –minimum: k = 1 –maximum: k = n –median: k = n/2 Presorting-based algorithm –sort list –return A[k] Partition-based algorithm (Variable decrease & conquer): –pivot/split at A[s] using partitioning algorithm from quicksort –if s=k return A[s] –else if s<k repeat with sublist A[s+1],…A[n]. –else if s>k repeat with sublist A[1],…A[s-1].

CSC401: Analysis of Algorithms Notes on Selection Problem Presorting-based algorithm: Ω(nlgn) + Θ(1) = Ω(nlgn) Partition-based algorithm (Variable decrease & conquer): –worst case: T(n) =T(n-1) + (n+1) -> Θ(n 2 ) –best case: Θ(n) –average case: T(n) =T(n/2) + (n+1) -> Θ(n) –Bonus: also identifies the k smallest elements (not just the k th ) Special cases max, min: better, simpler linear algorithm (brute force) Conclusion: Presorting does not help in this case.

CSC401: Analysis of Algorithms Finding repeated elements Presorting-based algorithm: –use mergesort (optimal): Θ(nlgn) –scan array to find repeated adjacent elements: Θ(n) Brute force algorithm: Θ(n 2 ) Conclusion: Presorting yields significant improvement Similar improvement for mode What about searching? Θ(nlgn)

CSC401: Analysis of Algorithms Taxonomy of Searching Algorithms Elementary searching algorithms –sequential search –binary search –binary tree search Balanced tree searching –AVL trees –red-black trees –multiway balanced trees (2-3 trees, trees, B trees) Hashing –separate chaining –open addressing