Divide-and-Conquer1 7 2  9 4   2  2 79  4   72  29  94  4 Modified by: Daniel Gomez-Prado, University of Massachusetts Amherst

Divide-and-Conquer2 Outline and Reading Divide-and-conquer paradigm (§5.2) Recurrence Equations (§5.2.1) Iterative substitution Recursion trees Guess-and-test The master method Merge-sort (§4.1.1) Algorithm Merging two sorted sequences Merge-sort tree Execution example Analysis Generic merging and set operations (§4.2.1) Quick-sort (§4.3) Algorithm Partition step Quick-sort tree Execution example Analysis of quick-sort (4.3.1) In-place quick-sort (§4.8) Summary of sorting algorithms

Divide-and-Conquer3 Divide-and conquer is a general algorithm design paradigm: Divide the input data S in two or more disjoint subsets S 1, S 2, … Conquer the subproblems by solving them recursively  base case for the recursion: If the subproblems are small enough just solve them Combine the solutions for S 1, S 2, …, into a solution for S Analysis can be done using recurrence equations

Divide-and-Conquer4 Example: Sum of Queue SumQueue(Q) if (Q.length == 0 ) return 0 else return Q.dequeue() + SumQueue(Q) One subproblem Linear reduction in size (decrease by 1) Combining: constant (cost of 1 add) T(0)  b T(n)  c + T(n – 1) for n>0 Subproblem size 1 # subproblems Work dividing and combining Base case Modified from: Mukund Narasimhan, University of Washington

Divide-and-Conquer5 Sum of Queue Solution T(n)  c + c + T(n-2)  c + c + c + T(n-3)  kc + T(n-k) for all k  nc + T(0) for k=n  cn + b = O(n) T(0)  b T(n)  c + T(n – 1) for n>0 Equation: Solution: (finding the closed form solution) Modified from: Mukund Narasimhan, University of Washington Iterative substitution method

Divide-and-Conquer6 Example: Binary Search BinarySearch(A, x) if (A.size == 1) return (x == A[0]) mid = A.size / 2 if (x == A[mid]) return true else if (x < A[mid]) return BinarySearch( A_LowerHalf, x) else if (x > A[mid]) return BinarySearch( A_UpperHalf, x) Search a sorted array for a given item, x  If x == middle array element, return true  Else, BinarySearch lower (x mid) sub-array  1 subproblem, half as large Modified from: Mukund Narasimhan, University of Washington Find: 9

Divide-and-Conquer7 Example: Binary Search BinarySearch(A, x) if (A.size == 1) return (x == A[0]) mid = A.size / 2 if (x == A[mid]) return true else if (x < A[mid]) return BinarySearch( A_LowerHalf, x) else if (x > A[mid]) return BinarySearch( A_UpperHalf, x) T(1)  b T(n)  T(n/2) + c for n>1 Equation: Search a sorted array for a given item, x  If x == middle array element, return true  Else, BinarySearch lower (x mid) sub-array  1 subproblem, half as large Subproblem size Work dividing and combining 1 # subproblems Base case Modified from: Mukund Narasimhan, University of Washington

Divide-and-Conquer8 Binary Search Solution T(n)  T(n/2) + c  T(n/4) + c + c  T(n/8) + c + c + c  T(n/2 k ) + kc  T(1) + c log n where k = log n  b + c log n = O(log n) T(1)  b T(n)  T(n/2) + c for n>1 Equation: Solution: (finding the closed form solution) Modified from: Mukund Narasimhan, University of Washington Iterative substitution method

Divide-and-Conquer9 Recursion Tree for Binary Search n n/2 n/4 … 1 O(1) … log n Θ(log n) O(1) Problem sizeCost per stage Modified from: Mukund Narasimhan, University of Washington

Divide-and-Conquer10 Example: Recursion Tree The recursion tree is of the form: where d > 0 is constant. And the base case has a running time b

Divide-and-Conquer11 Drawing the Recursion Tree With: b bn bn + dn logn

Divide-and-Conquer12 Iterative Substitution for The recursion tree In the iterative substitution, or “plug-and-chug,” technique, we iteratively apply the recurrence equation to itself and see if we can find a pattern: Note that base, T(n)=b, case occurs when 2 i =n. That is, i = log n. So, Thus, T(n) is O(n log n).

Divide-and-Conquer13 Guess-and-Test Method, Part 1 In the guess-and-test method, we guess a closed form solution and then try to prove it is true by induction: Guess: T(n) < cn log n. Wrong: we cannot make this last line be less than cn log n

Divide-and-Conquer14 Guess-and-Test Method, Part 2 Recall the recurrence equation: Guess #2: T(n) < cn log 2 n. if c > b. So, T(n) is O(n log 2 n). In general, to use this method, you need to have a good guess and you need to be good at induction proofs.

Divide-and-Conquer15 The divide-and-conquer design paradigm Divide the problem (instance) into subproblems. a subproblems, each of size n/b Conquer the subproblems by solving them recursively. The base case has a runtime c Combine subproblem solutions. Runtime is f(n) Modified from: Carola Wenk, University of Texas at San Antonio

Divide-and-Conquer16 Master Method Many divide-and-conquer recurrence equations have the form: The Master Theorem:

Divide-and-Conquer17 Master Method, Example 1 The form: The Master Theorem: Example: Solution: log b a=2, so case 1 says T(n) is O(n 2 ).

Divide-and-Conquer18 Master Method, Example 2 The form: The Master Theorem: Example: Solution: log b a=1, so case 2 says T(n) is O(n log 2 n).

Divide-and-Conquer19 Master Method, Example 3 The form: The Master Theorem: Example: Solution: log b a=0, so case 3 says T(n) is O(n log n).

Divide-and-Conquer20 Master Method, Example 4 The form: The Master Theorem: Example: Solution: log b a=3, so case 1 says T(n) is O(n 3 ).

Divide-and-Conquer21 Master Method, Example 5 The form: The Master Theorem: Example: Solution: log b a=2, so case 3 says T(n) is O(n 3 ).

Divide-and-Conquer22 Master Method, Example 6 The form: The Master Theorem: Example: Solution: log b a=0, so case 2 says T(n) is O(log n). (binary search)

Divide-and-Conquer23 Master Method, Example 7 The form: The Master Theorem: Example: Solution: log b a=1, so case 1 says T(n) is O(n). (heap construction)

Divide-and-Conquer24 Iterative “Proof” of the Master Theorem Using iterative substitution, let us see if we can find a pattern: We then distinguish the three cases as The first term is dominant Each part of the summation is equally dominant The summation is a geometric series

Divide-and-Conquer25 Divide-and-Conquer, Algorithm Examples: 1.Merge Sort Quick Sort

Divide-and-Conquer26 1.Merge Sort 7 2  9 4   2  2 79  4   72  29  94  4

Divide-and-Conquer27 Outline Merge-sort (§4.1.1) Algorithm Merging two sorted sequences Merge-sort tree Execution example Analysis Generic merging and set operations (§4.2.1)

Divide-and-Conquer28 Merge-Sort Merge-sort is a sorting algorithm based on the divide-and-conquer paradigm Like heap-sort It uses a comparator It has O(n log n) running time Unlike heap-sort It does not use an auxiliary priority queue It accesses data in a sequential manner (suitable to sort data on a disk)

Divide-and-Conquer29 Merge-Sort Merge-sort on an input sequence S with n elements consists of three steps: Divide: partition S into two sequences S 1 and S 2 of about n  2 elements each Recur: recursively sort S 1 and S 2 Conquer: merge S 1 and S 2 into a unique sorted sequence Algorithm mergeSort(S, C) Input sequence S with n elements, comparator C Output sequence S sorted according to C if S.size() > 1 (S 1, S 2 )  partition(S, n/2) mergeSort(S 1, C) mergeSort(S 2, C) S  merge(S 1, S 2 ) 2T(n/2) f(n)=bn Base case: merge(elem1,elem2) Running time = b

Divide-and-Conquer30 Merging Two Sorted Sequences The conquer step of merge-sort consists of merging two sorted sequences A and B into a sorted sequence S containing the union of the elements of A and B Merging two sorted sequences, each with n  2 elements and implemented by means of a doubly linked list, takes O(n) time Algorithm merge(A, B) Input sequences A and B with n  2 elements each Output sorted sequence of A  B S  empty sequence while  A.isEmpty()   B.isEmpty() if A.first().element() < B.first().element() S.insertLast(A.remove(A.first())) else S.insertLast(B.remove(B.first())) while  A.isEmpty() S.insertLast(A.remove(A.first())) while  B.isEmpty() S.insertLast(B.remove(B.first())) return S

Divide-and-Conquer31 Example: MERGE(L,R) Modified from: Monica Nicolescu, University of Nevada, Remo

Divide-and-Conquer32 Example: MERGE(L,R) Modified from: Monica Nicolescu, University of Nevada, Remo

Divide-and-Conquer33 Example (cont.) Modified from: Monica Nicolescu, University of Nevada, Remo

Divide-and-Conquer34 Example (cont.) Modified from: Monica Nicolescu, University of Nevada, Remo

Divide-and-Conquer35 Example (cont.) Done! Modified from: Monica Nicolescu, University of Nevada, Remo

Divide-and-Conquer36 Merge-Sort Tree An execution of merge-sort is depicted by a binary tree each node represents a recursive call of merge-sort and stores  unsorted sequence before the execution and its partition  sorted sequence at the end of the execution the root is the initial call the leaves are calls on subsequences of size 0 or  9 4   2  2 79  4   72  29  94  4

Divide-and-Conquer37 Execution Example Partition       1 67  72  29  94  43  38  86  61   

Divide-and-Conquer38 Execution Example (cont.) Recursive call, partition 7 2  9 4       1 67  72  29  94  43  38  86  61   

Divide-and-Conquer39 Execution Example (cont.) Recursive call, partition 7 2  9 4    2      72  29  94  43  38  86  61   

Divide-and-Conquer40 Execution Example (cont.) Recursive call, base case 7 2  9 4    2      77  7 2  29  94  43  38  86  61   

Divide-and-Conquer41 Execution Example (cont.) Recursive call, base case 7 2  9 4    2      77  72  22  29  94  43  38  86  61   

Divide-and-Conquer42 Execution Example (cont.) Merge 7 2  9 4    2      77  72  22  29  94  43  38  86  61   

Divide-and-Conquer43 Execution Example (cont.) Recursive call, …, base case, merge 7 2  9 4    2      77  72  22  23  38  86  61     94  4

Divide-and-Conquer44 Execution Example (cont.) Merge 7 2  9 4    2      77  72  22  29  94  43  38  86  61   

Divide-and-Conquer45 Execution Example (cont.) Recursive call, …, merge, merge 7 2  9 4    2      77  72  22  29  94  43  33  38  88  86  66  61  11   

Divide-and-Conquer46 Execution Example (cont.) Merge 7 2  9 4    2      77  72  22  29  94  43  33  38  88  86  66  61  11   

Divide-and-Conquer47 Analysis of Merge-Sort The height h of the merge-sort tree is O(log n) at each recursive call we divide in half the sequence, The overall amount or work done at the nodes of depth i is O(n) we partition and merge 2 i sequences of size n  2 i we make 2 i  1 recursive calls Thus, the total running time of merge-sort is O(n log n) depth#seqssize 01n 12 n2n2 i2i2i n2in2i ………

Divide-and-Conquer48 Recurrence Equation Analysis The conquer step of merge-sort consists of merging two sorted sequences, each with n  2 elements and implemented by means of a doubly linked list, takes at most bn steps, for some constant b. Likewise, the basis case ( n < 2) will take at b most steps. Therefore, if we let T(n) denote the running time of merge-sort: We can therefore analyze the running time of merge-sort by finding a closed form solution to the above equation. That is, a solution that has T(n) only on the left-hand side.

Divide-and-Conquer49 Iterative Substitution In the iterative substitution, or “plug-and-chug,” technique, we iteratively apply the recurrence equation to itself and see if we can find a pattern: Note that base, T(n)=b, case occurs when 2 i =n. That is, i = log n. So, Thus, T(n) is O(n log n).

Divide-and-Conquer50 2.Quick-Sort     29  9

Divide-and-Conquer51 Outline Quick-sort (§4.3) Algorithm Partition step Quick-sort tree Execution example Analysis of quick-sort (4.3.1) In-place quick-sort (§4.8) Summary of sorting algorithms

Divide-and-Conquer52 Quick-Sort Quick-sort is a randomized sorting algorithm based on the divide-and-conquer paradigm: Divide: pick a random element x (called pivot) and partition S into  L elements less than x  E elements equal x  G elements greater than x Recur: sort L and G Conquer: join L, E and G x x L G E x

Divide-and-Conquer53 Partition We partition an input sequence as follows: We remove, in turn, each element y from S and We insert y into L, E or G, depending on the result of the comparison with the pivot x Each insertion and removal is at the beginning or at the end of a sequence, and hence takes O(1) time Thus, the partition step of quick-sort takes O(n) time Algorithm partition(S, p) Input sequence S, position p of pivot Output subsequences L, E, G of the elements of S less than, equal to, or greater than the pivot, resp. L, E, G  empty sequences x  S.remove(p) while  S.isEmpty() y  S.remove(S.first()) if y < x L.insertLast(y) else if y = x E.insertLast(y) else { y > x } G.insertLast(y) return L, E, G

Divide-and-Conquer54 Quick-Sort Tree An execution of quick-sort is depicted by a binary tree Each node represents a recursive call of quick-sort and stores  Unsorted sequence before the execution and its pivot  Sorted sequence at the end of the execution The root is the initial call The leaves are calls on subsequences of size 0 or     29  9

Divide-and-Conquer55 Execution Example Pivot selection      38    94  4

Divide-and-Conquer56 Execution Example (cont.) Partition, recursive call, pivot selection    94     38  8 2  2

Divide-and-Conquer57 Execution Example (cont.) Partition, recursive call, base case   11    94      38  8

Divide-and-Conquer58 Execution Example (cont.) Recursive call, …, base case, join   38     11  14 3   94  44  4

Divide-and-Conquer59 Execution Example (cont.) Recursive call, pivot selection      11  14 3   94  44  4

Divide-and-Conquer60 Execution Example (cont.) Partition, …, recursive call, base case      11  14 3   94  44  4 9  99  9

Divide-and-Conquer61 Execution Example (cont.) Join, join      11  14 3   94  44  4 9  99  9

Divide-and-Conquer62 Worst-case Running Time The worst case for quick-sort occurs when the pivot is the unique minimum or maximum element One of L and G has size n  1 and the other has size 0 The running time is proportional to the sum n  (n  1)  …  2  Thus, the worst-case running time of quick-sort is O(n 2 ) depthtime 0n 1 n  1 …… 1 …

Divide-and-Conquer63 Expected Running Time Consider a recursive call of quick-sort on a sequence of size s Good call: the sizes of L and G are each less than 3s  4 Bad call: one of L and G has size greater than 3s  4 A call is good with probability 1  2 1/2 of the possible pivots cause good calls:  Good callBad call Good pivotsBad pivots

Divide-and-Conquer64 Expected Running Time, Part 2 Probabilistic Fact: The expected number of coin tosses required in order to get k heads is 2k For a node of depth i, we expect i  2 ancestors are good calls The size of the input sequence for the current call is at most ( 3  4 ) i  2 n Therefore, we have For a node of depth 2log 4  3 n, the expected input size is one The expected height of the quick-sort tree is O(log n) The amount or work done at the nodes of the same depth is O(n) Thus, the expected running time of quick-sort is O(n log n)

Divide-and-Conquer65 Summary of Sorting Algorithms AlgorithmTimeNotes selection-sort O(n2)O(n2) in-place slow (good for small inputs) insertion-sort O(n2)O(n2) in-place slow (good for small inputs) quick-sort O(n log n) expected in-place, randomized fastest (good for large inputs) heap-sort O(n log n) in-place fast (good for large inputs) merge-sort O(n log n) sequential data access fast (good for huge inputs)

Divide-and-Conquer66 Conclusions Divide and conquer is just one of several powerful techniques for algorithm design. Divide-and-conquer algorithms can be analyzed using recurrences and the master method (so practice this math). Can lead to more efficient algorithms

Divide-and-Conquer67 Remember Hw #2 is already posted. It is due on the day of the Exam. Midterm Exam is scheduled for Monday, March 28 at 5:30 pm or later. There is a mandatory seminar for CSE student on Monday, March 28, at 4:00, hence we cannot start the exam earlier than 5:30 pm.