1. 1 Divide & Conquer Algorithms Part 4

2. 2 Recursion Review A function that calls itself either directly or indirectly through another function Recursive solutions involve:  Base case – the function returns a solution  Recursive case divide the problem into one or more simpler or smaller parts of the problem call the function (recursively) on each part, and combine the solutions of the parts into a solution for the problem.

3. 3 Designing Algorithms Incremental Design  Most of the algorithms you have seen and programmed  Iterative An algorithmic technique where a function solves a problem by repeatedly working on successive parts of the problem

4. 4 Designing Algorithms (cont) Divide & Conquer Design Three steps  DIVIDE Problem is divided into a number of subproblems  CONQUER Solve the subproblems recursively Base cases are simple enough to solve directly  COMBINE The solutions to the subproblems are combined to solve the original problem

5. 5 Analyzing Divide & Conquer Algorithms Use a recurrence For small subproblem, solution takes constant time DIVIDE step creates a subproblems, each of size n/b  D(n) time to divide into subproblems  C(n) time to combine solutions of subproblems

6. 6 MergeSort Requires additional memory space as a function of n  Unlike Insertion Sort which sorts in place Requires only a constant amount of additional space Sorts in  (nlgn)

7. 7 MergeSort DIVIDE the n element sequence to be sorted into two subsequences of n/2 elements each CONQUER (sort) the two subsequences recursively using merge sort  The recursion stops when subproblem contains only one element COMBINE (merge) the two sorted subsequences to produce the sorted answer

8. 8 MergeSort (Cont) …199627494139048… 1996274 94139048 199 62749413 9048 19 9627494139048

9. 9 MergeSort (Cont) …913194862749094… 9196274 13489094 919 62741394 4890 19 9627494139048

10. 10 MergeSort (cont) To sort entire array: MergeSort( A, 1, length(A) ) MergeSort( A, p, r ) 1.if p < r 2.q   (p + r) / 2  3.MergeSort( A, p, q ) 4.MergeSort( A, q+1, r ) 5.Merge( A, p, q, r )

11. 11 MergeSort (Cont) Merge( A, p, q, r ) 1.n1  q – p + 1 2.n2  r – q 3.create arrays L[1..n1+1] and R[1..n2+1] 4.for i  1 to n1 5.L[ i ]  A[p+i-1] 6.for j  1 to n2 7.R[ i ]  A[q+j] 8.L[n1+1]   9.R[n2+1]   10.i  1 11.j  1 12.for k  p to r 13.if L[ i ]  R[ j ] 14.A[ k ]  L[ i ] 15.i = i + 1 16.else A[ k ]  R[ j ] 17.j = j + 1 Sentinel values

12. 12 Analysis of MergeSort Merge function:  Line: 1- 2   (1)  Line: 3   (n)  Line: 4 – 7  i loop + j loop =  (n)  Line: 8 – 11   (1)  Line: 12   (n)  Line: 13 – 17   (1) Total run time =  (n)

13. 13 Analysis of MergeSort (cont) MergeSort function  For simplicity, assume n is a power of 2  If n = 1, takes constant time  If n > 1 then DIVIDE – Lines 1, 2   (1) –D(n) =  (1) CONQUER – Lines 3, 4  2T(n/2) –two subproblems, each of size n/2 COMBINE – Line 5   (n) –C(n) =  (n)

14. 14 Analysis of MergeSort (cont) So the recurrence is: Note: D(n) + C(n) =  (1) +  (n) =  (n) The solution to the recurrence

15. 15 QuickSort Worst time:  (n 2 ) Expected time:  (nlgn)  Constants in the expected time are small Sorts in place

16. 16 QuickSort (cont) DIVIDE – Partition A[p..r] into two subarrays A[p..q-1] and A[q+1..r] such that each element of A[p..q-1] is  A[q]  each element of A[q+1..r] Conquer – Sort the two subarrays by recursive calls to Quicksort Combine – Since subarrays are sorted in place, they are already sorted

17. 17 QuickSort (cont) To sort entire array: QuickSort( A, 1, length(A) ) QuickSort( A, p, r ) 1.if p < r 2.q  Partition( A, p, r ) 3.QuickSort( A, p, q-1 ) 4.QuickSort( A, q+1, r )

18. 18 QuickSort (cont) Partition( A, p, r ) x  A[ r ] i  p – 1 for j  p to r-1 if A[ j ]  x i  i + 1 Exchange( A[ i ], A[ j ] ) Exchange( A[ i+1 ], A[ r ] ) return i+1

19. 19 QuickSort (cont)

20. 20 QuickSort (cont) Return i+1 which is 4

21. 21 Performance of QuickSort Partition function’s running time -  (n) Running time of QuickSort depends on the balance of the partitions  If balanced, QuickSort is asymptotically as fast as MergeSort  If unbalanced, it is asymptotically as bad as Insertion Sort

22. 22 Performance of QuickSort (cont) Worst-case Partitioning  Partitions always of size n-1 and 0  Occurs when array is already sorted  Recurrence for the running time:

23. 23 Performance of QuickSort (cont) cn c(n-1) c(n-2)...... n cn c(n-1) c(n-2) c Total:  (n 2 )

24. 24 Performance of QuickSort (cont) Best-case Partitioning  Partitions always of size  n/2  and  n/2  -1  The recurrence for the running time: Case 2 of Master Method

25. 25 Performance of QuickSort (cont) Balanced Partitioning  Average-case is closer to best-case  Any split of constant proportionality (say 99 to 1) will have a running time of  (nlgn) The recurrence will be Because it yields a recursion tree of depth  (lgn), where cost at each level is  (n) See page 151 (new book) for picture –Or next slide

26. 26 Performance of QuickSort (cont) c(n/100)c(99n/100) cn c(n/10000)c(99n/10000) c(9801n/10000) T(1) Total:  (nlgn) log 100 n cn  cn log 100/99 n

27. 27 Performance of QuickSort (cont) Intuition for the average case  The behavior depends on the relative ordering of the values Not the values themselves  We will assume (for now) that all permutations are equally likely  Some splits will be balanced, and some will be unbalanced

28. 28 Performance of QuickSort (cont)  In a recursion tree for an average-case, the “good” and “bad” splits are distributed randomly throughout the tree  For our example, suppose Bad splits and good splits alternate Good splits are best-case splits Bad splits are worst-case splits Boundary case (subarray size 0) has cost of 1

29. 29 Performance of QuickSort (cont)  The  (n-1) cost of the bad split can be absorbed into the  (n) cost of the good split, and the resulting split is good  Thus the running time is  (nlgn), but with a slightly larger constant n 0n - 1 (n-1) / 2 (n-1) / 2 -1 (n)(n) n (n-1) / 2 (n)(n)

30. 30 Randomized QuickSort How do we increase the chance that all permutations are equally likely?  Random Sampling Don’t always use last element in subarray Swap it with a randomly chosen element from the subarray  Pivot now is equally likely to be any of the r – p + 1 elements We can now expect the split to be reasonably well-balanced on average

31. 31 Randomized QuickSort (cont) Randomized-Partition( A, p, r ) 1.i  Random( p, r ) 2.Exchange( A[ r ], A[ i ] ) 3.return Partition( A, p, r ) Note that Partition( ) is same as before

32. 32 Randomized QuickSort (cont) Randomized-QuickSort( A, p, r ) 1.if p < r 2.q  Randomized-Partition( A, p, r ) 3.Randomized-QuickSort( A, p, q-1 ) 4.Randomized-QuickSort( A, q+1, r )

33. 33 Analysis of QuickSort A more rigorous analysis Begin with worst-case  We intuited that worst-case running time is  (n 2 )  Use substitution method to show this is true

34. 34 Analysis of QuickSort (cont)  Guess:  (n 2 )  Show: for some c > 0  Substitute: q 2 +(n-q-1) 2 is max at endpoints of range. Therefore it is  (n-1) 2 = n 2 – 2n +1

35. 35 Analysis of QuickSort (cont)  Problem 7.4-1 has you show that  Thus the worst-case running time of QuickSort is  (n 2 )

36. 36 Analysis of QuickSort (cont) We will show that the upper-bound on expected running time is  (nlgn) We’ve already shown that the best-case running time is  (nlgn) Combined, these will give an expected running time of  (nlgn)

37. 37 Analysis of QuickSort (cont) Expected Running Time  Work done is dominated by Partition  Each time a pivot is selected, this element is never included in subsequent calls to QuickSort And the pivot is in its correct place in the array  Therefore, at most n calls to Partition will be made Each call to Partition involves  (1) work plus the amount of work done in the for loop Count the total number of times line 4 is executed, we can bound the amount of time spent in the for loop Line 4: if A[j]  x

38. 38 Analysis of QuickSort (cont)  Lemma 7.1 Let X be the number of comparisons performed in line 4 of Partition over the entire execution of QuickSort on an n-element array. Then the running time of QuickSort is  (n + X) Proof: –There are n calls to Partition, each of which does  (1) work then executes the for loop (which includes line 4) some number of times –Since the for loop executes line 4 during each iteration, X represents the number of iterations of the for loop along with the number of comparisons performed –Therefore T(n) =  (n  (1) + X) =  (n + X)

39. 39 Analysis of QuickSort (cont)  We need to compute X  We do this by computing an overall bound on the total number of comparisons NOT by computing the number of comparisons at each call to Partition  Definitions: z 1, z 2, …, z n  elements in the array z i  i th smallest element set Z ij = {z i, z i+1, …, z j }

40. 40 Analysis of QuickSort (cont)  When does the algorithm compare z i and z j ?  Note: each pair of elements is compared at most once. Why?  Our analysis uses indicator random variables

41. 41 Indicator Random Variables They provide a convenient method for converting between probabilities and expectations These are random variables which take on only the value 0 or 1, so they “indicate” whether or not something has happened Indicator Random Variable I{A} is defined as:  Given a sample space S and an event A  I{A} = 1if A occurs 0if A does not occur

42. 42 Indicator Random Variables (cont) A simple example  Determine the number of heads when flipping a coin  Sample space S = {H, T}  Simple random variable Y These are random variables whose range contains only a finite number of elements In this case, it takes on the values H and T Each with a probability of ½  X H is associated with the event Y = H X H = I{Y = H} = 1if Y = H 0if Y = T

43. 43 Indicator Random Variables (cont)  The expected number of heads in one flip is the expected value of our indicator variable X H Thus the expected number of heads in one flip is ½

44. 44 Indicator Random Variables (cont) Lemma 5.1  Given a sample space S and an event A in the sample space S, let X A = I{A}. Then E[X A ] = Pr{A} See proof on page 95 To compute the number of heads in n coin flips  Method 1 – compute the probability of getting 0 heads, 1 heads, 2 heads, etc

45. 45 Indicator Random Variables (cont)  Method 2 Let X i be the indicator random variable associated with the event “the i th flip is heads” Let Y i be the random variable denoting the outcome of the i th flip –X i = I{Y i = H} Let X be the random variable denoting the total number of heads in the n coin flips

46. 46 Indicator Random Variables (cont) Take the expectation of both sides: By Lemma 5.1

47. 47 (Back to) Analysis of QuickSort  We will use indicator random variables X ij = I{z i is compared to z j } it indicates whether the comparison took place at any time during execution of QuickSort  Since each pair is compared at most once:

48. 48 Analysis of QuickSort (cont)  Take expectation of both sides

49. 49 Analysis of QuickSort (cont)  We still need to compute Pr{z i is compared to z j }  Start by thinking about when two items are not compared once a pivot x is chosen with z i < x < z j –z i and z j will never be compared if z i is the first pivot chosen in Z ij –z i will be compared to every other element in Z ij Similarly for z j  Thus, z i and z j are compared iff the first pivot from Z ij is either z i or z j

50. 50 Analysis of QuickSort (cont)  What is the probability that this event occurs? Before a pivot has been chosen from Z ij, all elements of Z ij are in the same partition Each element of Z ij is equally likely to be chosen as the first pivot the probability is

51. 51 Analysis of QuickSort (cont)  Thus we have Because the two events are mutually exclusive

52. 52 Analysis of QuickSort (cont) Combining the two boxed equations Change of variables: k = j – i Note the changes in the summation variables Bound on Harmonic Series:

53. 53 Analysis of QuickSort (cont) Thus, using Randomized-Partition, the expected running time of QuickSort is  (nlgn)

54. 54 Median and Order Statistics In this section, we will study algorithms for finding the i th smallest element in a set of n elements We will again use divide-and-conquer algorithms

55. 55 The Selection Problem Input: A set A of n (distinct) numbers and a number i, with 1  i  n Output: The element x  A that is larger than exactly i – 1 other elements of A  x is the i th smallest element i = 1  minimum i = n  maximum

56. 56 The Selection Problem (cont) A simple solution  Sort A  Return A[ i ]  This is  (nlgn)

57. 57 Minimum and Maximum Finding the minimum and maximum  Takes (n-1) comparisons (  (n))  This is the best we can do and is optimal with respect to the number of comparisons MINIMUM(A) min  A[1] for i  2 to length(A) if min > A[ i ] min  A[ i ] return min MAXIMUM(A) max  A[1] for i  2 to length(A) if max < A[ i ] max  A[ i ] return max

58. 58 Minimum and Maximum (cont) Simultaneous minimum and maximum  Obvious solution is 2(n-1) comparisons  But we can do better – namely  The algorithm If n is odd, set max and min to first element If n is even, compare first two elements and set max, min Process the remaining elements in pairs Find the larger and the smaller of the pair Compare the larger of the pair with the current max And the smaller of the pair with the current min

59. 59 Minimum and Maximum (cont)  Total number of comparisons If n is oddcomparisons If n is even –1 initial comparison –And 3(n – 2)/2 comparisons –For a total of 3n/2 – 2 comparisons In either case, total number of comparisons is at most

60. 60 Selection in Expected Linear Time Goal: Select i th smallest element from A[p..r]. Partition into A[p..q-1] and A[q+1..r] if i = q  then return A[q] If i th smallest element is in A[p..q-1]  then recurse on A[p..q-1]  else recurse on A[q+1..r]

61. 61 Selection in Expected Linear Time (cont) Randomized-Select(A, p, r, i) 1 if p = r 2 return A[p] 3 q  Randomized-Partition(A, p, r) 4 k  q - p + 1 //Why this statement? 5 if i = k 6 return A[q] 7else if i < k 8return Randomized-Partition(A, p, q-1, i) 9else 10return Randomized-Partition(A, q+1, r, i-k)

62. 62

63. 63 Analysis of Selection Algorithm Worst-case running time is  (n 2 )  Partition takes  (n)  If we always partition around the largest remaining element, we reduce the partition-size by one element each time What is best-case?

64. 64 Analysis of Selection Algorithm (cont) Average Case  Average-case running time is  (n)  The time required is the random variable T(n)  We want an upper bound on E[T(n)]  In Randomized-Partition, all elements are equally likely to be the pivot

65. 65 Analysis of Selection Algorithm (cont)  So, for each k such that 1  k  n, subarray A[p..q] has k elements All  the pivot with probability 1/n  For k = 1, 2, …, n we define indicator random variables X k where X k = I{the subarray A[p..q] has exactly k elements}  So, E[X k ] = 1/n

66. 66 Analysis of Selection Algorithm (cont)  When we choose the pivot element (which ends up in A[q]) we do not know what will happen next Do we return with the i th element (k = i)? Do we recurse on A[p..q-1]? Do we recurse on A[q+1..r]?  Decision depends on i in relation to k  We will find the upper-bound on the average case by assuming that the i th element is always in the larger partition

67. 67 Analysis of Selection Algorithm (cont)  Now, X k = 1 for just one value of k, 0 for all others  When X k = 1, the two subarrays have sizes k – 1 and n – k  Hence the recurrence:

68. 68 Analysis of Selection Algorithm (cont)  Taking the expected values:

69. 69 Analysis of Selection Algorithm (cont)  Looking at the expression max(k-1, n-k) If n is even, each term from appears twice in the summation If n is odd, each term from appears twice and appears once in the summation

70. 70 Analysis of Selection Algorithm (cont)  Thus we have  We use substitution to solve the recurrence  Note: T(1) =  (1) for n less than some constant  Assume that T(n)  cn for some constant c that satisfies the initial conditions of the recurrence

71. 71 Analysis of Selection Algorithm (cont)  Using this inductive hypothesis

72. 72 Analysis of Selection Algorithm (cont)

73. 73 Analysis of Selection Algorithm (cont)  To complete the proof, we need to show that for sufficiently large n, this last expression is at most cn i.e. As long as we choose the constant c so that c/4 – a > 0 (i.e., c > 4a), we can divide both sides by c/4 – a

74. 74 Analysis of Selection Algorithm (cont)  Thus, if we assume that T(n) =  (1) for, we have T(n) =  (n)

75. 75 Selection in Worst-Case Linear Time “Median of Medians” algorithm It guarantees a good split when array is partitioned  Partition is modified so that the pivot now becomes an input parameter The algorithm:  If n = 1 return A[n]

76. 76 Selection in Worst-Case Linear Time (cont) 1.Divide the n elements of the input array into  n/5  groups of 5 elements each and at most one group of (n mod 5) elements 2.Find the median of each of the  n/5  groups by using insertion sort to sort list and then pick the 3 rd element of each group 3.Use Select recursively to find the median x of the  n/5  medians found in step 2. –If even number of medians, choose lower median

77. 77 Selection in Worst-Case Linear Time (cont) 4.Partition the input array around the “median of medians” x using the modified version of Partition. Let k be one more than the number of elements on the low side of the partition, so that x is the k th smallest element and there are n – k elements on the high side of the partition 5.if i = k, then return x. Otherwise, use Select recursively to find the i th smallest element on the low side if i k

78. 78 Selection in Worst-Case Linear Time (cont) Example of “Median of Medians”  Input Array A[1..100]  Step 1: 25 groups of 5  Step 2: We get 25 medians  Step 3: Step 1: Using the 25 medians we get 5 groups of 5 Step 2: We get 5 medians Step 3: Step 1: Using the 5 medians, we get 1 group of 5 Step 2: We get 1 median  Step 4: Partition A around the median

79. 79 Analyzing “Median of Medians” Analyzing “median of medians”  The following diagram might be helpful:

80. 80 Analyzing “Median of Medians” (cont)  First, we need to put a lower bound on how many elements are greater than x  How many of the medians are greater than x? At least half of the medians from the groups –Why “at least half?” medians are greater than x

81. 81 The two discarded groups Analyzing “Median of Medians” (cont)  Each of these medians contribute at least 3 elements greater than x except for two groups The group that contains x –contributes only 2 elements greater than x The group that has less than 5 elements  So the total number of elements > x is at least:

82. 82 Analyzing “Median of Medians” (cont)  Similarly, there are at least elements smaller than x  Thus, in the worst case, for Step 5 Select is called recursively on the largest partition The largest partition has at most elements The size of the array minus the number of elements in the smaller partition

83. 83 Analyzing “Median of Medians” (cont)  Developing the recurrence: Step 1 takes  (n) time Step 2 takes  (n) time –  (n) calls to Insertion Sort on sets of size  (1) Step 3 takes Step 4 takes  (n) time Step 5 takes at most

84. 84 Analyzing “Median of Medians” (cont)  So the recurrence is  Now use substitution to solve Assume T(n)  cn for some suitable large constant c and all n  ??? Also pick a constant a such that the function described by the  (n) term is bounded above by an for all n > 0

85. 85 Comes from removing the   Analyzing “Median of Medians” (cont) Which is at most cn if If n = 70, then this inequality is undefined

86. 86 Analyzing “Median of Medians” (cont)  We assume that n  71, so  Choosing c  710a will satisfy the inequality on the previous slide  You could choose any constant > 70 to be the base case constant Thus, the selection problem can be solved in the worst-case in linear time

87. 87 Review of Sorts Review of sorts seen so far  Insertion Sort Easy to code Fast on small inputs (less than ~50) Fast on nearly sorted inputs Stable  (n) best case (sorted list)  (n 2 ) average case  (n 2 ) worst case (reverse sorted list)

88. 88 Review of Sorts Stable means that numbers with the same value appear in the output array in the same order as they do in the input array. That is, ties between two numbers are broken by the rule that whichever number appears first in the input array appears first in the output array. Normally, the property of stability is important only when satellite data are carried around with the element being sorted.

89. 89 Review of Sorts (cont)  MergeSort Divide and Conquer algorithm Doesn’t sort in place Requires memory as a function of n Stable  (nlgn) best case  (nlgn) average case  (nlgn) worst case

90. 90 Review of Sorts (cont)  QuickSort Divide and Conquer algorithm –No merge step needed Small constants Fast in practice Not stable  (nlgn) best case  (nlgn) average case  (n 2 ) worst case

91. 91 Review of Sorts (cont) Several of these algorithms sort in  (nlgn) time  MergeSort in worst case  QuickSort on average On some input we can achieve  (nlgn) time for each of these algorithms The sorted order they determine is based only on comparisons between the input elements They are called comparison sorts

92. 92 Review of Sorts (cont) Other techniques for sorting exist, such as Linear Sorting which is not based on comparisons Usually with some restrictions or assumptions on input elements Linear Sorting techniques include:  Counting Sort  Radix Sort  Bucket Sort

93. 93 Lower Bounds for Sorting In general, assuming unique inputs, comparison sorts are expressed in terms of comparisons.  are equivalent in learning about the order of a i and a j What is the best we can do on the worst case type of input? What is the best worst-case running time?

94. 94 The Decision-Tree Model 1:2 2:3 1:3  1,2,3   2,1,3   1,3,2  3,1,2  2,3,1  3,2,1           n = 3 input:  a 1,a 2,a 3  # possible outputs = 3! = 6 Each possible output is a leaf

95. 95 Analysis of Decision-Tree Model Worst Case Comparisons is equal to height of decision tree Lower bound on the worst case running time is the lower bound on the height of the decision tree. Note that the number of leaves in the decision tree  n!, where n = number elements in the input sequence

96. 96 Theorem 8.1 Any comparison sort algorithm requires  (nlgn) comparisons in the worst case Proof:  Consider a decision tree of height h that sorts n elements  Since there are n! permutations of n elements, each permutation representing a distinct sorted order, the tree must have at least n! leaves

97. 97 Theorem 8.1 (cont)  A binary tree of height h has at most 2 h leaves The best possible worst case running time for comparison sorts is thus  (nlgn) Mergesort, which is O(nlgn), is asymptotically optimal By equation 3.18

98. 98 Sorting in Linear Time But the name of this chapter is “Sorting in Linear Time” How can we do better?  CountingSort  RadixSort  BucketSort

99. 99 Counting Sort No comparisons between elements But depends on assumptions about the values being sorted  Each of the n input elements is an integer in the range 0 to k  When k =  (n), the sort runs in  (n) time  The algorithm: Input: A[1..n] where A[ j ]  {0, 1, …, k} Output: B[1..n], sorted –Notice: elements are not sorted in place Also: C[1..k] for auxiliary storage

100. 100 Counting Sort (cont) Counting-Sort(A, B, k) 1.for i  0 to k 2.C[i]  0 3.for j  1 to Length(A) 4.C[A[j]]  C[A[j]] + 1 5.// C[i] now contains the number of elements = i 6.for i  1 to k 7.C[i]  C[i] + C[i-1] 8.// C[i] now contains the number of elements  i 9.for j  Length(A) downto 1 10.B[C[A[j]]]  A[j] 11.C[A[j]]  C[A[j]] – 1 k+2 k+1 n+1 n k+1 k n+1 n -------------------------------------  ---------------------------------------  --------------------------  -----------------------  -------------------------------------  --------------------------  --------------------  ---------------------------  ----------------------- 

101. 101

102. 102 Analysis of CountingSort:  (nlgn) does not apply because CountingSort isn’t a comparison sort CountingSort is stable because 4 th loop goes from n downto 1

103. 103 Radix Sort This sort was originally used to sort computer punch-card decks. It is currently used for multi-key sorts  for example: year/month/day Consider each digit of the number as a separate key

104. 104 Radix Sort (cont) Idea 1: Sort on most significant digit n, then sort on digit n-1, etc. Problem: For old sorters, sort into 10 bins, but subsequent recursive sorts require all 10 bins  Operator must store the other 9 piles Idea 2: Sort on least significant digit, then sort on next least significant digit, etc.

105. 105 Radix Sort (cont) Radix-Sort(A, d) for i  1 to d use a stable sort to sort array A on digit d

106. 106 Radix Sort (cont) Radix-n: another way of indicating the sort used  Implies each digit can differentiate among n different symbols.  In the previous case we assumed radix-10. This is why the name Radix Sort is given

107. 107 Analysis of RadixSort If each digit is in the range 1 to k (or 0 to k-1), use CountingSort for each pass Each pass over a digit is  (n + k) For d digits  (d(n + k)) If d is a constant and k =  (n) T(n) =  (n)

108. 108 Proof by induction that RadixSort works Base Case: d = 1  since only one digit, sorting by that digit sorts the list Inductive Step: Assume it holds for d – 1digits Show: that it works for d digits  A radix sort of d digits is the same as a radix sort of d-1 digits followed by a sort on digit d  By our inductive hypothesis, the sort on d-1 digits works and the digits are in order according to their low-order d- 1 digits

109. 109 Proof by induction that RadixSort works (cont) The sort on digit d will order the elements by their d th digit Consider two elements a and b, with d th digits a d and b d respectively If a d < b d, the sort will put a before b, which is correct, since a < b regardless of their low order digits

110. 110 Proof by induction that RadixSort works (cont) If a d > b d, the sort will put a after b, which is correct, since a > b regardless of their low order digits if a d = b d, the sort will leave a and b in the same order they were in, because it is stable. But that order is already correct, since the correct order of a and b is determined by the low-order d-1 digits when their d th digits are equal

111. 111 Radix Sort Example Show how n integers in the range 1 to n 2 can be sorted in  (n) time  Subtract 1 from each number So they’re in the range 0 to n 2 -1 We’ll add the one back after they are sorted  Use a radix-n sort  Each digit requires n symbols, and log n n 2 digits are needed (d=2, k=n). i.e., treat the numbers as 2-digit numbers in radix n Each digit ranges from 0 to n-1

112. 112 Radix Sort Example (cont)  Sort these 2-digit numbers with radix sort  There are 2 calls to counting sort For  (2(n + n)) =  (n) time  The passes to subtract 1 and add 1each take  (n) time  Hence, total running time of  (n)

113. 113 Radix Sort Example (cont) Take 15 integers in the range of 1 to 15 2 (225) Subtract one from each number to get the range 0 to 224 We will use a radix-15 sort We will need 2 digits of n-many symbols  Each digit ranges from 0 to n-1 See Handout

114. 114 Bucket Sort Bucket Sort assumes that the input values are uniformly distributed over the range [0,1), 0  x < 1 Procedure:  Divide inputs into n equal-sized subintervals (buckets) over the range [0,1).  Sort each bucket and concatenate the buckets. T(n) =  (n)

115. 115 Bucket Sort (cont)

116. 116 Bucket Sort (cont) Bucket-Sort(A) 1.n  Length(A)1 2.for i  1 to nn+1 3.insert A[i] into list B[  nA[i]  ]n 4.for i  0 to n – 1n+1 5.sort list B[i] with InsertionSortn*T(n) 6.concatenate the lists B[0], B[1], …,B[n-1] n together in order

117. 117 Analysis of Bucket Sort All lines except Line 5 take  (n) time What is the cost of the calls to InsertionSort?  Let n i be the random variable denoting the number of elements in bucket B[i]  And we know that InsertionSort runs in  (n 2 ) time

118. 118 Analysis of Bucket Sort (cont) By equation c.21 EQ 8.2 EQ 8.1

119. 119 Analysis of Bucket Sort (cont)  To prove the above equation, define indicator random variables X ij = I{A[j] falls into bucket i] –for i = 0, 1, …, n-1 –for j = 1, 2, …, n  Thus…

120. 120 Analysis of Bucket Sort (cont) EQ 8.3

121. 121 Analysis of Bucket Sort (cont) Evaluate the two summations separately  Lemma 5.1 E[X A ] = Pr{A}

122. 122 Analysis of Bucket Sort (cont) Substitute these into EQ 8.3 Which proves EQ 8.2

123. 123 Thus, when input is drawn from a uniform distribution, BucketSort runs in linear time Analysis of Bucket Sort (cont) Use the expected value in EQ 8.1

124. 124 Strassen’s Algorithm for Matrix Multiplication It is another divide-and-conquer algorithm The naïve (straightforward) method for multiplying two n x n matrices is  (n 3 )  n 3 multiplications  n 2 (n-1) additions Strassen’s algorithm runs in  (n lg7 ) =  (n 2.81 )

125. 125 Strassen’s Algorithm for Matrix Multiplication (cont) We wish to compute C = AB  All are n x n matrices  Assume n is a power of 2 Divide each of A, B, and C into four n / 2 x n / 2 matrices, rewriting C = AB

126. 126 Strassen’s Algorithm for Matrix Multiplication (cont) To compute each of the sub-matrices of C:  r = ae + bg  s = af + bh  t = ce + dg  u = cf + dh These are the eight “essential terms”

127. 127 Strassen’s Algorithm for Matrix Multiplication (cont) To solve C  We need to make 8 recursive calls to “multiply” (with input size n / 2 )  And then perform 4 matrix additions Therefore,  No better than the “naïve” method The addition of two n x n matrices takes  (n 2 ) time

128. 128 Strassen’s Algorithm for Matrix Multiplication (cont) Strassen was the first to discover an asymptotically faster algorithm (in the 1970s) By case 1 of the Master Method

129. 129 Strassen’s Algorithm for Matrix Multiplication (cont) His method  Divide the input matrices into n / 2 x n / 2 submatrices  Using  (n 2 ) scalar additions and subtractions, compute 14 matrices A 1, B 1, A 2, B 2, …, A 7, B 7, each of which is n / 2 x n / 2  Recursively compute the seven matrix products P i = A i B i for i = 1, 2, …, 7

130. 130 Strassen’s Algorithm for Matrix Multiplication (cont)  Compute the desired submatrices r, s, t, u of the result matrix C by adding and/or subtracting various combinations of the P i matrices, using only  (n 2 ) scalar additions and subtractions

131. 131 Strassen’s Algorithm for Matrix Multiplication (cont) The P i matrices: Notice that each of these 7 P i matrices requires ONE multiplication only

132. 132 Strassen’s Algorithm for Matrix Multiplication (cont) Use the P i matrices to compute:

133. 133 Strassen’s Algorithm for Matrix Multiplication (cont) The simple recurrence to compute C is: The cost of the 7 recursive calls The cost of the 18 n / 2 x n / 2 matrix additions and subtractions

134. 134 An Example of Strassen’s Algorithm  Let C, A, and B be 2x2 matrices with  Therefore:

135. 135 An Example of Strassen’s Algorithm (cont)

136. 136 An Example of Strassen’s Algorithm (cont)

137. 137 Discussion of Strassen’s Algorithm Strassen’s algorithm is numerically stable, but not always as accurate as the naïve algorithm It requires a significant amount of workspace to store intermediate results The constant factor in the running time is greater than that in the naive algorithm

138. 138 Discussion of Strassen’s Algorithm (cont) When matrices are sparse, there are faster algorithms designed for sparse matrices A potential attraction is its natural block decomposition; instead of continuing the recursion until n = 1, one could instead use a more standard matrix multiply as soon as n is small enough

139. 139 Final Notes on Strassen’s Algorithm What if n is not a power of 2?  And you want to compute the product of two n x n matrices A and B Set Consider the following two matrices

140. 140 Final Notes on Strassen’s Algorithm (cont) Since is a power of two, Strassen’s algorithm can be used to multiply Since, the resulting algorithm still uses operations to multiply A and B together

141. 141 An Example

142. 142 Polynomials and FFT To add two polynomials of degree n takes  (n) time To multiply two polynomials of degree n takes  (n 2 ) time Using the Fast Fourier Transform, we can reduce the time of multiplying to  (nlgn) time

143. 143 Polynomials Coefficients  Degree k  The highest nonzero coefficient is a k Degree-bound  Any integer strictly greater than the degree of a polynomial Any polynomial with degree 0 to n-1 has a degree-bound n

144. 144 Polynomials (cont) Polynomial addition  If A(x) and B(x) are polynomials of degree- bound n, then A(x) + B(x) = C(x) is also of degree-bound n

145. 145 Polynomials (cont) Polynomial multiplication  degree(C) = degree(A) + degree(B) which implies Degree-bound (C) = degree-bound(A) + degree-bound(B) – 1 <= degree-bound(A) + degree-bound(B)

146. 146 Review of Complex Numbers Have the form a + bi where i =

147. 147 Coefficient Representation of Polynomials Is a vector of coefficients Evaluating the polynomial A(x 0 ) using Horner’s rule:  Takes time  (n) y  0 i  n while i >= 0 y  a i + x * y i  i - 1

148. 148 Coefficient Representation of Polynomials (cont) Adding two polynomials  Simply add the two coefficient vectors a and b representing A(x) and B(x)  Takes time  (n)

149. 149 Coefficient Representation of Polynomials (cont) Multiplying two polynomials  Multiply each element of coefficient vector a by each element in coefficient vector b giving coefficient vector c  c is called the convolution of vectors a and b It is denoted by c = a  b  Takes time  (n 2 )

150. 150 Coefficient Representation of Polynomials (cont) Multiplying polynomials and computing convolutions is done often in practice We need efficient algorithms for them

151. 151 Point-Value Representation of Polynomials A point-value representation of a polynomial A(x) of degree-bound n is a set of n point-value pairs such that all of the x k are distinct and y k = A(x k ) for k = 0, 1, …, n-1

152. 152 Point-Value Representation of Polynomials (cont) Uniquely specified by knowing A(x) at n different values of x x y = A(x) xjxj yjyj

153. 153 Point-Value Representation of Polynomials (cont) To compute the point-value representation for a polynomial in coefficient form  Select n distinct points x 0, x 1, …, x n-1  Evaluate A(x k ) for k = 0, 1, …, n-1  Using Horner’s method this takes  (n 2 ) time

154. 154 Point-Value Representation of Polynomials (cont) Interpolation  Taking the point-value representation and determining the coefficient form of the polynomial This is the same as “evaluating” the polynomial in point-value form

155. 155 Point-Value Representation of Polynomials (cont) Theorem 30.1 (Uniqueness of an interpolating polynomial)  For any set of n point-value pairs such that all the x k values are distinct, there is a unique polynomial A(x) of degree-bound n such that y k = A(x k ) for k = 0, 1, …, n-1  Proof – but first a review of some matrix operations

156. 156 Review of Matrix Operations n x n identity matrix (I n ) = Inverse of an n x n matrix A is the n x n matrix A -1 (if it exists), such that

157. 157 Review of Matrix Operations (cont) If a matrix has an inverse, it is called  Invertible  Nonsingular If a matrix has no inverse, it is called  Noninvertible  Singular When a matrix inverse exists, it is unique

158. 158 Review of Matrix Operations (cont) Minor  The ijth minor of an n x n matrix A, for n > 1, is the (n-1) x (n-1) matrix A [ i j ] obtained by deleting the i th row and j th column of A

159. 159 Review of Matrix Operations (cont) Determinant of an n x n matrix A  Defined recursively in terms of its minors by

160. 160 Review of Matrix Operations (cont) To easily find the determinant of a 2 x 2 matrix:

161. 161 Review of Matrix Operations (cont) To find the determinant of a 3 x 3 matrix

162. 162 Review of Matrix Operations (cont) Example: Find the determinant of:

163. 163 Review of Matrix Operations (cont) Same example, different notation

164. 164 Review of Matrix Operations (cont) Example: Find the determinant of:

165. 165 Review of Matrix Operations (cont)

166. 166 Review of Matrix Operations (cont) Same example, different notation

167. 167 Review of Matrix Operations (cont) Theorem 28.5  An n x n matrix A is singular (is noninvertible) if and only if det(A) = 0  Conversely, an n x n matrix A is nonsingular (is invertible) if and only if det(A)  0

168. 168 Review of Matrix Operations (cont) Theorem: Square Systems with Exactly One Solution  If A is an invertible square matrix, then the system of equations given by AX = B has the solution X = A -1 B  Proof:

169. 169 Back To Point-Value Representation of Polynomials Theorem 30.1 (Uniqueness of an interpolating polynomial)  For any set of n point-value pairs such that all the x k values are distinct, there is a unique polynomial A(x) of degree-bound n such that y k = A(x k ) for k = 0, 1, …, n-1

170. 170 Point-Value Representation of Polynomials (cont) Proof of Theorem 30.1:  y k = A(x k ) for k = 0, 1, …, n-1 is equivalent to the matrix equation

171. 171 Point-Value Representation of Polynomials (cont) The Vandermonde matrix (shown on previous slide), has the determinant And is therefore invertible if the x k are distinct. Thus, we can solve for vector a: See problem 28.1-11

172. 172 Point-Value Representation of Polynomials (cont) To interpolate by solving the set of linear equations using the standard method of Gaussian Elimination takes  (n 3 ) time A faster algorithm for n-point interpolation based on Lagrange’s formula

173. 173 Point-Value Representation of Polynomials (cont) To compute the coefficients takes time  (n 2 )

174. 174 Point-Value Representation of Polynomials (cont) An example using Lagrange’s Formula Why is this algorithm only  (n 2 ) time?

175. 175 Point-Value Representation of Polynomials (cont) Adding two polynomials:  Given point-value representation for two polynomials A and B:  Then the point-value representation for C:  The time is  (n) The x’s are the same!

176. 176 Point-Value Representation of Polynomials (cont) Multiplying two polynomials:  We just need to multiply the corresponding values of A and B to get the point-value representation of C  A problem arises because the degree- bound of C (2n) is the sum of the degree- bounds for A and B (n) And we need 2n point-value pairs to represent C

177. 177 Point-Value Representation of Polynomials (cont)  So, we must extend A and B to include 2n point-value pairs each  Then the point-value representation for C:  The time is  (n)

178. 178 Best of Both Worlds Can we get "fast" multiplication and evaluation? coefficient Representation O(n 2 ) Multiplication O(n) Evaluation point-valueO(n)O(n 2 ) FFTO(n log n)

179. 179 Best of Both Worlds (cont) Yes! Convert back and forth between two representations O(n) point-value multiplication O(n 2 ) coefficient multiplication O(n log n) evaluation FFT interpolation inverse FFT O(n log n)

180. 180 Best of Both Worlds (cont) Key idea: choose {x 0, x 1,..., x n-1 } to make computation easier This will allow us to convert between representations in  (nlgn) time We will use “complex roots of unity” as the evaluation points

181. 181 Complex Roots of Unity A complex n th root of unity is a complex number  such that  n = 1 There are exactly n many n th roots of unity

182. 182 Complex Roots of Unity (cont) The principal nth root of unity All the complex n th roots of unity are powers of  n For each term,

183. 183 Complex Roots of Unity (cont) Definition of the exponential of a complex number: And We get an n th root of unity by

184. 184 Complex Roots of Unity (cont) But 1 can be written using different arguments as follows:

185. 185 Complex Roots of Unity (cont) Hence dividing the argument in each case by n gives the following n th roots of unity

186. 186 Complex Roots of Unity (cont) An example: Find the cube roots of unity

187. 187 Complex Roots of Unity (cont) Another example: Find the fourth roots of unity

188. 188 Complex Roots of Unity (cont) The n complex roots of unity are equally spaced around the circle of unit radius The cube roots of unity

189. 189 Euler’s Formula: e  i +1 = 0 Properties of Roots of Unity Lemma 30.3 (Cancellations Lemma)  For any integers n  0, k  0, and d > 0,  Proof: Corollary 30.4  For any even integer n > 0,  Proof:

190. 190 Properties of Roots of Unity (cont) Corollary 99  Let n > 0 be even, and let  n and n/2 be the principal n th and ( n / 2 ) th roots of unity. Then  Proof:

191. 191 Properties of Roots of Unity (cont) Lemma 30.5 (Halving lemma)  Let n > 0 be even. Then, the squares of the n complex n th roots of unity are the n / 2 complex ( n / 2 ) th roots of unity.  Proof: If we square all of the n th roots of unity, then each (n/2) th root is obtained exactly twice since:

192. 192 Properties of Roots of Unity (cont) Lemma 30.6 (Summation Lemma)  For any integer n  1 and nonnegative integer k not divisible by n  Proof:

193. 193 The DFT Remember, we want to evaluate a polynomial of degree-bound n at  Actually, since we will eventually multiply, n is actually 2n We are working with complex (2n)th roots of unity  Also assume that n is a power of 2 We can add new high-order zero coefficients as necessary

194. 194 The DFT (cont) Polynomial A is in coefficient form The vector y is the Discrete Fourier Transform of the coefficient vector a

195. 195 The FFT The FFT allows us to compute the DFT n (a) in time  (nlgn) Define “even” and “odd” polynomials:

196. 196 The FFT (cont) Reduces problem of  Evaluating degree n polynomial A(x) at  0,  1,...,  n-1 to  evaluating two degree n/2 polynomials at: (  0 ) 2, (  1 ) 2,..., (  n-1 ) 2. Halving Lemma  A [E] (x) and A [O] (x) evaluated only at n/2 complex (n/2) th roots of unity

197. 197 The FFT Algorithm O(n) time multiplies if we pre-compute  k

198. 198 The FFT Algorithm (cont) Recursion Tree

199. 199 The FFT Algorithm (cont) The recurrence for this algorithm is

200. 200 Proof of the FFT Algorithm We need to show that for each k = 0, …, n-1 Base case: n = 1 Induction step: Assume algorithm correct for n/2

201. 201 Proof of the FFT Algorithm (cont) Let be the principal (n/2) th root of unity

202. 202 Proof of the FFT Algorithm (cont)

203. 203 The Inverse FFT O(n) point-value multiplication O(n 2 ) coefficient multiplication O(n log n) evaluation FFT interpolation inverse FFT O(n log n)

204. 204 The Inverse FFT (cont) Forward FFT  given {a 0, a 1,..., a n-1 } compute {y 0, y 1,..., y n-1 } VnVn

205. 205 The Inverse FFT (cont) Inverse FFT  given {y 0, y 1,..., y n-1 } compute {a 0, a 1,..., a n-1 } V n -1

206. 206 The Inverse FFT (cont) It is the same algorithm as FFT, except use  n -1 as “principal” n th root of unity (and divide by n).

207. 207 The Inverse FFT (cont) Theorem 30.7  For j, k = 0, 1, …, n-1, the (j, k) entry of  Proof: We will show that V n -1 V n = I n Consider the (j, j ) entry of V n -1 V n

208. 208 The Inverse FFT (cont) If j = j, the summation = 1 If j  j, the summation = 0 Therefore, the interpolation/DFT n -1 (y) is By the Summation Lemma

209. 209 The Inverse FFT Algorithm

210. 210 Summary Slides Divide & Conquer Algorithms Recursion Review Designing Algorithms Analyzing Divide & Conquer Algorithms MergeSort Analysis of MergeSort QuickSort Performance of QuickSort

211. 211 Summary Slides (cont.) Randomized QuickSort Analysis of QuickSort Indicator Random Variables Median and Order Statistics The Selection Problem Minimum and Maximum Selection in Expected Linear Time Analysis of Selection Algorithm

212. 212 Summary Slides (cont.) Selection in Worst-Case Linear Time Analyzing “Median of Medians” Review of Sorts Lower Bounds for Sorting The Decision-Tree Model Analysis of Decision-Tree Model Theorem 8.1 Sorting in Linear Time

213. 213 Summary Slides (cont.) Counting Sort Analysis of CountingSort: Radix Sort Analysis of RadixSort Proof by induction that RadixSort works Radix Sort Example Bucket Sort Analysis of Bucket Sort

214. 214 Summary Slides (cont.) Strassen’s Algorithm for Matrix Multiplication An Example of Strassen’s Algorithm Discussion of Strassen’s Algorithm Final Notes on Strassen’s Algorithm Polynomials and FFT Polynomials Review of Complex Numbers

215. 215 Summary Slides (cont.) Coefficient Representation of Polynomials Point-Value Representation of Polynomials Review of Matrix Operations Best of Both Worlds Complex Roots of Unity Properties of Roots of Unity

216. 216 Summary Slides (cont.) The DFT The FFT The FFT Algorithm Proof of the FFT Algorithm The Inverse FFT The Inverse FFT Algorithm