1. 1 Computer Algorithms Lecture 7 Master Theorem Some of these slides are courtesy of D. Plaisted, UNC and M. Nicolescu, UNR

2. Recurrence Relations Equation or an inequality that characterizes a function by its values on smaller inputs. Solution Methods (Chapter 4) –Substitution Method. –Recursion-tree Method. –Master Method. Recurrence relations arise when we analyze the running time of iterative or recursive algorithms. –Ex: Divide and Conquer. T(n) =  (1)if n  c T(n) = a T(n/b) + D(n) + C(n) otherwise 2

3. Master Method Based on the Master theorem. “Cookbook” approach for solving recurrences of the form T(n) = aT(n/b) + f(n) a  1, b > 1 are constants. –a subproblems of size n/b are solved recursively, each in time T(n/b) f(n) is asymptotically positive. –An asymptotically positive function is one that is positive for all –sufciently large n. –f(n) is the cost of dividing the problem and combining the results n/b may not be an integer, but we ignore floors and ceilings. Why? Requires memorization of three cases. Idea: compare f(n) with –f(n) is asymptotically smaller or larger than by a polynomial factor n  –f(n) is asymptotically equal with 3

4. Recursion tree view f(n)f(n) af(n/b) a 2 f(n/b 2 )  (n log b a ) Total: There are leaves Split problem into a parts at log b n levels. 4

5. The master method The master method applies to recurrences of the form T(n) = a T(n/b) + f (n), where a  1, b > 1, and f is asymptotically positive. 5

6. Why ? Assume n = b k  k = log b n At the end of iterations, i = k: to solve the recurrence, we need only to characterize the dominant term

7. Master Theorem Theorem 4.1 Let a  1 and b > 1 be constants, let f(n) be a function, and Let T(n) be defined on nonnegative integers by the recurrence T(n) = aT(n/b) + f(n), where we can replace n/b by  n/b  or  n/b . T(n) can be bounded asymptotically in three cases: 1.If f(n) = O(n log b a–  ) for some constant  > 0, then T(n) =  ( n log b a ). 2.If f(n) =  (n log b a ), then T(n) =  ( n log b a lg n). 3.If f(n) =  (n log b a+  ) for some constant  > 0, and if, for some constant c < 1 and all sufficiently large n, we have a·f(n/b)  c f(n), then T(n) =  (f(n)). Theorem 4.1 Let a  1 and b > 1 be constants, let f(n) be a function, and Let T(n) be defined on nonnegative integers by the recurrence T(n) = aT(n/b) + f(n), where we can replace n/b by  n/b  or  n/b . T(n) can be bounded asymptotically in three cases: 1.If f(n) = O(n log b a–  ) for some constant  > 0, then T(n) =  ( n log b a ). 2.If f(n) =  (n log b a ), then T(n) =  ( n log b a lg n). 3.If f(n) =  (n log b a+  ) for some constant  > 0, and if, for some constant c < 1 and all sufficiently large n, we have a·f(n/b)  c f(n), then T(n) =  (f(n)). Whether the problem cost is dominated by the root or by the leaves 7

8. Master Theorem – What it means? Case 1: If f(n) = O(n log b a–  ) for some constant  > 0, then T(n) =  ( n log b a ). –n log b a = a log b n : Number of leaves in the recursion tree. –f(n) = O(n log b a–  )  Sum of the cost of the nodes at each internal level asymptotically smaller than the cost of leaves by a polynomial factor. –Cost of the problem dominated by leaves, hence cost is  ( n log b a ). 8

9. Master Theorem – What it means? Case 2: If f(n) =  (n log b a ), then T(n) =  ( n log b a lg n). –n log b a = a log b n : Number of leaves in the recursion tree. –f(n) =  (n log b a )  Sum of the cost of the nodes at each level asymptotically the same as the cost of leaves. –There are  (lg n) levels. –Hence, total cost is  ( n log b a lg n). 9

10. Master Theorem – What it means? Case 3: If f(n) =  (n log b a+  ) for some constant  > 0, and if, for some constant c < 1 and all sufficiently large n, we have a·f(n/b)  c f(n), then T(n) =  (f(n)). –n log b a = a log b n : Number of leaves in the recursion tree. –f(n) =  (n log b a+  )  Cost is dominated by the root. Cost of the root is asymptotically larger than the sum of the cost of the leaves by a polynomial factor. –Hence, cost is  ( f(n)). 10

11. Summary Assume n = b k  k = log b n At the end of iterations, i = k: to solve the recurrence, we need only to characterize the dominant term Case 1: Running time dominated by cost at leaves –If f(n) is dominated by : T(n) =  ( ) Case 3: Running evenly distributed throughout the tree –If f(n) dominates : T(n) =  (f(n)) Case 2: Running time dominated by cost at root –If f(n) =  ( ): T(n) 11

12. Strategy Extract a, b, and f(n) from a given recurrence Determine Compare f(n) and asymptotically Determine appropriate MT case, and apply 12

13. Master Method – Examples T(n) = 16T(n/4)+n –a = 16, b = 4, n log b a = n log 4 16 = n 2. –f(n) = n = O(n log b a-  ) = O(n 2-  ), where  = 1  Case 1. –Hence, T(n) =  (n log b a ) =  (n 2 ). T(n) = T(3n/7) + 1 – a = 1, b=7/3, and n log b a = n log 7/3 1 = n 0 = 1 – f(n) = 1 =  (n log b a )  Case 2. –Therefore, T(n) =  (n log b a lg n) =  (lg n) 13

14. Master Method – Examples T(n) = 3T(n/4) + n lg n – a = 3, b=4, thus n log b a = n log 4 3 = O(n 0.793 ) – f(n) = n lg n =  (n log 4 3 +  ) where   0.2  Case 3. –Therefore, T(n) =  (f(n)) =  (n lg n). T(n) = 2T(n/2) + n lg n – a = 2, b=2, f(n) = n lg n, and n log b a = n log 2 2 = n – f(n) is asymptotically larger than n log b a, but not polynomially larger. The ratio lg n is asymptotically less than n  for any positive . Thus, the Master Theorem doesn’t apply here. 14

15. Master Theorem Summarized Given a recurrence of the form The master method cannot solve every recurrence of this form; there is a gap between cases 1 and 2, as well as cases 2 and 3 15

16. f (n/b) Idea of master theorem f (n/b)   (1) … Recursion tree: … f (n)f (n) a f (n/b 2 ) … a h = log b n f (n)f (n) a f (n/b) a 2 f (n/b 2 ) … n log b a   (1) C ASE 1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight.  (n log b a ) 16

17. f (n/b) Idea of master theorem f (n/b)   (1) … Recursion tree: … f (n)f (n) a f (n/b 2 ) … a h = log b n f (n)f (n) a f (n/b) a 2 f (n/b 2 ) … n log b a   (1) C ASE 2: (k = 0) The weight is approximately the same on each of the log b n levels.  (n log b a lg n) 17

18. f (n/b) Idea of master theorem f (n/b)   (1) … Recursion tree: … f (n)f (n) a f (n/b 2 ) … a h = log b n f (n)f (n) a f (n/b) a 2 f (n/b 2 ) … n log b a   (1) C ASE 3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight.  ( f (n)) 18

19. More Examples 19

20. Master Method Case 1 T(n) = a  T(n/b) + f(n) f(n) = O(n log b a -  ) for some  >0  T(n) =  (n log b a ) T(n) =  (n lg 7 ) cn 2 = ? O(n log b a -  ) = O(n log 2 7 -  )  O(n 2.8 -  ) Yes, for any   0.8. T(n) = 7T(n/2) + cn 2 a=7, b=2 20

21. Master Method Case 2 T(n) = a  T(n/b) + f(n) f(n) =  (n log b a )  T(n) =  (n log b a lg n) T(n) = 2T(n/2) + cna=2, b=2 E.g., mergesort. cn = ?  (n log b a ) =  (n log 2 2 ) =  (n) Yes. T(n) =  (n lg n) 21

22. Master Method Case 3 T(n) = a  T(n/b) + f(n) f(n) =  (n log b a +  ) for some  >0 and a  f(n/b)  c  f(n) for some c<1 and all large enough n  T(n) =  (f(n)) T(n) = 4  T(n/2) + n 3 a=4, b=2 n 3 = ?  (n log b a +  ) =  (n log 2 4 +  ) =  (n 2 +  ) Yes, for any   1. 4(n/2) 3 = ½  n 3  ? cn 3 Yes, for any c  ½. I.e., is the constant factor shrinking? T(n) =  (n 3 ) 22

23. Master Method Case 4 T(n) = a  T(n/b) + f(n) None of previous apply. Master method doesn’t help. T(n) = 4T(n/2) + n 2 /lg na=4, b=2 Case 1? n 2 /lg n = ? O(n log b a -  ) = O(n log 2 4 -  ) = O(n 2 -  ) = O(n 2 /n  ) No, since lg n is asymptotically less than n . Thus, n 2 /lg n is asymptotically greater than n 2 /n . 23

24. Examples Ex. T(n) = 4T(n/2) + n a = 4, b = 2  n log b a = n 2 ; f (n) = n. CASE 1: f (n) = O(n 2 –  ) for  = 1.  T(n) =  (n 2 ). Ex. T(n) = 4T(n/2) + n 2 a = 4, b = 2  n log b a = n 2 ; f (n) = n 2. CASE 2: f (n) =  (n 2 lg 0 n), that is, k = 0.  T(n) =  (n 2 lg n). 24

25. Examples Ex. T(n) = 4T(n/2) + n 3 a = 4, b = 2  n log b a = n 2 ; f (n) = n 3. CASE 3: f (n) =  (n 2 +  ) for  = 1 and 4(n/2) 3  cn 3 (reg. cond.) for c = 1/2.  T(n) =  (n 3 ). 25

26. Master Method Case 4 T(n) = a  T(n/b) + f(n) None of previous apply. Master method doesn’t help. T(n) = 4T(n/2) + n 2 /lg na=4, b=2 Case 2? n 2 /lg n = ?  (n log b a ) =  (n log 2 4 ) =  (n 2 ) No. 26

27. Master Method Case 4 T(n) = a  T(n/b) + f(n) None of previous apply. Master method doesn’t help. T(n) = 4T(n/2) + n 2 /lg na=4, b=2 Case 3? n 2 /lg n = ?  (n log b a +  ) =  (n log 2 4 +  ) =  (n 2 +  ) No, since 1/lg n is asymptotically less than n . 27

28. Examples Binary-search(A, p, r, s): q  (p+r)/2 if A[q]=s then return q else if A[q]>s then Binary-search(A, p, q-1, s) else Binary-search(A, q+1, r, s) Binary-search(A, p, r, s): q  (p+r)/2 if A[q]=s then return q else if A[q]>s then Binary-search(A, p, q-1, s) else Binary-search(A, q+1, r, s) 28

29. Examples (2) 29

30. Examples (3) 30