1. Introduction to Algorithms: Recurrences

2. CS 421 - Analysis of Algorithms
Solving Recurrences
Recurrences are like solving integrals, differential equations, etc.
Three Common Approaches:
Recursion Tree
Substitution
Master Theorem
Next Lecture: Applications of recurrences to divide-and-conquer algorithms.
CS Analysis of Algorithms

3. Recursion-Tree Method
A recursion tree models the costs (time) of a recursive execution of an algorithm.
The recursion-tree method can be unreliable, just like any method that uses ellipses (…).
The recursion-tree method promotes intuition, however.
The recursion tree method is good for generating guesses for the substitution method.
CS Analysis of Algorithms

4. Example of recursion tree
Solve:
T(n) = T(n/4) + T(n/2) + n2

5. Example of Recursion Tree
T(n) = T(n/4) + T(n/2) + n2:
T(n)
CS Analysis of Algorithms

6. Example of recursion tree
T(n) = T(n/4) + T(n/2) + n2: n2
T(n/4)
T(n/2)
CS Analysis of Algorithms

7. Example of recursion tree
T(n) = T(n/4) + T(n/2) + n2: n2
(n/4) 2
(n/2) 2
T(n/16)
T(n/8)
T(n/8)
T(n/4)
CS Analysis of Algorithms

8. Example of recursion tree
T(n) = T(n/4) + T(n/2) + n2: n2
(n/4) 2
(n/2) 2
(n/16) 2
(n/8) 2
(n/8) 2
(n/4) 2
(1)
CS Analysis of Algorithms

9. Example of recursion tree
T(n) = T(n/4) + T(n/2) + n2: n2 n2
(n/4) 2
(n/2) 2
(n/16) 2
(n/8) 2
(n/8) 2
(n/4) 2
(1)
CS Analysis of Algorithms

10. Example of recursion tree
T(n) = T(n/4) + T(n/2) + n2: n2 n2
(n/4) 2
5 n
(n/2) 2 2 16
(n/16) 2
(n/8) 2
(n/8) 2
(n/4) 2
(1)
CS Analysis of Algorithms

11. Example of recursion tree
T(n) = T(n/4) + T(n/2) + n2: n2 n2
(n/4) 2
5 n
(n/2) 2 2 16
25 n
(n/16) 2
(n/8) 2
(n/8) 2
(n/4) 2 2
256 …
(1)
CS Analysis of Algorithms

12. Example of recursion tree
T(n) = T(n/4) + T(n/2) + n2: n2 n2
(n/4) 2
5 n
(n/2) 2 2 16
25 n
(n/16) 2
(n/8) 2
(n/8) 2
(n/4) 2 2
256
Total = 𝑛 ( 5 16 ) 2 + ( 5 16 ) 3 + …
(1)
CS Analysis of Algorithms

13. CS 421 - Analysis of Algorithms
Geometric Series
𝐹𝑜𝑟 𝑟𝑒𝑎𝑙 𝑥 ≠1, 𝑡ℎ𝑒 𝑠𝑢𝑚𝑚𝑎𝑡𝑖𝑜𝑛:
𝑘=0 𝑛 𝑥 𝑘 =1+𝑥+ 𝑥 2 + …+ 𝑥 𝑛
𝑖𝑠 𝑎 𝑔𝑒𝑜𝑚𝑒𝑡𝑟𝑖𝑐 𝑜𝑟 𝑒𝑥𝑝𝑜𝑛𝑒𝑛𝑡𝑖𝑎𝑙 𝑠𝑒𝑟𝑖𝑒𝑠 𝑎𝑛𝑑 ℎ𝑎𝑠 𝑡ℎ𝑒 𝑣𝑎𝑙𝑢𝑒:
𝑘=0 𝑛 𝑥 𝑘 = 𝑥 𝑛+1 −1 𝑥 −1
CS Analysis of Algorithms

14. Geometric Series : Positive Fractions
𝑊ℎ𝑒𝑛 𝑡ℎ𝑒 𝑠𝑢𝑚𝑚𝑎𝑡𝑖𝑜𝑛 𝑖𝑠 𝑖𝑛𝑓𝑖𝑛𝑖𝑡𝑒 𝑎𝑛𝑑 𝑥 <1, 𝑤𝑒 ℎ𝑎𝑣𝑒 𝑎𝑛 𝑖𝑛𝑓𝑖𝑛𝑖𝑡𝑒 𝑑𝑒𝑐𝑟𝑒𝑎𝑠𝑖𝑛𝑔 𝑔𝑒𝑜𝑚𝑒𝑡𝑟𝑖𝑐 𝑠𝑒𝑟𝑖𝑒𝑠:
𝑘=0 ∞ 𝑥 𝑘 = − 𝑥
CS Analysis of Algorithms

15. Example of recursion tree
T(n) = T(n/4) + T(n/2) + n2: n2 n2
(n/4) 2
5 n
(n/2) 2 2 16
25 n
(n/16) 2
(n/8) 2
(n/8) 2
(n/4) 2 2
256
Total = 𝑛
(1)
= ( 𝑛 2 )
CS Analysis of Algorithms

16. CS 421 - Analysis of Algorithms
Substitution method
The most general method:
Guess the form of the solution.
Verify by induction.
Solve for constants.
CS Analysis of Algorithms

17. CS 421 - Analysis of Algorithms
Substitution method
EXAMPLE:
T(n) = 4T(n/2) + n
Make assumption that T(1) = (1).
Guess O(n3) .
Prove O and  separately.
Assume that T(k)  ck3 for k < n .
Prove T(n)  cn3 by induction.
Solve for any constants.
CS Analysis of Algorithms

18. Example of substitution
T (n)  4T (n / 2)  n
 4c(n / 2)3  n
 (c / 2)n3  n
 cn3  ((c / 2)n3  n)
desired – residual
 cn3
desired
whenever (c/2) ∗ n3 – n  0
For example, if c  2 and n  1.
residual
CS Analysis of Algorithms

19. CS 421 - Analysis of Algorithms
Example (continued)
We must also handle the initial conditions, i.e., ground the induction with base cases.
Base: T(n) = (1) for all n < n0, where n0
is a suitable constant.
For 1  n < n0, we have “(1)”  cn3, if we pick c big enough.
CS Analysis of Algorithms

20. CS 421 - Analysis of Algorithms
Example (continued)
We must also handle the initial conditions, i.e., ground the induction with base cases.
Base: T(n) = (1) for all n < n0, where n0
is a suitable constant.
For 1  n < n0, we have “(1)”  cn3, if we pick c big enough.
This bound is not tight!
CS Analysis of Algorithms

21. A Tighter Upper Bound
Prove that T(n) = O(n2).

22. CS 421 - Analysis of Algorithms
A Tighter Upper Bound
We shall prove that
T(n) = O(n2).
Assume that T(k)  ck2 for k < n T (n)  4T (n / 2)  n
 4c(n / 2)2  n
 cn2  n
 O(n2 )
CS Analysis of Algorithms

23. CS 421 - Analysis of Algorithms
A Tighter Upper Bound
We shall prove that
T(n) = O(n2).
Assume that T(k)  ck2 for k < n T (n)  4T (n / 2)  n
 4c(n / 2)2  n
 cn2  n
 O(n2 )
Wrong! We must prove the I.H.
CS Analysis of Algorithms

24. CS 421 - Analysis of Algorithms
A Tighter Upper Bound
We shall prove that
T(n) = O(n2).
Assume that T(k)  ck2 for k < n T (n)  4T (n / 2)  n
 4c(n / 2)2  n
 cn2  n
 O(n2 )
 cn2  (n)
[ desired – residual ]
 cn2
CS Analysis of Algorithms

25. CS 421 - Analysis of Algorithms
A Tighter Upper Bound
We shall prove that
T(n) = O(n2).
Assume that T(k)  ck2 for k < n T (n)  4T (n / 2)  n
 4c(n / 2)2  n
 cn2  n
 O(n2 )
 cn2  (n)
[ desired – residual ]
Lose!
 cn2
for no choice of c > 0.
CS Analysis of Algorithms

26. CS 421 - Analysis of Algorithms
A Tighter Upper Bound!
IDEA: Strengthen the inductive hypothesis.
Subtract a low-order term.
Inductive hypothesis:
T(k)  c1k2 – c2k for k < n.
CS Analysis of Algorithms

27. CS 421 - Analysis of Algorithms
A Tighter Upper Bound!
T(k)  c1k2 – c2k for k < n:
T(n) = 4T(n/2) + n
= 4(c1(n/2)2 – c2(n/2)) + n
= c1n2 – 2c2n + n
= c1n2 – c2n – (c2n – n)
 c1n2 – c2n, if c2  1.
CS Analysis of Algorithms

28. CS 421 - Analysis of Algorithms
A Tighter Upper Bound!
T(k)  c1k2 – c2k for k < n:
T(n) = 4T(n/2) + n
= 4(c1(n/2)2 – c2(n/2)) + n
= c1n2 – 2c2n + n
= c1n2 – c2n – (c2n – n)
 c1n2 – c2n, if c2  1.
Pick c1 big enough to handle the initial conditions.
CS Analysis of Algorithms

29. CS 421 - Analysis of Algorithms
The Master Method
The Master Method applies to eventually non-decreasing functions that satisfy recurrences of the form:
T(n) = a T 𝑛 𝑏 + f (n)
for 𝑛= 𝑏 𝑘 , 𝑘=1, 2, …
T(1) = c
where a  1, b > 1, c > 0.
CS Analysis of Algorithms

30. CS 421 - Analysis of Algorithms
Three Common Cases
If 𝑓 𝑛 ∈ Θ (𝑛 𝑑 ), where 𝑑≥0,
then
Case 1: 𝑎 <𝑏 𝑑
Solution: T(n) ∈ ( 𝑛 𝑑 ) .
Case 2: 𝑎 =𝑏 𝑑
Solution: T(n) = ( 𝑛 𝑑 log n) .
Case 3: 𝑎 >𝑏 𝑑
Solution: T(n) ∈ ( 𝑛 𝑙𝑜𝑔𝑏𝑎 ) .
CS Analysis of Algorithms

31. Examples : Master Method
Example 1: T(n) = 4T 𝑛 n
a = 4, b = 2
f (n) ∈ Θ 𝑛 ⇒𝑑=1
CASE 3: 4>21
 T(n) = ( 𝑛 𝑙𝑜𝑔𝑏𝑎 )
= ( 𝑛 𝑙𝑜𝑔24 )
= ( 𝑛 2 )
CS Analysis of Algorithms

32. Examples : Master Method
Example 2: T(n) = 4T 𝑛 n2
a = 4, b = 2
f (n) ∈ Θ 𝑛 2 ⇒𝑑=2
CASE 2: 4= 2 2
 T(n) = (nd log n)
= (n2 log n)
CS Analysis of Algorithms

33. Examples : Master Method
Example 3: T(n) = 4T 𝑛 n3
a = 4, b = 2
f (n) ∈Θ 𝑛 3 ⇒𝑑=3
CASE 1: 4< 2 3
 T(n) = (nd)
= (n3)
CS Analysis of Algorithms

34. Examples : Master Method
Example 4: T(n) = 4T 𝑛 n2 log n
a = 4, b = 2
f (n) ∈ Θ 𝑛 ⇒𝑑= ?
Because n not of the form 𝑏 𝑘 , 𝑘=1, 2, …,
Master Method does not apply.
CS Analysis of Algorithms