Introduction to Algorithms: Recurrences

CS 421 - Analysis of Algorithms Solving Recurrences Recurrences are like solving integrals, differential equations, etc. Three Common Approaches: Recursion Tree Substitution Master Theorem Next Lecture: Applications of recurrences to divide-and-conquer algorithms. CS 421 - Analysis of Algorithms

Recursion-Tree Method A recursion tree models the costs (time) of a recursive execution of an algorithm. The recursion-tree method can be unreliable, just like any method that uses ellipses (…). The recursion-tree method promotes intuition, however. The recursion tree method is good for generating guesses for the substitution method. CS 421 - Analysis of Algorithms

Example of recursion tree Solve: T(n) = T(n/4) + T(n/2) + n2

Example of Recursion Tree T(n) = T(n/4) + T(n/2) + n2: T(n) CS 421 - Analysis of Algorithms

Example of recursion tree T(n) = T(n/4) + T(n/2) + n2: n2 T(n/4) T(n/2) CS 421 - Analysis of Algorithms

Example of recursion tree T(n) = T(n/4) + T(n/2) + n2: n2 (n/4) 2 (n/2) 2 T(n/16) T(n/8) T(n/8) T(n/4) CS 421 - Analysis of Algorithms

Example of recursion tree T(n) = T(n/4) + T(n/2) + n2: n2 (n/4) 2 (n/2) 2 (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 (1) CS 421 - Analysis of Algorithms

Example of recursion tree T(n) = T(n/4) + T(n/2) + n2: n2 n2 (n/4) 2 (n/2) 2 (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 (1) CS 421 - Analysis of Algorithms

Example of recursion tree T(n) = T(n/4) + T(n/2) + n2: n2 n2 (n/4) 2 5 n (n/2) 2 2 16 (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 (1) CS 421 - Analysis of Algorithms

Example of recursion tree T(n) = T(n/4) + T(n/2) + n2: n2 n2 (n/4) 2 5 n (n/2) 2 2 16 25 n (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 2 256 … (1) CS 421 - Analysis of Algorithms

Example of recursion tree T(n) = T(n/4) + T(n/2) + n2: n2 n2 (n/4) 2 5 n (n/2) 2 2 16 25 n (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 2 256 Total = 𝑛 2 + 5 16 + ( 5 16 ) 2 + ( 5 16 ) 3 + … (1) CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms Geometric Series 𝐹𝑜𝑟 𝑟𝑒𝑎𝑙 𝑥 ≠1, 𝑡ℎ𝑒 𝑠𝑢𝑚𝑚𝑎𝑡𝑖𝑜𝑛: 𝑘=0 𝑛 𝑥 𝑘 =1+𝑥+ 𝑥 2 + …+ 𝑥 𝑛 𝑖𝑠 𝑎 𝑔𝑒𝑜𝑚𝑒𝑡𝑟𝑖𝑐 𝑜𝑟 𝑒𝑥𝑝𝑜𝑛𝑒𝑛𝑡𝑖𝑎𝑙 𝑠𝑒𝑟𝑖𝑒𝑠 𝑎𝑛𝑑 ℎ𝑎𝑠 𝑡ℎ𝑒 𝑣𝑎𝑙𝑢𝑒: 𝑘=0 𝑛 𝑥 𝑘 = 𝑥 𝑛+1 −1 𝑥 −1 CS 421 - Analysis of Algorithms

Geometric Series : Positive Fractions 𝑊ℎ𝑒𝑛 𝑡ℎ𝑒 𝑠𝑢𝑚𝑚𝑎𝑡𝑖𝑜𝑛 𝑖𝑠 𝑖𝑛𝑓𝑖𝑛𝑖𝑡𝑒 𝑎𝑛𝑑 𝑥 <1, 𝑤𝑒 ℎ𝑎𝑣𝑒 𝑎𝑛 𝑖𝑛𝑓𝑖𝑛𝑖𝑡𝑒 𝑑𝑒𝑐𝑟𝑒𝑎𝑠𝑖𝑛𝑔 𝑔𝑒𝑜𝑚𝑒𝑡𝑟𝑖𝑐 𝑠𝑒𝑟𝑖𝑒𝑠: 𝑘=0 ∞ 𝑥 𝑘 = 1 1 − 𝑥 CS 421 - Analysis of Algorithms

Example of recursion tree T(n) = T(n/4) + T(n/2) + n2: n2 n2 (n/4) 2 5 n (n/2) 2 2 16 25 n (n/16) 2 (n/8) 2 (n/8) 2 (n/4) 2 2 256 Total = 𝑛 2 + 16 7 (1) = ( 𝑛 2 ) CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms Substitution method The most general method: Guess the form of the solution. Verify by induction. Solve for constants. CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms Substitution method EXAMPLE: T(n) = 4T(n/2) + n Make assumption that T(1) = (1). Guess O(n3) . Prove O and  separately. Assume that T(k)  ck3 for k < n . Prove T(n)  cn3 by induction. Solve for any constants. CS 421 - Analysis of Algorithms

Example of substitution T (n)  4T (n / 2)  n  4c(n / 2)3  n  (c / 2)n3  n  cn3  ((c / 2)n3  n) desired – residual  cn3 desired whenever (c/2) ∗ n3 – n  0 For example, if c  2 and n  1. residual CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms Example (continued) We must also handle the initial conditions, i.e., ground the induction with base cases. Base: T(n) = (1) for all n < n0, where n0 is a suitable constant. For 1  n < n0, we have “(1)”  cn3, if we pick c big enough. CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms Example (continued) We must also handle the initial conditions, i.e., ground the induction with base cases. Base: T(n) = (1) for all n < n0, where n0 is a suitable constant. For 1  n < n0, we have “(1)”  cn3, if we pick c big enough. This bound is not tight! CS 421 - Analysis of Algorithms

A Tighter Upper Bound Prove that T(n) = O(n2).

CS 421 - Analysis of Algorithms A Tighter Upper Bound We shall prove that T(n) = O(n2). Assume that T(k)  ck2 for k < n T (n)  4T (n / 2)  n  4c(n / 2)2  n  cn2  n  O(n2 ) CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms A Tighter Upper Bound We shall prove that T(n) = O(n2). Assume that T(k)  ck2 for k < n T (n)  4T (n / 2)  n  4c(n / 2)2  n  cn2  n  O(n2 ) Wrong! We must prove the I.H. CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms A Tighter Upper Bound We shall prove that T(n) = O(n2). Assume that T(k)  ck2 for k < n T (n)  4T (n / 2)  n  4c(n / 2)2  n  cn2  n  O(n2 )  cn2  (n) [ desired – residual ]  cn2 CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms A Tighter Upper Bound We shall prove that T(n) = O(n2). Assume that T(k)  ck2 for k < n T (n)  4T (n / 2)  n  4c(n / 2)2  n  cn2  n  O(n2 )  cn2  (n) [ desired – residual ] Lose!  cn2 for no choice of c > 0. CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms A Tighter Upper Bound! IDEA: Strengthen the inductive hypothesis. Subtract a low-order term. Inductive hypothesis: T(k)  c1k2 – c2k for k < n. CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms A Tighter Upper Bound! T(k)  c1k2 – c2k for k < n: T(n) = 4T(n/2) + n = 4(c1(n/2)2 – c2(n/2)) + n = c1n2 – 2c2n + n = c1n2 – c2n – (c2n – n)  c1n2 – c2n, if c2  1. CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms A Tighter Upper Bound! T(k)  c1k2 – c2k for k < n: T(n) = 4T(n/2) + n = 4(c1(n/2)2 – c2(n/2)) + n = c1n2 – 2c2n + n = c1n2 – c2n – (c2n – n)  c1n2 – c2n, if c2  1. Pick c1 big enough to handle the initial conditions. CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms The Master Method The Master Method applies to eventually non-decreasing functions that satisfy recurrences of the form: T(n) = a T 𝑛 𝑏 + f (n) for 𝑛= 𝑏 𝑘 , 𝑘=1, 2, … T(1) = c where a  1, b > 1, c > 0. CS 421 - Analysis of Algorithms

CS 421 - Analysis of Algorithms Three Common Cases If 𝑓 𝑛 ∈ Θ (𝑛 𝑑 ), where 𝑑≥0, then Case 1: 𝑎 <𝑏 𝑑 Solution: T(n) ∈ ( 𝑛 𝑑 ) . Case 2: 𝑎 =𝑏 𝑑 Solution: T(n) = ( 𝑛 𝑑 log n) . Case 3: 𝑎 >𝑏 𝑑 Solution: T(n) ∈ ( 𝑛 𝑙𝑜𝑔𝑏𝑎 ) . CS 421 - Analysis of Algorithms

Examples : Master Method Example 1: T(n) = 4T 𝑛 2 + n a = 4, b = 2 f (n) ∈ Θ 𝑛 ⇒𝑑=1 CASE 3: 4>21  T(n) = ( 𝑛 𝑙𝑜𝑔𝑏𝑎 ) = ( 𝑛 𝑙𝑜𝑔24 ) = ( 𝑛 2 ) CS 421 - Analysis of Algorithms

Examples : Master Method Example 2: T(n) = 4T 𝑛 2 + n2 a = 4, b = 2 f (n) ∈ Θ 𝑛 2 ⇒𝑑=2 CASE 2: 4= 2 2  T(n) = (nd log n) = (n2 log n) CS 421 - Analysis of Algorithms

Examples : Master Method Example 3: T(n) = 4T 𝑛 2 + n3 a = 4, b = 2 f (n) ∈Θ 𝑛 3 ⇒𝑑=3 CASE 1: 4< 2 3  T(n) = (nd) = (n3) CS 421 - Analysis of Algorithms

Examples : Master Method Example 4: T(n) = 4T 𝑛 2 + n2 log n a = 4, b = 2 f (n) ∈ Θ 𝑛 ⇒𝑑= ? Because n not of the form 𝑏 𝑘 , 𝑘=1, 2, …, Master Method does not apply. CS 421 - Analysis of Algorithms