Recurrence Relation

Outline  What is a recurrence relation ?  Solving linear recurrence relations  Divide-and-conquer algorithms and recurrence relations

What is a recurrence relation ?

Definition A recurrence relation for the sequence {a n } is an equation that expresses a n in terms of one or more of the previous terms of the sequence, namely a 0, a 1, …, a n-1, for all integers n  n 0, where n 0 is a non- negative integer. A sequence is called a solution of a recurrence relation if its terms satisfy the recurrence relation. Example: Given the recurrence relation a n =2a n-1 - a n-2. 3, 5, 7, 9, 11, … satisfies the recurrence relation. 2, 3, 4, 5, 6, … also satisfies the recurrence relation.

Example: Compound interest Let P n be the amount of money in the account after n years. If you initially deposit P n =1000 in your account and the bank gives k % per year with interest compounded annually. P n = (1+k/100) P n-1 P 1 = (1+k/100) P 0 = (1+k/100) P 0 P 2 = (1+k/100) P 1 = (1+k/100) 2 P 0 P 3 = (1+k/100) P 2 = (1+k/100) 3 P 0 P n = (1+k/100) P n-1 = (1+k/100) n P 0

Example: Fibonacci numbers  A pair of rabbits is placed on an island. After 2 months old, a pair of rabbits produces another pair each month. How many rabbits are there after n months? f n =f n-1 + f n-2 month1-month rabbits >=2-month rabbits Total f n

Tower of Hanoi

Example: Tower of Hanoi H(1) = 1 H(n) = 2H(n-1) + 1 H(n) = 2(2H(n-2) +1) +1 = 4H(n-2) = 4(2(H(n-3) + 1) = 8H(n-3) = 8(2H(n-4) +1) = 16H(n-4) = 2 n n n-3 + … + 1 = 2 n - 1 Let x = 2 n n-2 + … x +1 = (2 n n-2 + … ) +1 x + 1 = 2 n n-2 + … x + 1 = 2 n n-2 + … x + 1 = 2 n n-2 + … + 8 x + 1 = 2 n x = 2 n - 1

Linear recurrence relation Homogeneous and non-homogeneous

What is a linear recurrence relation ? Linear recurrence relation of degree k a n = c 1 a n-1 + c 2 a n-2 + … + c k a n-k + F(n)  c i is a constant, for i = 1, 2, …, k  c k  0 Linear homogeneous recurrence relation  F(n) = 0 Linear non-homogeneous recurrence relation  F(n)  0

Examples Linear homogeneous  a n = 1.2 a n-1 : degree 1  f n = f n-1 + f n-2 : degree 2  a n = 3a n-3 : degree 3 Non-linear homogeneous  a n = a 2 n-1 + a n-2  a n = na n-1 - 2a n-2 Linear non-homogeneous  a n = a n n  h n = 2h n  a n = 3a n-1 + n Non-linear non-homogeneous  a n = a 2 n n  a n = n 2 a n-1 + n

Solving linear homogeneous recurrence relation

Theorem  The sequence { a n } is a solution of the recurrence relation a n = c 1 a n-1 + c 2 a n-2 iff a n =  1 r n 1 +  2 r n 2, where : r 1 and r 2 are two distinct roots of r 2 - c 1 r - c 2 = 0, a 0 =  1 +  2 a 1 =  1 r 1 +  2 r 2 [  1 = (a 1 – a 0 r 2 )/(r 1 - r 2 )  2 = (a 0 r 1 –a 1 )/(r 1 - r 2 ) ]

Informal proof Recurrence relation: a n = c 1 a n-1 + c 2 a n-2 Let r 1 and r 2 be two distinct roots of r 2 – c 1 r – c 2 = 0. Let   and   be constants.  Show if a n =  1 r n 1 +  2 r n 2, the sequence { a n } is a solution of the recurrence relation.  Show if the sequence { a n } is a solution of the recurrence relation, then a n =  1 r n 1 +  2 r n 2.

Informal proof Show if a n =  1 r n 1 +  2 r n 2, the sequence { a n } is a solution of the recurrence relation. Let a n = c 1 a n-1 + c 2 a n-2 be a recurrence relation. Let r 1 and r 2 be two distinct roots of r 2 – c 1 r – c 2 = 0. Let   and   be constants. Since r 1 and r 2 are roots of r 2 - c 1 r - c 2 = 0, r c 1 r 1 - c 2 = 0 (r 1 2 = c 1 r 1 + c 2 ) and r c 1 r 2 - c 2 = 0 ( r 2 2 = c 1 r 2 + c 2 ).

Informal proof r 1 2 = c 1 r 1 + c 2 and r 2 2 = c 1 r 2 + c 2. c 1 a n-1 + c 2 a n-2 = c 1 (  1 r 1 n-1 +  2 r 2 n-1 ) + c 2 (  1 r 1 n-2 +  2 r 2 n-2 ) = c 1  1 r 1 n-1 + c 1  2 r 2 n-1 + c 2  1 r 1 n-2 + c 2  2 r 2 n-2 = c 1  1 r 1 n-1 + c 2  1 r 1 n-2 +c 1  2 r 2 n-1 + c 2  2 r 2 n-2 =  1 r 1 n-2 (c 1 r 1 + c 2 ) +  2 r 2 n-2 (c 1 r 2 + c 2 ) =  1 r 1 n-2 r  2 r 2 n-2 r 2 2 =  1 r 1 n +  2 r 2 n = a n

Informal proof  Show if the sequence { a n } is a solution of the recurrence relation, then there are  1 and  2 such that a n =  1 r n 1 +  2 r n 2.  From the initial conditions a 0 and a 1, a 0 =  1 +  2 and a 1 =  1 r 1 +  2 r 2  2 = a 0 -  1  1 r 1 = a 1 -  2 r 2 = a 1 - (a 0 -  1 ) r 2 = a 1 – a 0 r 2 +  1 r 2  1 (r 1 - r 2 ) = a 1 – a 0 r 2 >>  1 = (a 1 – a 0 r 2 )/(r 1 - r 2 )  2 =a 0 -  1 =a 0 -(a 1 –a 0 r 2 )/(r 1 -r 2 ) = a 0 r 1 -a 0 r 2 -a 1 +a 0 r 2 /(r 1 -r 2 )  2 = (a 0 r 1 -a 1 ) / (r 1 -r 2 )

Example What is the solution of a n = a n-1 + 2a n-2 with a 0 =2 a 1 =7 ? We have c 1 = 1 and c 2 = 2. The characteristic equation is r 2 - r - 2 = 0, with roots r 1 = 2 and r 2 = -1.  1 = (a 1 – a 0 r 2 )/(r 1 - r 2 ) = (7+2)/(2+1) = 3  2 = (a 0 r 1 – a 1 )/(r 1 - r 2 ) = (4-7)/(2+1) = -1 a n =  1 r n 1 +  2 r n 2 = 3  2 n - (-1) n

Example: Fibonacci number What is the solution of a n = a n-1 + a n-2 with a 0 =0 a 1 =1 ? We have c 1 = 1 and c 2 = 1. The characteristic equation is r 2 - r - 1 = 0, with roots r 1 = (1+  5)/2 and r 2 = (1-  5)/2.  1 = (a 1 – a 0 r 2 )/(r 1 - r 2 ) = (1-0)/  5 = 1/  5  2 = (a 0 r 1 – a 1 )/(r 1 - r 2 ) = (0-1)/  5 = -1/  5 a n =  1 r n 1 +  2 r n 2 = ((1+  5)/2) n /  5 - ((1-  5)/2) n /  5 a n = ( (1+  5) n - (1-  5) n )/(  5  2 n )

Theorem  The sequence { a n } is a solution of the recurrence relation a n = c 1 a n-1 + c 2 a n-2 iff a n =  1 r n 0 +  2 nr n 0, where : r 0 is the only root of r 2 - c 1 r - c 2 = 0, a 0 =  1 a 1 =  1 r 0 +  2 r 0 = r 0 (  1 +  2 ) [  1 = a 0  2 = a 1 /r 0 – a 0 ]

Example What is the solution of a n = 6a n-1 -9a n-2 with a 0 =1 a 1 =6 ? We have c 1 = 6 and c 2 = -9. The characteristic equation is r 2 - 6r + 9 = 0, with roots r 0 = 3.  1 = a 0 = 1  2 = a 1 /r 0 – a 0 = 6/3-1 = 1 a n =  1 r n 0 +  2 nr n 0 = 3 n + n 3 n = 3 n (1+ n).

Theorem  The sequence { a n } is a solution of the recurrence relation a n = c 1 a n-1 + c 2 a n-2 +… + c k a n-k iff a n =  1 r n 1 +  2 r n 2 +…+  k r n k, where : r 1, r 2,…, r k are k distinct roots of r k -c 1 r k-1 - c 2 r k-2 -…- c k = 0, a 0 =  1 +  2 +…+  k a 1 =  1 r 1 +  2 r 2 +…+  k r k a 2 =  1 r  2 r 2 2 +…+  k r 2 k …

Theorem  The sequence { a n } is a solution of the recurrence relation a n = c 1 a n-1 + c 2 a n-2 +… + c k a n-k iff a n =  1 r n 1 +  2 r n 2 +…+  k r n k, where : r 1, r 2,…, r k are k distinct roots of r k -c 1 r k-1 - c 2 r k-2 -…- c k = 0, a 0 =  1 +  2 +…+  k a 1 =  1 r 1 +  2 r 2 +…+  k r k a 2 =  1 r  2 r 2 2 +…+  k r 2 k …

Theorem  The sequence { a n } is a solution of the recurrence relation a n = c 1 a n-1 + c 2 a n-2 +… + c k a n-k iff a n = (  1,0 +  1,1 n + …+  1,m 1 n m 1 -1 ) r n 1 + (  2,0 +  2,1 n + …+  2,m 2 n m 2 -1 ) r n 2 +…+ (  t,0 +  t,1 n + …+  t,m t n m t -1 ) r n t, where : r 1, r 2,…, r t are t distinct roots of r k -c 1 r k-1 - c 2 r k-2 -…- c k = 0, with multiplicities m 1, m 2,…, m t

Example What is the solution of a n =6a n-1 -11a n-2 +6a n-3 with a 0 =2, a 1 =5, a 2 =15 ? We have c 1 = 6, c 2 = -11 and c 3 = 6. The characteristic equation is r 3 - 6r r - 6 = 0, with roots r 1 = 1, r 2 = 2 and r 3 = 3. a 0 =  1 +  2 +  3 a 1 =  1 r 1 +  2 r 2 +  3 r 3 =    3 a 2 =  1 r  2 r  3 r 3 2 =    3  1 = 1,  2 = -1,  3 = 2 a n = 1 – 2 n + 2  3 n

Solving linear non-homogeneous recurrence relation

Theorem If the sequence { a n (p) } is a particular solution of the recurrence relation a n = c 1 a n-1 + c 2 a n-2 +…+ c k a n-k + F(n) then every solution is of the form {a n (p) + a n (h) }, where { a n (h) } is a solution of the associated homogeneous recurrence relation a n = c 1 a n-1 + c 2 a n-2 +…+ c k a n-k.

Example What are the solutions of a n = 3a n-1 + 2n with a 1 =3 ? We have c 1 = 3. The associated characteristic eq n is r-3=0, with root r = 3. Then, a n (h) =  3 n. Let p n = cn +d. Then, from a n = 3a n-1 + 2n, cn +d = 3(c(n-1)+d ) + 2n. Thus, c = -1 d = -3/2, and a n (p) = -n -3/2 a n = a n (h) + a n (p) =  3 n -n -3/2. From a 1 = 3 = 3  /2,  = 11/6. The solution is a n = (11/6)  3 n -n -3/2.

Example: summation from 1 to n What are the solutions of a n = a n-1 + n with a 1 =1 ? We have c 1 = 1. The associated characteristic eq n is r-1=0, with root r = 1. Then, a n (h) =  1 n = . Let p n = n(cn +d) = cn 2 +dn. Then, from a n = a n-1 + n, cn 2 +dn = c(n-1) 2 +d (n-1) + n. Thus, cn 2 +dn = cn 2 -2cn+c +d n - d + n, c-d + n(1-2c)=0. c-d =0 and 1-2c=0. That is, c=d=1/2. a n = a n (h) + a n (p) =  + n(n+1)/2. From a 1 = 1 =  +1,  = 0. The solution is a n = n(n+1)/2.

Divide-and-conquer Algorithms

Divide-and-Conquer Concept To solve a problem of size n, we solve a sub- problems of size n/b, and combine solutions from sub-problems. X X1X1 X2X2 X3X3 XaXa... n/b n elements

Binary Search  To search for an element a from a sorted list X of size n elements. If a = m, then stop If a < m, then search for a from X 1. If a > m, then search for a from X 2. X X1X1 X2X2 m

Fast multiplication of large integers  a = 2 n A 1 + A 0. b = 2 n B 1 + B 0. A 1 A 0 B 1 B 0 a b = 2 2n A 1 B n (A 0 B 1 + A 1 B 0 ) +A 0 B 0. a b = (2 2n + 2 n ) A 1 B n (A 1 -A 0 )(B 0 – B 1 ) + (2 n +1) A 0 B 0. a b = 2 2n A 1 B n A 1 B n A 1 B 0 –2 n A 0 B 0 +2 n A 0 B 1 – 2 n A 1 B n A 0 B 0 + A 0 B

Recurrence Relations In Divide-and-conquer algorithms

Divide-and-Conquer Recurrence Relations Let f ( n ) be the number of operations required to solve a problem of size n. f ( n ) = a f(n/b ) + g(n). X X1X1 X2X2 X3X3 XaXa... n/b n elements

Binary Search  To search for an element a from a sorted list X of size n elements. If a = m, then stop If a < m, then search for a from X 1. If a > m, then search for a from X 2. Let f ( n ) be the # of comparison in binary search within n elements. f ( n ) = f ( n/2 ) + 2. X X1X1 X2X2 m

Fast multiplication of large integers  a = 2 n A 1 + A 0. b = 2 n B 1 + B 0. A 1 A 0 B 1 B 0 a b = (2 2n + 2 n ) A 1 B n (A 1 -A 0 )(B 0 – B 1 ) + (2 n +1) A 0 B 0. Let f ( n ) be the number of operations needed to multiply two n -bit integers. f ( 2n ) = 3f ( n ) + Cn

Theorem Let f be an increasing function that satisfies the recurrence relation f(n) = a f(n/b) + c whenever n is divisible by b, b is an integer greater than 1, a  1, c is a positive real number. When n=b k, f(n) = t n log b a + s, for some t and s. When a>1, f(n) is O(n log b a ). When a=1, f(n) is O(log n).

Examples Binary Search: f ( n ) = f ( n/2 ) + 2. Because a=1, f(n) is O(log n). Find Max. : f ( n ) = 2f ( n/2 ) + 2. Because a>1, f(n) is O(n log 1 1 ) = O(n).

Master Theorem Let f be an increasing function that satisfies the recurrence relation f(n) = a f(n/b) + cn d whenever n=b k for some positive integer k, b is an integer greater than 1, a  1, c is a positive real number, d is a non-negative real number. When a<b d, f(n) is O(n d ). When a=b d, f(n) is O(n d log n). When a>b d, f(n) is O(n log b a ).

Examples Merge sort: f ( n ) = 2f ( n/2 ) + n. a = 2, b = 2, c = 1, d = 1 Because a=b d, f(n) is O(n log n). Integer multiplication: f ( n ) = 3f ( n/2 ) + Cn a = 3, b = 2, c = C, d = 1 Because a>b d, f(n) is O(n log 2 3 ).