Instructor Neelima Gupta ngupta@cs.du.ac.in MS 101: Algorithms Instructor Neelima Gupta ngupta@cs.du.ac.in

Mathematical Induction: Review Growth Functions Table Of Contents Mathematical Induction: Review Growth Functions

Review: Mathematical Induction Suppose S(k) is true for fixed constant k Often k = 0 S(n)  S(n+1) for all n >= k Then S(n) is true for all n >= k

Proof By Mathematical Induction Claim:S(n) is true for all n >= k Basis: Show formula is true when n = k Inductive hypothesis: Assume formula is true for an arbitrary n Step: Show that formula is then true for n+1

Strong Induction Strong induction also holds Another variation: Basis: show S(0) Hypothesis: assume S(k) holds for arbitrary k <= n Step: Show S(n+1) follows Another variation: Basis: show S(0), S(1) Hypothesis: assume S(n) and S(n+1) are true Step: show S(n+2) follows

Lets do it Prove 1 + 2 + 3 + … + n = n(n+1) / 2 Prove a0 + a1 + … + an = (an+1 - 1)/(a - 1) for all a  1

Growth Functions Big O Notation In general a function Formally f(n) is O(g(n)) if there exist positive constants c and n0 such that f(n)  c  g(n) for all n  n0 Formally O(g(n)) = { f(n):  positive constants c and n0 such that f(n)  c  g(n)  n  n0 Intuitively, it means f(n) grows no faster than g(n). Examples: n^2, n^2 – n n^3, n^3 – n^2 – n

f(n) = n2 g(n) = n2 – n Is f(n) = O(g(n))? Sol: g(n) = n2 – n = n2/2 + n2/2 - n ≥ n2/2 for n ≥ 2 = ½ f(n) f(n) ≤ 2g(n) for n ≥ 2 Hence, f(n) = O(g(n))

f(n) = n3 g(n) = n3 - n2 – n Is f(n) = O(g(n))? Sol: g(n) = n3 - n2 – n = n3/2 + n3/2 - n2 - n ≥ n3/2 + n3/2 - n3/4 - n3/4 for n ≥ 4 = ½ f(n) f(n) ≤ 2g(n) for n ≥ 2 Hence, f(n) = O(g(n))

f(n) = n3 g(n) = n3 - n2 – n Is g(n) = O(f(n))? Sol: Clearly, g(n) = n3 - n2 – n ≤3 n3 for all n ≥ 1 = 3 f(n) g(n) ≤ 3f(n) V n ≥ 1 Hence, g(n) = O(f(n)).

Omega Notation In general a function f(n) is (g(n)) if  positive constants c and n0 such that 0  cg(n)  f(n)  n  n0 Intuitively, it means f(n) grows at least as fast as g(n). Examples: n^2, n^2 + n n^3, n^3 + n^2 – n

Ques: f(n) = n2, g(n) = n2 + n Is f(n) = Ω(g(n))? Sol: n2 = ½ (n2 + n2 ) ≥ ½ (n2 + n ) f(n) ≥ c g(n) for c = ½ m =1

f(n) = n3 g(n) = n3 + 4n2 - 5n Is g(n) = Ω(f(n))? Sol: g(n) = n3 + 4n2 - 5n ≥ n3 /2 + n3 /2 - 4 n2 - 5n2 for all n ≥ 1 = n3 /2 + n2 (n/2 - 9) ≥ n3 /2 for n ≥ 18 = 1/2 f(n) Hence, g(n) = Ω (f(n)).

Theta Notation A function f(n) is (g(n)) if  positive constants c1, c2, and n0 such that c1 g(n)  f(n)  c2 g(n)  n  n0

Relations Between Q, W, O Theorem : For any two functions g(n) and f(n), f(n) = (g(n)) iff f(n) = O(g(n)) and f(n) = (g(n)). Proof: Simple. Do it.

Assignment No 1 Q1 Already proved a0 + a1 + … + an = (an+1 - 1)/(a - 1) for all a  1 What is the sum for a = 2/3 as n  infinity? Is it O(1)? Is it big or small? For a = 2, is the sum = O(2^n)? Is it big or small? Q2 Show that a polynomial of degree k = theta(n^k).

Other Asymptotic Notations A function f(n) is o(g(n)) if for every positive constant c, there exists a constant n0 > 0 such that f(n) < c g(n)  n  n0 A function f(n) is (g(n)) if for every positive constant c, there exists a constant n0 > 0 such that c g(n) < f(n)  n  n0 Intuitively, () is like > () is like  () is like = o() is like < O() is like 

Arrange some functions f(n) = O(g(n)) => f(n) = o(g(n)) ? Is the converse true? Let us arrange the following functions in ascending order (assume log n = o(n) is known) n, n^2, n^3, sqrt(n), n^epsilon, log n, log^2 n, n log n, n/log n, 2^n, 3^n

Relation between n & n2 Solution: let c > 0 be any constant such that n < c n2 i.e. 1 < c n i.e. n > 1 / c Hence n < c n2 for n > 1 / c i.e. n = o( n2) we can also write it as n < n2.

Relation between n2 & n3 Solution: let c> 0 be any constant such that n2 < c n3 Þ 1 < c n Þ n > 1 / c Hence n2 < c n3 " n > 1 / c i.e. n2 = o( n3) we can also write it as n2 < n3. Combining the previous result we can write n < n2 < n3

Relation between n & n1/2 i.e. n2 = o( n3) Solution: let c> 0 be any constant such that n1/2 < c n i.e n < c2 n2 i.e 1 < c2 n i.e n > 1/ c2 i.e. n1/2 < c n for n > 1/c2 i.e. n2 = o( n3) we can also write it as n1/2 < n. Combining the previous result n1/2 < n < n2 < n3

Relation between n & log n For the time being we can assume the result log ( n ) = o(n) Þ log ( n ) < n we will prove it later.

Relation between n1/2 & log n Assume log n = o(n) let c > 0 be any constant for c/2 > 0 there exists m > 0 such that log n < (c/2) n for n > m changing variables from n to n1/2 we get log(n1/2 ) < (c/2) n1/2 for n1/2 > m ½ log( n ) < (c/2) n1/2 for n > m2

Contd.. let m2 = k log( n ) < c n1/2 for n > k Since c > 0 was chosen arbitrarily hence log n = o( n1/2 ) or log n < n1/2 Combining the results we get log n < n1/2 < n < n2 < n3

Relation between n2 & nlog n Since log n = o(n) for c > 0,  n0 > 0 such that  n  n0, we have log n < c n Multiplying by n on both sides we get n log( n ) < c n2  n  n0 nlog n = o( n2 ) nlog n < n2

Relation between n & nlog n Solution: let c> 0 be any constant such that n < c n log (n) Þ 1 < c log( n ) Þ log( n) > 1 / c Þ n > e1/c i.e. n < c n log n " n > e1/c Since c was chosen arbitrarily \ n = o( n log n ) or n < n log n Combining the results we can get log n < n1/2 < n < n logn < n2 < n3

Relation between n & n/log n We know that n = o(nlogn) for c > 0,  n0 > 0 such that  n  n0, we have n < c n log n dividing both sides by log n we get n/ log( n) < c n  n  n0 Þ n / logn = o(n) i.e. n / logn < n

Assignment No 2 Show that log^M n = o(n^epsilon) for all constants M>0 and epsilon > 0. Assume that log n = o(n). Also prove the following Corollary: log n = o(n/log n) Show that n^epsilon = o(n/logn ) for every 0 < epsilon < 1 .

Hence we have, log n < n/log n < n1/2 < n < n logn < n2 < n3

Assignment No 3 Show that Show that log n = o(n). lim f(n)/g(n) = 0 => f(n) = o(g(n)). n → ∞ lim f(n)/g(n) = c => f(n) = θ(g(n)). n → ∞, where c is a positive constant. Show that log n = o(n). Show that n^k = o(2^n) for every positive constant k.

Show by definition of ‘small o’ that a^n = o(b^n) whenever a < b , a and b are positive constants. Hence we have, log n < n/log n < n1/2 < n < n logn < n2 < n3 <2n < 3n

Why the constants ‘c’ and ‘m’? Suppose we have two algorithms to solve the problem say sorting: Insertion Sort and Merge sort for eg. Why should we have more than one algorithm to solve the same problem? Ans: efficiency. What’s the measure of efficiency? Ans: System resources for example ‘time’. How do we measure time?

Contd.. IS(n) = O(n^2) MS(n) = O(nlog n) MS(n) is faster than IS(n). Suppose we run IS on a fast machine and MS on a slow machine and measure the time (since they were developed by two different people living in different part of the globe), we may get less time for IS and more for MS…wrong analysis Solution: count the number of steps on a generic computational model

Computational Model: Analysis of Algorithms Analysis is performed with respect to a computational model We will usually use a generic uniprocessor random-access machine (RAM) All memory equally expensive to access No concurrent operations All reasonable instructions take unit time Except, of course, function calls Constant word size Unless we are explicitly manipulating bits

Running Time Number of primitive steps that are executed Except for time of executing a function call, in this model most statements roughly require the same amount of time y = m * x + b c = 5 / 9 * (t - 32 ) z = f(x) + g(y) We can be more exact if need be

But why ‘c’ and ‘m’? Because We compare two algorithms on the basis of their number of steps and the actual time taken by an algorithm is ‘c’ times the number of steps.

Why ‘m’? We need efficient algorithms and computational tools to solve problems on big data. For example, it is not very difficult to sort a pack of 52 cards manually. However, to sort all the books in a library on their accession number might be tedious if done manually. So we want to compare algorithms for large input.

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } }

Assignment 4 Show that Insertion Sort takes O(n^2) steps by counting each and every step. Is it O(n)? Is it O(n^3)?

Lower Bound Notation We say InsertionSort’s run time is (n) Proof: Suppose run time is an + b Assume a and b are positive (what if b is negative?) an  an + b

Up Next Solving recurrences Substitution method Master theorem