1. Instructor Neelima Gupta ngupta@cs.du.ac.in
MS 101: Algorithms
Instructor
Neelima Gupta

2. Mathematical Induction: Review Growth Functions
Table Of Contents
Mathematical Induction: Review
Growth Functions

3. Review: Mathematical Induction
Suppose
S(k) is true for fixed constant k
Often k = 0
S(n)  S(n+1) for all n >= k
Then S(n) is true for all n >= k

4. Proof By Mathematical Induction
Claim:S(n) is true for all n >= k
Basis:
Show formula is true when n = k
Inductive hypothesis:
Assume formula is true for an arbitrary n
Step:
Show that formula is then true for n+1

5. Strong Induction Strong induction also holds Another variation:
Basis: show S(0)
Hypothesis: assume S(k) holds for arbitrary k <= n
Step: Show S(n+1) follows
Another variation:
Basis: show S(0), S(1)
Hypothesis: assume S(n) and S(n+1) are true
Step: show S(n+2) follows

6. Lets do it Prove 1 + 2 + 3 + … + n = n(n+1) / 2
Prove a0 + a1 + … + an = (an+1 - 1)/(a - 1) for all a  1

7. Growth Functions Big O Notation In general a function Formally
f(n) is O(g(n)) if there exist positive constants c and n0 such that f(n)  c  g(n) for all n  n0
Formally
O(g(n)) = { f(n):  positive constants c and n0 such that f(n)  c  g(n)  n  n0
Intuitively, it means f(n) grows no faster than g(n).
Examples:
n^2, n^2 – n
n^3, n^3 – n^2 – n

8. f(n) = n2 g(n) = n2 – n Is f(n) = O(g(n))?
Sol: g(n) = n2 – n
= n2/2 + n2/2 - n
≥ n2/2 for n ≥ 2
= ½ f(n)
f(n) ≤ 2g(n) for n ≥ 2
Hence, f(n) = O(g(n))

9. f(n) = n3 g(n) = n3 - n2 – n Is f(n) = O(g(n))?
Sol: g(n) = n3 - n2 – n
= n3/2 + n3/2 - n2 - n
≥ n3/2 + n3/2 - n3/4 - n3/4 for n ≥ 4
= ½ f(n)
f(n) ≤ 2g(n) for n ≥ 2
Hence, f(n) = O(g(n))

10. f(n) = n3 g(n) = n3 - n2 – n Is g(n) = O(f(n))?
Sol: Clearly, g(n) = n3 - n2 – n
≤3 n3 for all n ≥ 1
= 3 f(n)
g(n) ≤ 3f(n) V n ≥ 1
Hence, g(n) = O(f(n)).

11. Omega Notation In general a function
f(n) is (g(n)) if  positive constants c and n0 such that 0  cg(n)  f(n)  n  n0
Intuitively, it means f(n) grows at least as fast as g(n).
Examples:
n^2, n^2 + n
n^3, n^3 + n^2 – n

12. Ques: f(n) = n2, g(n) = n2 + n
Is f(n) = Ω(g(n))?
Sol: n2 = ½ (n2 + n2 )
≥ ½ (n2 + n )
f(n) ≥ c g(n)
for c = ½ m =1

13. f(n) = n3 g(n) = n3 + 4n2 - 5n Is g(n) = Ω(f(n))?
Sol: g(n) = n3 + 4n2 - 5n
≥ n3 /2 + n3 /2 - 4 n2 - 5n2 for all n ≥ 1
= n3 /2 + n2 (n/2 - 9)
≥ n3 /2 for n ≥ 18
= 1/2 f(n)
Hence, g(n) = Ω (f(n)).

14. Theta Notation
A function f(n) is (g(n)) if  positive constants c1, c2, and n0 such that c1 g(n)  f(n)  c2 g(n)  n  n0

15. Relations Between Q, W, O
Theorem : For any two functions g(n) and f(n), f(n) = (g(n)) iff
f(n) = O(g(n)) and f(n) = (g(n)).
Proof:
Simple. Do it.

16. Assignment No 1 Q1 Already proved
a0 + a1 + … + an = (an+1 - 1)/(a - 1) for all a  1
What is the sum for a = 2/3 as n  infinity? Is it O(1)? Is it big or small?
For a = 2, is the sum = O(2^n)? Is it big or small? Q2
Show that a polynomial of degree k = theta(n^k).

17. Other Asymptotic Notations
A function f(n) is o(g(n)) if for every positive constant c, there exists a constant n0 > 0 such that f(n) < c g(n)  n  n0
A function f(n) is (g(n)) if for every positive constant c, there exists a constant n0 > 0 such that c g(n) < f(n)  n  n0
Intuitively,
() is like >
() is like 
() is like =
o() is like <
O() is like 

18. Arrange some functions
f(n) = O(g(n)) => f(n) = o(g(n)) ?
Is the converse true?
Let us arrange the following functions in ascending order (assume log n = o(n) is known)
n, n^2, n^3, sqrt(n), n^epsilon, log n, log^2 n, n log n, n/log n, 2^n, 3^n

19. Relation between n & n2 Solution:
let c > 0 be any constant such that
n < c n2
i.e. 1 < c n
i.e. n > 1 / c
Hence n < c n2 for n > 1 / c
i.e. n = o( n2)
we can also write it as n < n2.

20. Relation between n2 & n3 Solution:
let c> 0 be any constant such that
n2 < c n3
Þ 1 < c n
Þ n > 1 / c
Hence n2 < c n3 " n > 1 / c
i.e. n2 = o( n3)
we can also write it as n2 < n3.
Combining the previous result we can write
n < n2 < n3

21. Relation between n & n1/2 i.e. n2 = o( n3)
Solution: let c> 0 be any constant such that
n1/2 < c n
i.e n < c2 n2
i.e 1 < c2 n
i.e n > 1/ c2
i.e. n1/2 < c n for n > 1/c2
i.e. n2 = o( n3)
we can also write it as n1/2 < n.
Combining the previous result
n1/2 < n < n2 < n3

22. Relation between n & log n
For the time being we can assume the result
log ( n ) = o(n)
Þ log ( n ) < n
we will prove it later.

23. Relation between n1/2 & log n
Assume log n = o(n)
let c > 0 be any constant
for c/2 > 0 there exists m > 0 such that
log n < (c/2) n for n > m
changing variables from n to n1/2 we get
log(n1/2 ) < (c/2) n1/2 for n1/2 > m
½ log( n ) < (c/2) n1/2 for n > m2

24. Contd.. let m2 = k log( n ) < c n1/2 for n > k
Since c > 0 was chosen arbitrarily hence
log n = o( n1/2 ) or
log n < n1/2
Combining the results we get
log n < n1/2 < n < n2 < n3

25. Relation between n2 & nlog n
Since log n = o(n)
for c > 0,  n0 > 0 such that  n  n0, we have
log n < c n
Multiplying by n on both sides we get
n log( n ) < c n2  n  n0
nlog n = o( n2 )
nlog n < n2

26. Relation between n & nlog n
Solution: let c> 0 be any constant such that
n < c n log (n)
Þ 1 < c log( n )
Þ log( n) > 1 / c
Þ n > e1/c
i.e. n < c n log n " n > e1/c
Since c was chosen arbitrarily
\ n = o( n log n ) or n < n log n
Combining the results we can get
log n < n1/2 < n < n logn < n2 < n3

27. Relation between n & n/log n
We know that n = o(nlogn)
for c > 0,  n0 > 0 such that  n  n0, we have
n < c n log n
dividing both sides by log n we get
n/ log( n) < c n  n  n0
Þ n / logn = o(n)
i.e. n / logn < n

28. Assignment No 2
Show that log^M n = o(n^epsilon) for all constants M>0 and epsilon > 0. Assume that log n = o(n). Also prove the following
Corollary: log n = o(n/log n)
Show that n^epsilon = o(n/logn ) for every 0 < epsilon < 1 .

29. Hence we have,
log n < n/log n < n1/2 < n < n logn < n2 < n3

30. Assignment No 3 Show that Show that log n = o(n).
lim f(n)/g(n) = 0 => f(n) = o(g(n)).
n → ∞
lim f(n)/g(n) = c => f(n) = θ(g(n)).
n → ∞, where c is a positive constant.
Show that log n = o(n).
Show that n^k = o(2^n) for every positive constant k.

31. Show by definition of ‘small o’ that
a^n = o(b^n) whenever a < b , a and b are positive constants.
Hence we have,
log n < n/log n < n1/2 < n < n logn < n2 < n3 <2n < 3n

32. Why the constants ‘c’ and ‘m’?
Suppose we have two algorithms to solve the problem say sorting: Insertion Sort and Merge sort for eg.
Why should we have more than one algorithm to solve the same problem?
Ans: efficiency.
What’s the measure of efficiency?
Ans: System resources for example ‘time’.
How do we measure time?

33. Contd.. IS(n) = O(n^2) MS(n) = O(nlog n) MS(n) is faster than IS(n).
Suppose we run IS on a fast machine and MS on a slow machine and measure the time (since they were developed by two different people living in different part of the globe), we may get less time for IS and more for MS…wrong analysis
Solution: count the number of steps on a generic computational model

34. Computational Model: Analysis of Algorithms
Analysis is performed with respect to a computational model
We will usually use a generic uniprocessor random-access machine (RAM)
All memory equally expensive to access
No concurrent operations
All reasonable instructions take unit time
Except, of course, function calls
Constant word size
Unless we are explicitly manipulating bits

35. Running Time Number of primitive steps that are executed
Except for time of executing a function call, in this model most statements roughly require the same amount of time
y = m * x + b
c = 5 / 9 * (t - 32 )
z = f(x) + g(y)
We can be more exact if need be

36. But why ‘c’ and ‘m’? Because
We compare two algorithms on the basis of their number of steps and
the actual time taken by an algorithm is ‘c’ times the number of steps.

37. Why ‘m’?
We need efficient algorithms and computational tools to solve problems on big data. For example, it is not very difficult to sort a pack of 52 cards manually. However, to sort all the books in a library on their accession number might be tedious if done manually.
So we want to compare algorithms for large input.

38. An Example: Insertion Sort
InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } }

39. Assignment 4
Show that Insertion Sort takes O(n^2) steps by counting each and every step.
Is it O(n)?
Is it O(n^3)?

40. Lower Bound Notation We say InsertionSort’s run time is (n) Proof:
Suppose run time is an + b
Assume a and b are positive (what if b is negative?)
an  an + b

41. Up Next
Solving recurrences
Substitution method
Master theorem