1. Algorithm Course Dr. Aref Rashad
Algorithms Lecture 2
Asymptotic Analysis
February 2013
Algorithms Course Dr. Aref Rashad

2. Algorithms Course ..... Dr. Aref Rashad
Objectives
Explaining the basic idea of asymptotic analysis including:
The “Big-Oh” Notation
Upper and Lower bounds
Omega and Theta Notation
Calculating Running Time
Dominance Concept
Illustrating Examples for application of asymtotic analysis to some algorithms
February 2013
Algorithms Course Dr. Aref Rashad

3. “Asymptotic Analysis”
No. of Steps
Upper bound
n large
Algorithm 1
g(n) = 3n
f(n) = 2n+3
O(n)
Algorithm 2
g(n) =5n
f(n) = 4n+8
O(n2)
Algorithm 3
g(n) =n2
f(n) = n2+8
O(n3)
O(log n)
Algorithm ..
“Asymptotic Analysis”
February 2013
Algorithms Course Dr. Aref Rashad

4. Algorithms Course ..... Dr. Aref Rashad
Algorithm Course
February 2013
Complexity Analysis
The “Big-Oh” Notation
Big-oh notation indicates an upper bound.
How bad things can get – perhaps things are not nearly bad
Lowest possible upper bound
Asymptotic analysis is a useful tool to help to structure our thinking when n gets large enough
Example: linear search: n2 is an upper bound, but n is the tightest upper bound.
It provides more information in this example to say O(n2) than O(n3).
February 2013
Algorithms Course Dr. Aref Rashad

5. Calculating Running Time T(n)
Use the source/pseudo code
Ignore constants
Ignore lower order terms
Explicitly assume either
The average case (harder to do)
The worst case (easier)
Most analysis uses the worst case

6. 2-Nested Loops  Quadratic
Linear-time Loop
for x = 1 to n
constant-time operation
Note: Constant-time mean independent of the input size.
2-Nested Loops  Quadratic
for x = 0 to n-1
for y = 0 to n-1
constant-time operation
3-Nested Loops  Cubic
for x = 0 to n-1
for y = 0 to n-1
for z = 0 to n-1
constant-time operation
f(n) = n3 …….. The number of nested loops determines the exponent
February 2013
Algorithms Course Dr. Aref Rashad

7. Algorithms Course ..... Dr. Aref Rashad
Non-trivial loops
for x = 1 to n
for y = 1 to x
constant-time operation
for z = 1 to n
for y = 1 to z
for x = 1 to y
constant-op
February 2013
Algorithms Course Dr. Aref Rashad

8. This code operates in O(log(n)) time.
i=8, print 8, i=8/2=4….…………..1
i=4, print 4, i=4/2=2..…………….2
i=2, print 2, i=2/2=1………………3
i = N;
while i >= 1
{ print i , i = i / 2 }
How many times can we divide by N by 2 until we get below 1? If we approached this in reverse, we could say: how many times can we multiply 1 by 2 until we get to N? This would be the value x, where 2^x = n. This for loop, therefore, iterates x times.
Now we just need to solve for x:    2^x = n    log(2^x) = log(n)    x log(2) = log(n)    x = log(n) / log(2)
This code operates in O(log(n)) time.
February 2013
Algorithms Course Dr. Aref Rashad

9. Algorithms Course ..... Dr. Aref Rashad
Program Segments
f(n) = n + n2 + n + Log n
= n2 + 2n + Log n
for x = 0 to n-1
constant-time op
for y = 0 to n-1
for z = 0 to n-1
for w = 0 to n-1
i = N;
while i >= 1
{ print i , i = i / 2 }
February 2013
Algorithms Course Dr. Aref Rashad

10. Algorithms Course ..... Dr. Aref Rashad
The “Big-Oh” Notation
EG: 3x 3 + 5x 2 – 9 = O (x 3) Doesn’t mean “3x 3 + 5x 2 – 9 equals the function O (x 3)” Which actually means “3x 3+5x 2 –9 is dominated by x 3” Read as: “3x 3+5x 2 –9 is big-Oh of x 3”
In fact, 3x3+5x2 –9 is smaller than 5x3 for large enough values of x
February 2013
Algorithms Course Dr. Aref Rashad

11. “Asymptotic Analysis”
No. of Steps
Upper bound
n large
Algorithm 1
g(n) = 3n
f(n) = 2n+3
O(n)
Algorithm 2
g(n) =5n
f(n) = 4n+8
O(n2)
Algorithm 3
g(n) =n2
f(n) = n2+8
O(n3)
O(log n)
Algorithm ..
“Asymptotic Analysis”
February 2013
Algorithms Course Dr. Aref Rashad

12. Big-Oh Rules
If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e.,
Drop lower-order terms
Drop constant factors
3x 3 + 5x 2 – 9 = O (x 3)
Use the smallest possible class of functions
Say “2n is O(n)” instead of “2n is O(n2)”
Use the simplest expression of the class
Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)” =

13. Algorithms Course ..... Dr. Aref Rashad
Program Segments
f(n) = n + n2 + n + Log n
= n2 + 2n + Log n
= n2
for x = 0 to n-1
constant-time op
for y = 0 to n-1
for z = 0 to n-1
for w = 0 to n-1
i = N;
while i >= 1
{ print i , i = i / 2 }
February 2013
Algorithms Course Dr. Aref Rashad

14. Algorithms Course ..... Dr. Aref Rashad
Which of the below expressions are equivalent to O(n3)?
O(3 n3)
O(n(n2 + 3))
O(n3 - 2)
O(n3 + n lg n)
O(n3 – n2 + n)
O((n2 + 3)(n+1))
All of them!
February 2013
Algorithms Course Dr. Aref Rashad

15. O-notation (upper bounds):
So g(n) is an asymptotic upper-bound for f(n) as n increases
(g(n) bounds f(n) from above)
February 2013
Algorithms Course Dr. Aref Rashad

16. f(n) is O(g(n)) if f grows at most as fast as g.
cg(n) is an approximation to f(n), bounding from above
February 2013
Algorithms Course Dr. Aref Rashad

17. Algorithms Course ..... Dr. Aref Rashad
Algorithm Course
February 2013
Example 1: If f(n) = 3n2 then f(n) is in O(n2). Example 2: f(n) = c1n2 + c2n in average case. c1n2 + c2n <= c1n2 + c2n2 <= (c1 + c2)n2 for all n > 1. f(n) <= cn2 for c = c1 + c2 and n0 = 1. Therefore, f(n) is in O(n2) by the definition. Example 3: f(n) = c. We say this is in O(1).
February 2013
Algorithms Course Dr. Aref Rashad

18. Algorithms Course ..... Dr. Aref Rashad
By definition, we need to find:
a real constant c > 0
an integer constant n0 >= 1
Such that 7n - 2 <= c n,
for every integer n >= 0
Possible choice is:
c = 7
n = 1
7n - 2 <= 7n, for n >= 1
February 2013
Algorithms Course Dr. Aref Rashad

19. Algorithms Course ..... Dr. Aref Rashad
Example
f(n) = 10 n g(n( = n O(n)
To show f(n) is O(g)
We must show constants c and k such that
f(n( <= cg(n) for all n >=k or n+5 <= c n for all n >= k
We are allowed to choose c and k to be integers we want as long as they are positive. They can be as big as we want, but they can't be functions of n.
Try c =15
Then we need to show: 10 n + 5 <= 15 n
Solving for n we get: <= 5 n or 1 <= n.
So 10 n+5 <= cn for all n >= 1, c =15, k=1
Therefore we have shown f(n) is O)n)
February 2013
Algorithms Course Dr. Aref Rashad

20. Algorithms Course ..... Dr. Aref Rashad
Example
f(n) = 3n2 + 4n + 1
Show f(n) is O(n2) Using the facts given in class,
we know that:
4n <= 4n for all n >= 1 and
1 <= n for all n >= 1
So 3n2 + 4n + 1<= 3n2 + 4n2 + n for all n >= 1
<= 8n for all n >= 1
So we have shown f(n) <= 8n2 for all n >= 1
so f(n) is O(n2) , (c = 8, k = 1)
February 2013
Algorithms Course Dr. Aref Rashad

21. Algorithms Course ..... Dr. Aref Rashad
Definition
Let f and g be real-valued functions.
We say that f(x) is O(g(x)) if there are constants C and k such that |f(x)| ≤ C|g(x)| for all x > k.
Example
Show that f(x) = 4x2 − 5x + 3 is O(x2 ).
|f(x)| = |4x2 − 5x + 3| ≤ |4x2 | + | − 5x| + |3|
≤ 4x2 + 5x + 3, for all x > 0
≤ 4x2 + 5x2 + 3x2 , for all x > 1
≤ 12x2 , for all x > 1
We conclude that f(x) is O(x2). Observe that C = 12 and k = 1 from the definition of big-O.
February 2013
Algorithms Course Dr. Aref Rashad

22. Algorithms Course ..... Dr. Aref Rashad
Example
Show that f(x) = (x + 5) log2 (3x2 + 7) is O(x log2 x).
|f(x)| = |(x + 5) log2 (3x2 + 7)|
= (x + 5) log2 (3x2 + 7) , for all x > −5
≤ (x + 5x) log2 (3x2 + 7x2 ) , for all x > 1
≤ 6x log2 (10x2 ) , for all x > 1
≤ 6x log2 (x3 ) , for all x > 10
≤ 18x log2 x , for all x > 10
We conclude that f(x) is O(x log2 x) for C = 18 and k = 10
February 2013
Algorithms Course Dr. Aref Rashad

23. Algorithms Course ..... Dr. Aref Rashad
Example
February 2013
Algorithms Course Dr. Aref Rashad

24. Algorithms Course ..... Dr. Aref Rashad
February 2013
Algorithms Course Dr. Aref Rashad

25. Algorithms Course ..... Dr. Aref Rashad
Meaning: For all data sets big enough (i.e., n > n0),
the algorithm always executes in more than cg(n) steps.
February 2013
Algorithms Course Dr. Aref Rashad

26. Algorithms Course ..... Dr. Aref Rashad
Algorithm Course
February 2013
f(n) = c1n2 + c2n. c1n2 + c2n >= c1n2 for all n > 1. f(n) >= cn2 for c = c1 and n0 = 1. Therefore, f(n) is in (n2) by the definition. We want the greatest lower bound.
February 2013
Algorithms Course Dr. Aref Rashad

27. Algorithms Course ..... Dr. Aref Rashad
When big-Oh and  meet, we indicate this by using
 (big-Theta) notation.
Definition: An algorithm is said to be (h(n))
if it is in O(h(n)) and it is in (h(n)).
February 2013
Algorithms Course Dr. Aref Rashad

28. Relations Between Q, O, W

29. Intuition for Asymptotic Notation
Big-Oh
f(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n)
big-Omega
f(n) is (g(n)) if f(n) is asymptotically greater than or equal to g(n)
big-Theta
f(n) is (g(n)) if f(n) is asymptotically equal to g(n)

30. Algorithms Course ..... Dr. Aref Rashad
February 2013
Algorithms Course Dr. Aref Rashad

31. Algorithms Course ..... Dr. Aref Rashad
February 2013
Algorithms Course Dr. Aref Rashad

32. Algorithms Course ..... Dr. Aref Rashad
The “Big-Oh” Notation
Constant Time: O(1)
O(1) actually means that an algorithm takes constant time to run; in other words, performance isn’t affected by the size of the problem.
Linear Time: O(N)
An algorithm runs in O(N) if the number of operations required to perform a function is directly proportional to the number of items being processed
Example: waiting line at a supermarket
February 2013
Algorithms Course Dr. Aref Rashad

33. Algorithms Course ..... Dr. Aref Rashad
Quadratic Time: O(N2)
An algorithm runs in O(N2) if the number of operations required to perform a function is directly proportional to the quadratic number of items being processed
Example: Group shakehand
February 2013
Algorithms Course Dr. Aref Rashad

34. Algorithms Course ..... Dr. Aref Rashad
Logarithmic Time: O(log N) and O(N log N)
The running time of a logarithmic algorithm increases with the log of the problem size.
When the size of the input data set increases by a factor of a million, the run time will only increase by some factor of log(1,000,000) = 6.
Factorial Time: O(N!)
It is far worse than even O(N2) and O(N3)
It’s fairly unusual to encounter functions with this kind of behavior
February 2013
Algorithms Course Dr. Aref Rashad

35. Algorithms Course ..... Dr. Aref Rashad
February 2013
Algorithms Course Dr. Aref Rashad

36. Algorithms Course ..... Dr. Aref Rashad
February 2013
Algorithms Course Dr. Aref Rashad

37. Practical Considerations
No such big difference in running time between Θ1(n) and Θ2(nlogn).
Θ1(10,000) = 10,000
Θ2(10,000) =10,000 log1010,000 = 40,000
There is an enormous difference between Θ1(n2) and Θ2(nlogn).
Θ1(10,000) = 100,000,000
Θ2(10,000) = 10,000 log10 10,000 = 40,000
February 2013
Algorithms Course Dr. Aref Rashad

38. Algorithms Course ..... Dr. Aref Rashad
Remarks
Most statements in a program do not have much effect on the running time of that program;
There is little point to cutting in half the running time of a subroutine that accounts for only 1% of the total.
Focus your attention on the parts of the program that have the most impact
The greatest time and space improvements come from a better data structure or algorithm
“FIRST TUNE THE ALGORITHM, THEN TUNE THE CODE”
February 2013
Algorithms Course Dr. Aref Rashad

39. Algorithms Course ..... Dr. Aref Rashad
Remarks
When tuning code, it is important to gather good timing statistics;
Be careful not to use tricks that make the program unreadable;
Make use of compiler optimizations;
Check that your optimizations really improve the program.
February 2013
Algorithms Course Dr. Aref Rashad

40. Algorithms Course ..... Dr. Aref Rashad
Remarks
An algorithm with time equation T(n) = 2n2 does not receive nearly as great an improvement from the faster machine as an algorithm with linear growth rate.
Instead of an improvement by a factor of ten, the improvement is only the square root of 10 (≈3.16)
Instead of buying a faster computer, consider what happens if you replace an algorithm with quadratic running time with a new algorithm with n logn running time
February 2013
Algorithms Course Dr. Aref Rashad

41. Basic Relaions and Examples
February 2013
Algorithms Course Dr. Aref Rashad

42. Logarithms x = logba is the exponent for a = bx. lg2a = (lg a)2
Natural log: ln a = logea
Binary log: lg a = log2a
lg2a = (lg a)2
lg lg a = lg (lg a)

43. Logarithms and exponentials
If the base of a logarithm is changed from one constant to another, the value is altered by a constant factor.
Ex: log10 n * log210 = log2 n.
Base of logarithm is not an issue in asymptotic notation.
Exponentials with different bases differ by a exponential factor (not a constant factor).
Ex: 2n = (2/3)n*3n.

44. Examples A B 5n2 + 100n 3n2 + 2 A  (B)
Express functions in A in asymptotic notation using functions in B.
A B
5n n n2 + 2
A  (B)
A  (n2), n2  (B)  A  (B)
log3(n2) log2(n3)
A  (B)
logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)
nlg lg n
A  (B)
alog b = blog a; B =3lg n=nlg 3; A/B =nlg(4/3)   as n 
lg2n n1/2
A  o (B)
lim ( lga n / nb ) = 0 (here a = 2 and b = 1/2)  A  o (B)
n

45. Math Basics Constant Series: For integers a and b, a  b,
Linear Series (Arithmetic Series): For n  0,
Quadratic Series: For n  0,

46. Harmonic Series: nth harmonic number, nI+,
Math Basics
Linear-Geometric Series: For n  0, real c  1,
Harmonic Series: nth harmonic number, nI+,

47. Algorithms Course ..... Dr. Aref Rashad
2^7 = 128
2^10 = 1024
2^20=
February 2013
Algorithms Course Dr. Aref Rashad

48. Algorithms Course ..... Dr. Aref Rashad
Faster Computer or Algorithm?
What happens when we buy a computer 10 times faster
T(n) n n’
Change
n’/n
10n
1,000
10,000
n’ = 10n 10
20n
500
5,000
5n log n
250
1,842
10 n < n’ < 10n
7.37
2n2 70
223
n’ = 10n
3.16 2n 13 16
n’ = n + 3
-----
February 2013
Algorithms Course Dr. Aref Rashad

49. Algorithms Course ..... Dr. Aref Rashad
Let Old Speed = Hz New Speed = Hz
10 n
Old……………. 10n / S1 = n1 = S1= S2=
New ……………. 10n / 10^5 = n2=10000
2n^2
Old ……………… 2n^2 / S1 = n1=70 S1= 2* 70^2= 10000, S2=
New………………2n^2 / 10^5 = n^2= 10^5 / 2 = 10^4 * 5
n2 = 100 * 2.23= 223
n2/n1= 100 sq. 5 / sq = 10 sq. 50 = 10/Sq. 10 = Sq ^4 * sq. 5
5 n log n
Old…………………5n log n / S1 =1 n1=250 S1`= 250 log 250 = 3000
S2= 30000
New………………... 5n log n / =1 n log n = n2=1842
February 2013
Algorithms Course Dr. Aref Rashad