Algorithm Course Dr. Aref Rashad Algorithms Lecture 2 Asymptotic Analysis February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Objectives Explaining the basic idea of asymptotic analysis including: The “Big-Oh” Notation Upper and Lower bounds Omega and Theta Notation Calculating Running Time Dominance Concept Illustrating Examples for application of asymtotic analysis to some algorithms February 2013 Algorithms Course ..... Dr. Aref Rashad

“Asymptotic Analysis” No. of Steps Upper bound n large Algorithm 1 g(n) = 3n f(n) = 2n+3 O(n) Algorithm 2 g(n) =5n f(n) = 4n+8 O(n2) Algorithm 3 g(n) =n2 f(n) = n2+8 O(n3) O(log n) Algorithm .. “Asymptotic Analysis” February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Algorithm Course February 2013 Complexity Analysis The “Big-Oh” Notation Big-oh notation indicates an upper bound. How bad things can get – perhaps things are not nearly bad Lowest possible upper bound Asymptotic analysis is a useful tool to help to structure our thinking when n gets large enough Example: linear search: n2 is an upper bound, but n is the tightest upper bound. It provides more information in this example to say O(n2) than O(n3). February 2013 Algorithms Course ..... Dr. Aref Rashad

Calculating Running Time T(n) Use the source/pseudo code Ignore constants Ignore lower order terms Explicitly assume either The average case (harder to do) The worst case (easier) Most analysis uses the worst case

2-Nested Loops  Quadratic Linear-time Loop for x = 1 to n constant-time operation Note: Constant-time mean independent of the input size. 2-Nested Loops  Quadratic for x = 0 to n-1 for y = 0 to n-1 constant-time operation 3-Nested Loops  Cubic for x = 0 to n-1 for y = 0 to n-1 for z = 0 to n-1 constant-time operation f(n) = n3 …….. The number of nested loops determines the exponent February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Non-trivial loops for x = 1 to n for y = 1 to x constant-time operation for z = 1 to n for y = 1 to z for x = 1 to y constant-op February 2013 Algorithms Course ..... Dr. Aref Rashad

This code operates in O(log(n)) time. i=8, print 8, i=8/2=4….…………..1 i=4, print 4, i=4/2=2..…………….2 i=2, print 2, i=2/2=1………………3 i = N; while i >= 1 { print i , i = i / 2 } How many times can we divide by N by 2 until we get below 1? If we approached this in reverse, we could say: how many times can we multiply 1 by 2 until we get to N? This would be the value x, where 2^x = n. This for loop, therefore, iterates x times. Now we just need to solve for x:    2^x = n    log(2^x) = log(n)    x log(2) = log(n)    x = log(n) / log(2) This code operates in O(log(n)) time. February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Program Segments f(n) = n + n2 + n + Log n = n2 + 2n + Log n for x = 0 to n-1 constant-time op for y = 0 to n-1 for z = 0 to n-1 for w = 0 to n-1 i = N; while i >= 1 { print i , i = i / 2 } February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad The “Big-Oh” Notation EG: 3x 3 + 5x 2 – 9 = O (x 3) Doesn’t mean “3x 3 + 5x 2 – 9 equals the function O (x 3)” Which actually means “3x 3+5x 2 –9 is dominated by x 3” Read as: “3x 3+5x 2 –9 is big-Oh of x 3” In fact, 3x3+5x2 –9 is smaller than 5x3 for large enough values of x February 2013 Algorithms Course ..... Dr. Aref Rashad

“Asymptotic Analysis” No. of Steps Upper bound n large Algorithm 1 g(n) = 3n f(n) = 2n+3 O(n) Algorithm 2 g(n) =5n f(n) = 4n+8 O(n2) Algorithm 3 g(n) =n2 f(n) = n2+8 O(n3) O(log n) Algorithm .. “Asymptotic Analysis” February 2013 Algorithms Course ..... Dr. Aref Rashad

Big-Oh Rules If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e., Drop lower-order terms Drop constant factors 3x 3 + 5x 2 – 9 = O (x 3) Use the smallest possible class of functions Say “2n is O(n)” instead of “2n is O(n2)” Use the simplest expression of the class Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)” =

Algorithms Course ..... Dr. Aref Rashad Program Segments f(n) = n + n2 + n + Log n = n2 + 2n + Log n = n2 for x = 0 to n-1 constant-time op for y = 0 to n-1 for z = 0 to n-1 for w = 0 to n-1 i = N; while i >= 1 { print i , i = i / 2 } February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Which of the below expressions are equivalent to O(n3)? O(3 n3) O(n(n2 + 3)) O(n3 - 2) O(n3 + n lg n) O(n3 – n2 + n) O((n2 + 3)(n+1)) All of them! February 2013 Algorithms Course ..... Dr. Aref Rashad

O-notation (upper bounds): So g(n) is an asymptotic upper-bound for f(n) as n increases (g(n) bounds f(n) from above) February 2013 Algorithms Course ..... Dr. Aref Rashad

f(n) is O(g(n)) if f grows at most as fast as g. cg(n) is an approximation to f(n), bounding from above February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Algorithm Course February 2013 Example 1: If f(n) = 3n2 then f(n) is in O(n2). Example 2: f(n) = c1n2 + c2n in average case. c1n2 + c2n <= c1n2 + c2n2 <= (c1 + c2)n2 for all n > 1. f(n) <= cn2 for c = c1 + c2 and n0 = 1. Therefore, f(n) is in O(n2) by the definition. Example 3: f(n) = c. We say this is in O(1). February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad By definition, we need to find: a real constant c > 0 an integer constant n0 >= 1 Such that 7n - 2 <= c n, for every integer n >= 0 Possible choice is: c = 7 n = 1 7n - 2 <= 7n, for n >= 1 February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Example f(n) = 10 n + 5 g(n( = n O(n) To show f(n) is O(g) We must show constants c and k such that f(n( <= cg(n) for all n >=k or 10 n+5 <= c n for all n >= k We are allowed to choose c and k to be integers we want as long as they are positive. They can be as big as we want, but they can't be functions of n. Try c =15 Then we need to show: 10 n + 5 <= 15 n Solving for n we get: 5 <= 5 n or 1 <= n. So 10 n+5 <= cn for all n >= 1, c =15, k=1 Therefore we have shown f(n) is O)n) February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Example f(n) = 3n2 + 4n + 1 Show f(n) is O(n2) Using the facts given in class, we know that: 4n <= 4n2 for all n >= 1 and 1 <= n2 for all n >= 1 So 3n2 + 4n + 1<= 3n2 + 4n2 + n2 for all n >= 1 <= 8n2 for all n >= 1 So we have shown f(n) <= 8n2 for all n >= 1 so f(n) is O(n2) , (c = 8, k = 1) February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Definition Let f and g be real-valued functions. We say that f(x) is O(g(x)) if there are constants C and k such that |f(x)| ≤ C|g(x)| for all x > k. Example Show that f(x) = 4x2 − 5x + 3 is O(x2 ). |f(x)| = |4x2 − 5x + 3| ≤ |4x2 | + | − 5x| + |3| ≤ 4x2 + 5x + 3, for all x > 0 ≤ 4x2 + 5x2 + 3x2 , for all x > 1 ≤ 12x2 , for all x > 1 We conclude that f(x) is O(x2). Observe that C = 12 and k = 1 from the definition of big-O. February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Example Show that f(x) = (x + 5) log2 (3x2 + 7) is O(x log2 x). |f(x)| = |(x + 5) log2 (3x2 + 7)| = (x + 5) log2 (3x2 + 7) , for all x > −5 ≤ (x + 5x) log2 (3x2 + 7x2 ) , for all x > 1 ≤ 6x log2 (10x2 ) , for all x > 1 ≤ 6x log2 (x3 ) , for all x > 10 ≤ 18x log2 x , for all x > 10 We conclude that f(x) is O(x log2 x) for C = 18 and k = 10 February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Example February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Meaning: For all data sets big enough (i.e., n > n0), the algorithm always executes in more than cg(n) steps. February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Algorithm Course February 2013 f(n) = c1n2 + c2n. c1n2 + c2n >= c1n2 for all n > 1. f(n) >= cn2 for c = c1 and n0 = 1. Therefore, f(n) is in (n2) by the definition. We want the greatest lower bound. February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad When big-Oh and  meet, we indicate this by using  (big-Theta) notation. Definition: An algorithm is said to be (h(n)) if it is in O(h(n)) and it is in (h(n)). February 2013 Algorithms Course ..... Dr. Aref Rashad

Relations Between Q, O, W

Intuition for Asymptotic Notation Big-Oh f(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n) big-Omega f(n) is (g(n)) if f(n) is asymptotically greater than or equal to g(n) big-Theta f(n) is (g(n)) if f(n) is asymptotically equal to g(n)

Algorithms Course ..... Dr. Aref Rashad February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad The “Big-Oh” Notation Constant Time: O(1) O(1) actually means that an algorithm takes constant time to run; in other words, performance isn’t affected by the size of the problem. Linear Time: O(N) An algorithm runs in O(N) if the number of operations required to perform a function is directly proportional to the number of items being processed Example: waiting line at a supermarket February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Quadratic Time: O(N2) An algorithm runs in O(N2) if the number of operations required to perform a function is directly proportional to the quadratic number of items being processed Example: Group shakehand February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Logarithmic Time: O(log N) and O(N log N) The running time of a logarithmic algorithm increases with the log of the problem size. When the size of the input data set increases by a factor of a million, the run time will only increase by some factor of log(1,000,000) = 6. Factorial Time: O(N!) It is far worse than even O(N2) and O(N3) It’s fairly unusual to encounter functions with this kind of behavior February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad February 2013 Algorithms Course ..... Dr. Aref Rashad

Practical Considerations No such big difference in running time between Θ1(n) and Θ2(nlogn). Θ1(10,000) = 10,000 Θ2(10,000) =10,000 log1010,000 = 40,000 There is an enormous difference between Θ1(n2) and Θ2(nlogn). Θ1(10,000) = 100,000,000 Θ2(10,000) = 10,000 log10 10,000 = 40,000 February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Remarks Most statements in a program do not have much effect on the running time of that program; There is little point to cutting in half the running time of a subroutine that accounts for only 1% of the total. Focus your attention on the parts of the program that have the most impact The greatest time and space improvements come from a better data structure or algorithm “FIRST TUNE THE ALGORITHM, THEN TUNE THE CODE” February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Remarks When tuning code, it is important to gather good timing statistics; Be careful not to use tricks that make the program unreadable; Make use of compiler optimizations; Check that your optimizations really improve the program. February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Remarks An algorithm with time equation T(n) = 2n2 does not receive nearly as great an improvement from the faster machine as an algorithm with linear growth rate. Instead of an improvement by a factor of ten, the improvement is only the square root of 10 (≈3.16) Instead of buying a faster computer, consider what happens if you replace an algorithm with quadratic running time with a new algorithm with n logn running time February 2013 Algorithms Course ..... Dr. Aref Rashad

Basic Relaions and Examples February 2013 Algorithms Course ..... Dr. Aref Rashad

Logarithms x = logba is the exponent for a = bx. lg2a = (lg a)2 Natural log: ln a = logea Binary log: lg a = log2a lg2a = (lg a)2 lg lg a = lg (lg a)

Logarithms and exponentials If the base of a logarithm is changed from one constant to another, the value is altered by a constant factor. Ex: log10 n * log210 = log2 n. Base of logarithm is not an issue in asymptotic notation. Exponentials with different bases differ by a exponential factor (not a constant factor). Ex: 2n = (2/3)n*3n.

Examples A B 5n2 + 100n 3n2 + 2 A  (B) Express functions in A in asymptotic notation using functions in B. A B 5n2 + 100n 3n2 + 2 A  (B) A  (n2), n2  (B)  A  (B) log3(n2) log2(n3) A  (B) logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3) nlg4 3lg n A  (B) alog b = blog a; B =3lg n=nlg 3; A/B =nlg(4/3)   as n  lg2n n1/2 A  o (B) lim ( lga n / nb ) = 0 (here a = 2 and b = 1/2)  A  o (B) n

Math Basics Constant Series: For integers a and b, a  b, Linear Series (Arithmetic Series): For n  0, Quadratic Series: For n  0,

Harmonic Series: nth harmonic number, nI+, Math Basics Linear-Geometric Series: For n  0, real c  1, Harmonic Series: nth harmonic number, nI+,

Algorithms Course ..... Dr. Aref Rashad 2^7 = 128 2^10 = 1024 2^20= 1048576 February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Faster Computer or Algorithm? What happens when we buy a computer 10 times faster T(n) n n’ Change n’/n 10n 1,000 10,000 n’ = 10n 10 20n 500 5,000 5n log n 250 1,842 10 n < n’ < 10n 7.37 2n2 70 223 n’ = 10n 3.16 2n 13 16 n’ = n + 3 ----- February 2013 Algorithms Course ..... Dr. Aref Rashad

Algorithms Course ..... Dr. Aref Rashad Let Old Speed = 10000 Hz New Speed = 100000 Hz 10 n Old……………. 10n / S1 =1 n1 =1000 S1= 10000 S2= 100000 New ……………. 10n / 10^5 = 1 n2=10000 2n^2 Old ……………… 2n^2 / S1 = 1 n1=70 S1= 2* 70^2= 10000, S2= 100000 New………………2n^2 / 10^5 = 1 n^2= 10^5 / 2 = 10^4 * 5 n2 = 100 * 2.23= 223 n2/n1= 100 sq. 5 / sq. 5000 = 10 sq. 50 = 10/Sq. 10 = Sq. 10 10^4 * sq. 5 5 n log n Old…………………5n log n / S1 =1 n1=250 S1`= 250 log 250 = 3000 S2= 30000 New………………... 5n log n / 30000 =1 n log n = 6000 n2=1842 February 2013 Algorithms Course ..... Dr. Aref Rashad