Analyzing algorithms & Asymptotic Notation BIO/CS 471 – Algorithms for Bioinformatics

Analyzing Algorithms2 Algorithms and Problems Algorithm: a method or a process followed to solve a problem. A recipe. A problem is a mapping of input to output. An algorithm takes the input to a problem (function) and transforms it to the output. A problem can have many algorithms.

Analyzing Algorithms3 Algorithm Properties An algorithm possesses the following properties: It must be correct. It must be composed of a series of concrete steps. There can be no ambiguity as to which step will be performed next. It must be composed of a finite number of steps. It must terminate. A computer program is an instance, or concrete representation, for an algorithm in some programming language.

Analyzing Algorithms4 The RAM model of computing  Linear, random access memory  READ/WRITE = one operation  Simple mathematical operations are also unit operations  Can only read one location at a time, by address  Registers …

Analyzing Algorithms5 How fast is an algorithm?  To compare two sorting algorithms, should we talk about how fast the algorithms can sort 10 numbers, 100 numbers or 1000 numbers?  We need a way to talk about how fast the algorithm grows or scales with the input size. Input size is usually called n An algorithm can take 100n steps, or 2n 2 steps, which one is better?

Analyzing Algorithms6 Introduction to Asymptotic Notation  We want to express the concept of “about”, but in a mathematically rigorous way  Limits are useful in proofs and performance analyses  notation:  (n 2 ) = “this function grows similarly to n 2 ”.  Big-O notation: O (n 2 ) = “this function grows at least as slowly as n 2 ”. Describes an upper bound.

Analyzing Algorithms7 Big-O  What does it mean? If f(n) = O(n 2 ), then:  f(n) can be larger than n 2 sometimes, but…  I can choose some constant c and some value n 0 such that for every value of n larger than n 0 : f(n) < cn 2  That is, for values larger than n 0, f(n) is never more than a constant multiplier greater than n 2  Or, in other words, f(n) does not grow more than a constant factor faster than n 2.

Analyzing Algorithms8 Visualization of O(g(n)) n0n0 cg(n) f(n)f(n)

Analyzing Algorithms9 Big-O

10 More Big-O  Prove that:  Let c = 21 and n 0 = 4  21n 2 > 20n 2 + 2n + 5 for all n > 4 n 2 > 2n + 5 for all n > 4 TRUE

Analyzing Algorithms11 Tight bounds  We generally want the tightest bound we can find.  While it is true that n 2 + 7n is in O(n 3 ), it is more interesting to say that it is in O(n 2 )

Analyzing Algorithms12 Big Omega – Notation  () – A lower bound n 2 =  (n) Let c = 1, n 0 = 2 For all n  2, n 2 > 1  n

Analyzing Algorithms13 Visualization of  (g(n)) n0n0 cg(n) f(n)f(n)

Analyzing Algorithms14  -notation  Big-O is not a tight upper bound. In other words n = O(n 2 )  provides a tight bound  In other words,

Analyzing Algorithms15 Visualization of  (g(n)) n0n0 c2g(n)c2g(n) f(n)f(n) c1g(n)c1g(n)

Analyzing Algorithms16 A Few More Examples  n = O(n 2 ) ≠  (n 2 )  200n 2 = O(n 2 ) =  (n 2 )  n 2.5 ≠ O(n 2 ) ≠  (n 2 )

Analyzing Algorithms17 Some Other Asymptotic Functions  Little o – A non-tight asymptotic upper bound n = o(n 2 ), n = O(n 2 ) 3n 2 ≠ o(n 2 ), 3n 2 = O(n 2 )  () – A lower bound Similar definition to Big-O n 2 =  (n)  () – A non-tight asymptotic lower bound  f(n) =  (n)  f(n) = O(n) and f(n) =  (n)

Analyzing Algorithms18 Visualization of Asymptotic Growth n0n0 O(f(n)) f(n)f(n)  (f(n))  (f(n)) o(f(n))  (f(n))

Analyzing Algorithms19 Analogy to Arithmetic Operators

Analyzing Algorithms20 Example 2  Prove that:  Let c = 21 and n 0 = 10  21n 3 > 20n 3 + 7n for all n > 10 n 3 > 7n + 5 for all n > 10 TRUE, but we also need…  Let c = 20 and n 0 = 1  20n 3 < 20n 3 + 7n for all n  1 TRUE

Analyzing Algorithms21 Example 3  Show that  Let c = 2 and n 0 = 5

Analyzing Algorithms22 Looking at Algorithms  Asymptotic notation gives us a language to talk about the run time of algorithms.  Not for just one case, but how an algorithm performs as the size of the input, n, grows.  Tools: Series sums Recurrence relations

Analyzing Algorithms23 Running Time Examples (1) Example 1: a = b; This assignment takes constant time, so it is  (1). Example 2: sum = 0; for (i=1; i<=n; i++) sum += n;

Analyzing Algorithms24 Running Time Examples (2) Example 2: sum = 0; for (j=1; j<=n; j++) for (i=1; i<=j; i++) sum++; for (k=0; k<n; k++) A[k] = k;

Analyzing Algorithms25 Series Sums  The arithmetic series: … + n =  Linearity:

Analyzing Algorithms26 Series Sums  … + n – 1 =  Example:

Analyzing Algorithms27 More Series  Geometric Series: 1 + x + x 2 + x 3 + … + x n  Example:

Analyzing Algorithms28 Telescoping Series  Consider the series:  Look at the terms:

Analyzing Algorithms29 Telescoping Series  In general:

Analyzing Algorithms30 The Harmonic Series

Analyzing Algorithms31 Running Time Examples (3) Example 3: sum1 = 0; for (i=1; i<=n; i++) for (j=1; j<=n; j++) sum1++; sum2 = 0; for (i=1; i<=n; i++) for (j=1; j<=i; j++) sum2++;

Analyzing Algorithms32 Best, Worst, Average Cases Not all inputs of a given size take the same time to run. Sequential search for K in an array of n integers: Begin at first element in array and look at each element in turn until K is found Best case: Worst case: Average case:

Analyzing Algorithms33 Space Bounds Space bounds can also be analyzed with asymptotic complexity analysis. Time: Algorithm Space Data Structure

Analyzing Algorithms34 Space/Time Tradeoff Principle One can often reduce time if one is willing to sacrifice space, or vice versa. Encoding or packing information Boolean flags Table lookup Factorials Disk-based Space/Time Tradeoff Principle: The smaller you make the disk storage requirements, the faster your program will run.

Analyzing Algorithms35 Faster Computer or Faster Algorithm?  Suppose, for your algorithm, f(n) = 2n 2  In T seconds, you can process k inputs  If you get a computer 64 times faster, how many inputs can you process in T seconds?

Analyzing Algorithms36 Faster Computer or Algorithm? If we have a computer that does 10,000 operations per second, what happens when we buy a computer 10 times faster? T(n)T(n)nn’n’Changen’/n 10n1,00010,000n’ = 10n10 20n5005,000n’ = 10n10 5n log n2501,842  10 n < n’ < 10n n22n n’ =  10n n2n 1316n’ = n

Analyzing Algorithms37 Traveling Salesman Problem  n cities Traveling distance between each pair is given Find the circuit that includes all cities A C D G B E F

Analyzing Algorithms38 Is there a “real difference”?  10^1  10^2  10^3Number of students in the college of engineering  10^4 Number of students enrolled at Wright State University  10^6 Number of people in Dayton  10^8 Number of people in Ohio  10^10 Number of stars in the galaxy  10^20 Total number of all stars in the universe  10^80 Total number of particles in the universe  10^100 << Number of possible solutions to traveling salesman (100)  Traveling salesman (100) is computable but it is NOT tractable.

Analyzing Algorithms39 Growth of Functions

Analyzing Algorithms40 Is there a “real” difference?  Growth of functions

Analyzing Algorithms41 Approaches to Solving Problems  Direct/iterative SelectionSort Can by analyzed using series sums  Divide and Conquer Recursion and Dynamic Programming Cut the problem in half MergeSort

Analyzing Algorithms42 Recursion  Computing factorials sub fact($n) { if ($n <= 1) { return(1); } else { $temp = $fact($n-1); $result = $temp + 1; return($result); } print(fact(4). “\n”); fib(5)

Analyzing Algorithms43 Fibonacci Numbers int fib(int N) { int prev, pprev; if (N == 1) { return 0; } else if (N == 2) { return 1; } else { prev = fib(N-1); pprev = fib(N-2); return prev + pprev; }

Analyzing Algorithms44 MergeSort  Let M n be the time to MergeSort n items M n = 2(M n-1 ) + n