1 Growth of Functions CS 202 Epp, section ??? Aaron Bloomfield

2 How does one measure algorithms We can time how long it takes a computer –What if the computer is doing other things? –And what happens if you get a faster computer? A 3 Ghz Windows machine chip will run an algorithm at a different speed than a 3 Ghz Macintosh So that idea didn’t work out well…

3 How does one measure algorithms We can measure how many machine instructions an algorithm takes –Different CPUs will require different amount of machine instructions for the same algorithm So that idea didn’t work out well…

4 How does one measure algorithms We can loosely define a “step” as a single computer operation –A comparison, an assignment, etc. –Regardless of how many machine instructions it translates into This allows us to put algorithms into broad categories of efficient-ness –An efficient algorithm on a slow computer will always beat an inefficient algorithm on a fast computer

5 Bubble sort running time The bubble step take (n 2 -n)/2 “steps” Let’s say the bubble sort takes the following number of steps on specific CPUs: –Intel Pentium IV CPU: 58*(n 2 -n)/2 –Motorola CPU: 84.4*(n 2 -2n)/2 –Intel Pentium V CPU: 44*(n 2 -n)/2 Notice that each has an n 2 term –As n increases, the other terms will drop out

6 Bubble sort running time This leaves us with: –Intel Pentium IV CPU: 29n 2 –Motorola CPU: 42.2n 2 –Intel Pentium V CPU: 22n 2 As processors change, the constants will always change –The exponent on n will not Thus, we can’t care about the constants

7 An aside: inequalities If you have a inequality you need to show: x < y You can replace the lesser side with something greater: x+1 < y If you can still show this to be true, then the original inequality is true Consider showing that 15 < 20 –You can replace 15 with 16, and then show that 16 < 20. Because 15 < 16, and 16 < 20, then 15 < 20

8 An aside: inequalities If you have a inequality you need to show: x < y You can replace the greater side with something lesser: x < y-1 If you can still show this to be true, then the original inequality is true Consider showing that 15 < 20 –You can replace 20 with 19, and then show that 15 < 19. Because 15 < 19, and 19 < 20, then 15 < 20

9 An aside: inequalities What if you do such a replacement and can’t show anything? –Then you can’t say anything about the original inequality Consider showing that 15 < 20 –You can replace 20 with 10 –But you can’t show that 15 < 10 –So you can’t say anything one way or the other about the original inequality

10 Review of last time Searches –Linear: n steps –Binary: log 2 n steps –Binary search is about as fast as you can get Sorts –Bubble: n 2 steps –Insertion: n 2 steps –There are other, more efficient, sorting techniques In principle, the fastest are heap sort, quick sort, and merge sort These each take take n * log 2 n steps In practice, quick sort is the fastest (although this is not guaranteed!), followed by merge sort

11 Big-Oh notation Let b(x) be the bubble sort algorithm We say b(x) is O(n 2 ) –This is read as “b(x) is big-oh n 2 ” –This means that the input size increases, the running time of the bubble sort will increase proportional to the square of the input size In other words, by some constant times n 2 Let l(x) be the linear (or sequential) search algorithm We say l(x) is O(n) –Meaning the running time of the linear search increases directly proportional to the input size

12 Big-Oh notation Consider: b(x) is O(n 2 ) –That means that b(x)’s running time is less than (or equal to) some constant times n 2 Consider: l(x) is O(n) –That means that l(x)’s running time is less than (or equal to) some constant times n

13 Big-Oh proofs Show that f(x) = x 2 + 2x + 1 is O(x 2 ) –In other words, show that x 2 + 2x + 1 ≤ c*x 2 Where c is some constant For input size greater than some x We know that 2x 2 ≥ 2x whenever x ≥ 1 And we know that x 2 ≥ 1 whenever x ≥ 1 So we replace 2x+1 with 3x 2 –We then end up with x 2 + 3x 2 = 4x 2 –This yields 4x 2 ≤ c*x 2 This, for input sizes 1 or greater, when the constant is 4 or greater, f(x) is O(x 2 ) We could have chosen values for c and x that were different

14 Big-Oh proofs

15 Sample Big-Oh problems Show that f(x) = x is O(x 2 ) –In other words, show that x ≤ c*x 2 We know that x 2 > 1000 whenever x > 31 –Thus, we replace 1000 with x 2 –This yields 2x 2 ≤ c*x 2 Thus, f(x) is O(x 2 ) for all x > 31 when c ≥ 2

16 Sample Big-Oh problems Show that f(x) = 3x+7 is O(x) –In other words, show that 3x+7 ≤ c*x We know that x > 7 whenever x > 7 –Duh! –So we replace 7 with x –This yields 4x ≤ c*x Thus, f(x) is O(x) for all x > 7 when c ≥ 4

17 A variant of the last question Show that f(x) = 3x+7 is O(x 2 ) –In other words, show that 3x+7 ≤ c*x 2 We know that x > 7 whenever x > 7 –Duh! –So we replace 7 with x –This yields 4x < c*x 2 –This will also be true for x > 7 when c ≥ 1 Thus, f(x) is O(x 2 ) for all x > 7 when c ≥ 1

18 What that means If a function is O(x) –Then it is also O(x 2 ) –And it is also O(x 3 ) Meaning a O(x) function will grow at a slower or equal to the rate x, x 2, x 3, etc.

19 Function growth rates For input size n = 1000 O(1)1 O(log n)≈10 O(n)10 3 O(n log n)≈10 4 O(n 2 )10 6 O(n 3 )10 9 O(n 4 )10 12 O(n c )10 3*c c is a consant 2 n ≈ n!≈ n n Many interesting problems fall into these categories

20 Function growth rates Logarithmic scale!

21 Integer factorization Factoring a composite number into it’s component primes is O(2 n ) –Where n is the number of bits in the number This, if we choose 2048 bit numbers (as in RSA keys), it takes steps –That’s about steps!

22 Formal Big-Oh definition Let f and g be functions. We say that f(x) is O(g(x)) if there are constants c and k such that |f(x)| ≤ C |g(x)| whenever x > k

23 Formal Big-Oh definition

24 A note on Big-Oh notation Assume that a function f(x) is O(g(x)) It is sometimes written as f(x) = O(g(x)) –However, this is not a proper equality! –It’s really saying that |f(x)| ≤ C |g(x)| In this class, we will write it as f(x) is O(g(x))

25 NP Completeness A full discussion of NP completeness takes 3 hours for somebody who already has a CS degree –We are going to do the 15 minute version of it today –More will (probably) follow on Friday Any term of the form n c, where c is a constant, is a polynomial –Thus, any function that is O(n c ) is a polynomial-time function –2 n, n!, n n are not polynomial functions

26 Satisfiability Consider a Boolean expression of the form: (x 1   x 2  x 3 )  (x 2   x 3  x 4 )  (  x 1  x 4  x 5 ) –This is a conjunction of disjunctions Is such an equation satisfiable? –In other words, can you assign truth values to all the x i ’s such that the equation is true? –The above problem is easy (only 3 clauses of 3 variables) – set x 1, x 2, and x 4 to true There are other possibilities: set x 1, x 2, and x 5 to true, etc. –But consider an expression with 1000 variables and thousands of clauses

27 Satisfiability If given a solution, it is easy to check if such a solution works –Plug in the values – this can be done quickly, even by hand However, there is no known efficient way to find such a solution –The only definitive way to do so is to try all possible values for the n Boolean variables –That means this is O(2 n )! –Thus it is not a polynomial time function NP stands for “Not Polynomial” Cook’s theorem (1971) states that SAT is NP-complete –There still may be an efficient way to solve it, though!

28 NP Completeness There are hundreds of NP complete problems –It has been shown that if you can solve one of them efficiently, then you can solve them all –Example: the traveling salesman problem Given a number of cities and the costs of traveling from any city to any other city, what is the cheapest round-trip route that visits each city once and then returns to the starting city? Not all algorithms that are O(2 n ) are NP complete –In particular, integer factorization (also O(2 n )) is not thought to be NP complete

29 NP Completeness It is “widely believed” that there is no efficient solution to NP complete problems –In other words, everybody has that belief If you could solve an NP complete problem in polynomial time, you would be showing that P = NP –And you’d get a million dollar prize (and lots of fame!) If this were possible, it would be like proving that Newton’s or Einstein’s laws of physics were wrong In summary: –NP complete problems are very difficult to solve, but easy to check the solutions of –It is believed that there is no efficient way to solve them