1. Comparing Runtime Functions

2. Objective: Compare runtimes of algorithms
We want to compare the runtimes of two algorithms:
Not of architectures
Or programming languages
Or syntax
Basic idea: we compare how fast runtime increases as input grows.

3. Sample Comparison – why it matters
Multiply two n x n matrices, X and Y:
MMultA: Requires ≈ n3 seconds
60003 = 2.16x1011
MMultB: requires about 9n2.376 seconds
9* = 1.42x1010
About 25 times faster
1 run per day, vs. 1 run per hour

4. Second Sample Comparison
Computer 2:
50 times faster
Regular matrix
multiplication
Computer 1:
Strassen’s algorithm for matrix multiplication
Research: for a 1600x1600 matrix,
Comp 1 is about 3.5 times faster

5. If we move to a faster computer they just change anyway.
Runtime Functions
The runtime of an algorithm is determined by input size – runtime increases as size increases.
We can usually express this as a function: f(n) is the runtime for an input of size n.
If we move to a faster computer they just change anyway.
Example: “JunkSort requires 5n2 + 7 nanoseconds to sort n elements”
We ignore the actual units of time.
JunkSort: f(n) = 5n2 + 7

6. Properties of Runtime Functions
Runtime functions are defined for n ≥ 0
Input cannot have a negative size
Runtime functions are non-negative
An algorithm cannot have negative runtime
n ≥ 0 f(n) ≥ 0
Runtime functions are non-decreasing
Increasing the input size does not decrease runtime.
n ≥ 0 f(n) ≤ f(n+1)

7. Judging Runtime What makes a runtime “good” or “bad”?
Should we define this generally, or within a context?
How do we compare two runtimes?
Compare at a single value of n?
Compare over a range of n?
Compare over all n?

8. Comparing Functions
Consider two functions:
f(n) = n2
g(n) = 100n

9. Plot over [1,10]
n2 is clearly much better over this range

10. Plot over [1,100]
100n catches up by the end of this range – n2 has been “growing” faster

11. Plot over [1,1000]
100n is clearly much better over this range… and will continue to get even better – n2 continues to grow faster

12. Summing Up Comparison
Graphically: n2 is clearly worse than 100n for “all large enough” n
Mathematically: it is easy to show that n2 > 100n for all n > 100
Or 101, or 246, or 3948…
Ratio: 100n / n2 = 100/n
For large n, 100/n is very small
Hence for large n, 100n is much smaller than n2

13. Crossover Point
Crossover point: N=100.
g(n) <= f(n) for all n N

14. Scaling the Functions Suppose we have runtimes n3 and 200n2.
What if we move the blue to a computer 50% slower?
Multiply 200n2 by 1.5 and the crossover still exists (shifting to N=300).
Scaling the function moves the crossover, but doesn’t eliminate it.

15. Adding a Term Start with n3 and 200n2 and find the crossover N=200.
Add 10000n to 200n2 and the crossover shifts again
Adding “lower order” terms moves the crossover, but doesn’t eliminate it.

16. Comparing Functions
First try – we won’t stick with this
Proposal We could say that runtime f(n) is “at least as good as” g(n) if there is an N such that:
f(n)  g(n) for all n > N
In other words: “g(n) might be better for some limited range of n, but f(n) does as well in the long term – it doesn’t grow faster.”
This is the (x-location) of the crossover point

17. Crossover Point
Crossover point: N=100.
g(n)  f(n) for all n N

18. Scaling the Comparison
Scaling changes our crossover point location – but doesn’t change our crossover point existance
What we want: if f(n) is “at least as good” as g(n), then it is at least as good as cg(n) for any c > 0.
In other words: f(n) is at least as good as g(n) if there exists some N, c > 0 such that f(n) ≤ cg(n) for all n > N.
This definition will actually be easier to work with

19. Crossover Point Thus g(n) is “at least as good” as f(n).
Crossover point: N=100.
g(n)  cf(n) for all n N if c = 1
Thus g(n) is “at least as good” as f(n).

20. Notation
This “at least as good as” notation is bulky – lets find something easier Standard: If f(n) is “at least as good as” g(n) then we say f(n) = O(g(n)) “f(n) is big-oh of g(n)” “f(n) is order g(n)” “g(n) is an asymptotic bound of f(n)”

21. Just repeating what we said two slides ago.
Formally Defined
Just repeating what we said two slides ago.
For functions f(n) and g(n), f(n) = O(g(n)) if N, c > 0 such that f(n) < cg(n) n > N

22. Proving big-oh Is 100n = O(n2)?
Does there exist an N and c > 0 s.t. 100n < cn2 for all n > N?
Try:
N=1, c = 100
N=1
For any n > N:
100n < 100*n*n
= 100n2
= cn2
 100n = O(n2)

23. Proving big-oh Is n2 + n = O(n3)?
Does there exist an N and c > 0 s.t. n2 + n < cn3 for all n > N?
Try:
N=1, c = 2
N=1
For any n > N:
n2 + n < n2 + n2
= 2n2
< 2n2*n
= cn3
 n2 + n = O(n3)

24. Proving big-oh Is 10n2 + 20n = O(n2)?
Does there exist an N and c > 0 s.t. 10n2 + 20n ≤ cn2 for all n > N?
Try:
N=1, c = 30
N=1
For any n > N:
10n2 + 20n < 10n2 + 20n2
= 30n2
= cn2
 10n2 + 20n = O(n2)
Notice: Easy to show that
n2 = O(10n2 + 20n)
(Use N=1 and c=1)

25. Dispoving big-oh Is n3 = O(100n2 )? No How do we prove this?
Proof by contradiction
Assume it is true: N, c > 0 s.t: n>N n3  c*100n2
n > N c > n3 / 100n2
n > N c > n/100
Notice: If n > max{100c,N}, than c < n/100
 No matter what N is, these cannot both be true
 We have a contradiction, so n3  O(100n2)
Solved for C
Canceled variables
The 100 was irrelevant in the proof.
n3  O(kn2) for all k > 0.

26. Formal (Final) Definition of O()
O(g(n)) = {f(n) : N, c > 0 s.t. n > N f(n) < cg(n)}
Technically: We should say n2  O(n3)
In practice: Nobody ever says it this way.

27. logan = O(n) logan = log2n / log2a Is logan = O(n) (a > 1)?
Does there exist an N and c > 0 s.t. loga n < cn for all n > N?
Try:
N=2, c = 1/log2a
N=2
For any n > N:
logan = log2n / log2a
< n / log2a
= cn
 logan = O(n)

28. n  O(logan) Is n = O(logan)? No How do we prove this?
Proof by contradiction
Assume its true: N, c > 0 s.t: n>N n < c*logan
n > N c > n / loga n
But: limn(n/loga n) = limn(1/(n-1logea))
= limn n/logea
= 
 Eventually c < n/loga n
 We have a contradiction, and n  O(logan)
Solved for C
L'Hôpital's
Rule

29. “Simplifying” Runtimes
nk = O(nj) if j ≥ k
loga n = O(logb n)a,b > 0
The base doesn’t matter
logbn = O(nk ) b, k > 0
nk = O(nk log n) k > 0
Constants don’t matter
10478n9 = O(n9)
Lower order terms don’t mater
n n = O(n12)
There are certain rules that make it easy to “simplify” an expression to a basic big-oh bound.
We will not prove these – but you should be able to.

30. Examples What is the “simple” bound on: 3n2 + 2n + 10? O(n2)
n2 log2n + n2?
O(n2log n)
4n3 + n2 log n?
O(n3)
n2 log2 4n?
O(n3)
55n1/2 +17log12 n?
O(n1/2)
2n + 5nk (for any k)?
O(2n)

31. Tight Bounds
It is correct to say f(n) = O(n27) – but that is not tight
It is similar to saying that the population of the United States is “more than 4”.
f(n) = O(n2) is the tight bound
A tight bound is the best bound possible.
Suppose an algorithm has runtime :
f(n)=10n2 + 20n +10

32. Common Runtimes Most runtimes will fall into one of these categories.
Bounds
English
O(1)
“Constant time”
O(log n)
“Log time”
O(n)
“Linear time”
O(n log n)
“n log n time”
O(n2)
“quadratic time”
O(n3)
“cubic time”
O(2n)
“Exponential time”
O(n!)
“Factorial time”
Most runtimes will fall into one of these categories.
If a runtime is O(nk) for any k, then it is polynomial.
Super-polynomial – if your algorithm has a runtime in this area (and no better), you have a problem.

33. Other Bounds O(g(n)): The set of functions “at least as good as” g(n)
What about functions that are “no better than” g(n)?
Or functions that are better than g(n)?
Or functions that are worse than g(n)?

34. The only difference from the O(n) definition is here
(g(n))
(g(n)) = Set of all functions that are “no better than” g(n)
Formally:
(g(n)) = {f(n) : N, c > 0 n>N f(n) > cg(n)}
The only difference from the O(n) definition is here

35. Example
n2= (n)
n3= (n3)
log2 n= (1)

36. (g(n)) = Set of all functions that are “no better than” g(n)
(x3)
(x2)
(x)
(log2 n)
(1)

37. (g(n))
(g(n)) = Set of all functions that are “exactly as good as” g(n)
Formally:
(g(n)) = {f(n) : f(n)=O(g(n)) and f(n)=(g(n))}
Alternative:
(g(n)) = {f(n) : c > 0 s.t. limng(n)/f(n) = c}

38. Example
n2= (n2)
n3 + 8n = (5n3)
log2 n= (log6 n)

39. o(g(n)) o(g(n)) = Set of all functions that are “better than” g(n)
Formally: o(g(n)) = {f(n) : limn g(n)/f(n) = }

40. Example
n= o(n2)
n3= o(n3log2n)
log2 n= o(n0.001)

41. (g(n)) (g(n)) = Set of all functions that are “worse than” g(n)
Formally: (g(n)) = {f(n) : limn g(n)/f(n) = 0}

42. Example
n2= (n)
n3= (10000n2)
log2 n ≠ (log2n)

43. Really, any c ≤ 1/100 will work
Prove: n/100 + n1/2 = (n)
Is n/100 + n1/2 = (n)?
Does there exist an N and c > 0 s.t. n/100 + n1/2  cn for all n > N?
Try:
N=1, c = 1/100
N=1
For any n > N:
n/100 + n1/2 n/100
= cn
 n/100 + n1/2 = (n)
Really, any c ≤ 1/100 will work

44. L’Hospital’s rule again – take the 1st derivative of each part
Prove: log n = o(n)
L’Hospital’s rule again – take the 1st derivative of each part
Prove: log2n = o(n) for any  > 0 limn n / log2n = limn (n)’ / (log2n)’ = limn n-1 / n-1loge2 = limn n / loge2 = limn cn = 
c is just some constant (in this case, c =  / loge2 – but the point is that we don’t care about its value)

45. Graphic Demonstration

46. Simplification methods are similar for each bound
Sample Exercises
You should be able to prove any of these.
7n14 = (n14)
n = o(n2)
Simplification methods are similar for each bound
3n3 + 7n = (n3)
3n = o(4n)
log n = o(n1/2)
n = (log n)
True for all values of k
True for all  > 0
2n = (nk)

47. Only a surface level analogy – don’t take it too far
As an Ordering
Only a surface level analogy – don’t take it too far
Notation
Analogous Relation
English
o()
<
“Faster than”
O() ≤
“As fast as”
() =
“Equal to”
() ≥
“As slow as”
()
>
“Slower than”

48. O() v. ≤
For integers a and b: either a  b or b  a ( is a total ordering)
Is this true for O()?
Is it possible that f(n)O(g(n)) and g(n) O(f(n))?

49. Example
Consider:
f(n) = n sin(n)
g(n) = log2 n

50. No final crossover point for either – hence no relation.
log(n)O(n sin(n))
n sin(n)O(log(n))

51. Bounding Algorithm Runtimes

52. Analyzing Algorithm Runtime
We don’t want to be influenced by:
The programming language
The implementation details
The architecture
We want to be able to rate and compare algorithm runtime
Concern: We want to judge the underlying algorithm

53. Runtime of a Search Algorithm
int search(int A[], int n, int v)
int i = 0;
while (i < n && A[i] != v)
i++;
if (i < n)
return True;
else
return False;
Let ci be the runtime of line i on a specific machine How much time will we spend in line 1? At least c1 How much in line 2? At least c2 Possibly as much as (n+1)c2
This would be the worst case

54. Runtime of a Search Algorithm
int search(int A[], int n, int v)
int i = 0;
while (i < n && A[i] != n)
i++;
if (i < n)
return True;
else
return False;
Maximum runtime: c1(1) + c2(n+1) + c3(n) + c4(1) + max{c5(1), c6(1) + c7(1)} = c1 + c2 + c4 + max{c5, c6 + c7} + n(c2 + c3) = c’ + cn
c’ = c1+c2+c4+max{c5 ,c6 +c7}
c = c2 + c3
But we don’t care.

55. Runtime of a Search Algorithm
int search(int A[], int n, int v)
int i = 0;
while (i < n && A[i] != n)
i++;
if (i < n)
return True;
else
return False;
Runtime: f(n) = c’ + cn
f(n) = O(n)
What if we move this to a computer exactly twice the speed to get runtime g(n)?
ci is cut in half for each i
g(n) = ½ f(n)
= ½ c’ + ½ cn
= O(n)
 Our bound doesn’t change
This method of analysis is independent of architecture speed.

56. Runtime of a Search Algorithm
int search(int A[], int n, int v)
int i = 0;
while (i < n && A[i] != n)
i++;
if (i < n)
return True;
else
return False;
Runtime: f(n) = c’ + cn
f(n) = O(n)
What if our language does not require a type declaration?
Reduces c’ to c’’
g(n) = c’’ + cn
= O(n)
Our bound still doesn’t change
This method of analysis is independent of implementation / programming details.

57. “Timing” a Function
The runtime of an algorithm is determined by two things
Amount of time spent on each line
Number of times each line is executed

58. Runtime of a Line Examples: ++, -- +, -, *, / &&, ||, !, == div, mod
We think of runtime in terms of basic operations
Examples:
++, --
+, -, *, /
&&, ||, !, ==
div, mod
<<, >>
Crucial factor: Each operates independent of input size

59. Simple Code add6 (int v) { v++ }
Number of basic operations: 6 Number of times each operation is invoked: 1 Runtime: f(n) = 6 = O(1)

60. Simple Code add6 (int v) { for i  0 to 5 v++; }
Number of basic operations: Line 1: c Line 2: 1 Number of times each operation is invoked: Line 1: 7 Line 2: 6 Runtime: f(n) = 7c + 6(1) = O(1)

61. Simple Code addN (int v, int n) { for i  0 to n-1 v++ }
Number of basic operations: Line 1: c Line 2: 1 Number of times each operation is invoked: Line 1: n+1 Line 2: n Runtime: f(n) = c(n+1) + n = O(n)

62. Simple Code addN2(int v, int n) { for i  0 to n-1 for j  0 to n-1}
Number of basic operations: Line 1: c Line 2: c’ Line 3: 1 Number of times each operation is invoked: Line 1: n+1 Line 2: (n+1)2 Line 3: n2 Runtime: f(n) = (n+1)c + (n+1)2c + n2 = O(n2)

63. Simple Code foo(int v, int n) { for i  0 to n-1 v++
addN 2(int v, int n) {
foo(n,v) }
Number of basic operations: foo(): O(n) (from before) Line 3: c Line 4: 1 Number of times each operation is invoked: Line 3: n+1 Line 2: n Runtime: f(n) = (n+1)c + nO(n) = O(n2)

64. Why is thinking about the “best” case is pointless?
Simple Code
search(Array A, int n, int v) {
for i  0 to n-1
if A[i] == v
return true
return false }
Number of basic operations: Line 1: c Line 2: c’ Line 3: 1 Line 4: 1 Number of times each operation is invoked: Line 1: Between 0 and n+1 Line 2: Between 0 and n Line 3: 0 or 1 Line 4: 0 or 1 Runtime: At best: f(n) = c(1) + c’(1) + 1 = O(1) At worst: f(n) = c(n+1) + c’(n) + 1 = O(n)
Why is thinking about the “best” case is pointless?

65. EXCEPT… indefinite loops int totalWeight(Graph G) { total = 0
for v in G.nodes()
for w in v.adj()
total += G.weight(v,w)
return total/2
This is a definite loop – the number of iterations depend only on the size of the graph, n.
EXCEPT…
So line (3) will execute ϴ(n) times.
This is not – we cannot tell, from n alone, how many times we will “go around” this loop.
Best we can say is O(n2) executions of line (4)
From this: we get a worse case bound of O(n3) time.

66. indefinite loops int totalWeight(Graph G) { total = 0
for v in G.nodes()
for w in v.adj()
total += G.weight(v,w)
return total
This line is executed exactly twice for each edge (and as much as any other line)
What is the maximum possible number of edges in a (simple) graph of n nodes?
n(n-1)/2
So line (4) will be executed most 2n(n-1)/2 = O(n2) times.
Hence this is an O(n2) algorithm.

67. Three types of analysis
Worst case: what is the worst the runtime can be?
Amortized: We will deal with this later
We will mostly use this – usually pretty straight forward
Average case: what is the runtime for the average input?
Requires a statistical model of what the input might look like

68. We are assuming the array is sorted
Binary Search
We are assuming the array is sorted
BSearch(Array A, int n, int v)
if n  1
return A[0] == v
int mid   (i+j)
if A[mid] == v
return true
if A[mid] > v
return BSearch(A[0..mid], mid, v)
else
return BSearch(A[mid…(n-1)], n-mid, v)
Recursive algorithm – trickier to analyze
How long does the non-recursive portion take?
O(1) time
So if we make r calls, how much time does the algorithm take?
O(r)
Hence: Need to bound r

69. Binary Search
BSearch(Array A, int n, int v)
if n  1
return A[0] == v
int mid   (i+j)
if A[mid] == v
return true
if A[mid] > v
return BSearch(A[0..mid], mid, v)
else
return BSearch(A[mid…(n-1)], n-mid, v)
Assume n is a power of 2 Notice: With each recursion we cut n in half. How many recursions does it take to reduce the size to 1? r = log2n So the algorithm is: O(1*r) = O(log2n)

70. Assumption: “n is a power of 2”
We assumed n was a power of 2: does this affect the analysis? If n is not a power of 2 than 2log n < n < 2log n It follows that f(2log n)≤ f(n) ≤ f(2log n) Hence f(n) is bound by f(2log n) This assumption doesn’t matter – make it whenever it is convenient

71. Need to elimination a fraction– not a fixed amount – to get a speedup
Variation Search
5Search(Array A, int n, int v)
if n  5
for i  0 to 4
if (A[i] == v)
return true;
return false;
if A[5] == v
return true
if A[5] > v
return 5Search(A[0…4], 5, v)
else
return 5Search(A[5…(n-1)], n-5, v)
What is the runtime for the non-recursive portion?
O(1)
What is the maximum number of recursive calls?
n/5
What is the worst-case runtime of 5Search?
O(n/5) = O(n)
Need to elimination a fraction– not a fixed amount – to get a speedup

72. Asymptotic Bounds: Algorithm v. Problem

73. Worst-Case Bound on Algorithm
When we say an algorithm is O(g(n)) in worst case, this means:
n the algorithm takes at most f(n) time.
f(n) = O(g(n))
Here we are talking about a specific algorithm
When we say a problem is O(g(n)) in worst case, this means:
There exists some algorithm that solves the problem in O(g(n)) time.
Slower algorithms are irrelevant
Here we are talking about the best algorithm for a problem

74. Example: Sorting The bubble sort sorting algorithm is O(n2)
Regardless of implementation, the worst-case runtime function f(n)=O(n2)
The bubble sort algorithm is not O(n log n)
If f(n) = O(n log n) then for every large enough n, f(n) underestimates the runtime of some input of size n
The sorting problem is
O(n log n)
There is an algorithm (merge sort) that solves the problem and has O(n log n) runtime.
This principle can be extended to the other asymptotic notations (e.g. , , etc…)

75. Consider… For two n by n matrices:
The standard multiplication algorithm take O(n3) time…
…so the “matrix multiplication problem” is an O(n3) problem.
(Actually, its an O(n2.32) problem – and maybe better.)
Now prove that matrix multiplication is an (n2) problem

76. Sorting Bound Sorting is a (n log n) problem. Meaning:
There exists an O(n log n) sorting algorithm.
Every possible sorting algorithm is (n log n).