1. CS221: Algorithms and Data Structures Lecture #1 Complexity Theory and Asymptotic Analysis
Steve Wolfman
2010W2

2. Today’s Outline Brief Proof Reminder Asymptotic Analysis, Briefly
Silicon Downs and the SD Cheat Sheet
Asymptotic Analysis, Proofs and Programs
Examples and Exercises

3. Learning Goals By the end of this unit, you will be able to…
Define which program operations we measure in an algorithm in order to approximate its efficiency.
Define “input size” and determine the effect (in terms of performance) that input size has on an algorithm.
Give examples of common practical limits of problem size for each complexity class.
Give examples of tractable, intractable, and undecidable problems.
Given code, write a formula which measures the number of steps executed as a function of the size of the input (N).
Continued…

4. Learning Goals By the end of this unit, you will be able to…
Compute the worst-case asymptotic complexity of an algorithm (e.g., the worst possible running time based on the size of the input (N)).
Categorize an algorithm into one of the common complexity classes.
Explain the differences between best-, worst-, and average-case analysis.
Describe why best-case analysis is rarely relevant and how worst-case analysis may never be encountered in practice.
Given two or more algorithms, rank them in terms of their time and space complexity.

5. Proof by... Counterexample Contradiction Induction
show an example which does not fit with the theorem
QED (the theorem is disproven)
Contradiction
assume the opposite of the theorem
derive a contradiction
QED (the theorem is proven)
Induction
prove for a base case (e.g., n = 1)
assume for an anonymous value (n)
prove for the next value (n + 1)
QED
One of the things we’ll need to do in this class is prove correctness and time bounds. In order to do that, we need methods of proof.

6. Simple Example Proof by Induction
Theorem: Sum of the integers 1..n is n(n+1)/2. Base case: Induction Hypothesis: Inductive Step:

7. More Interesting Example Proof by Induction
A number is divisible by 3 iff the sum of its digits is divisible by three
Base Case:
X1x2x3x4…xn SD(x)
x % 3 = SD(x) % 3
tack on digit at end
10 % 3 = 1
Good way to convince them that multiplying by 10 mod 3 is the same as multiplying by 1 mod 3:
10x mod 3 = ( )x mod 3 = 3x + 3x + 3x + x mod 3
Obviously, adding something that’s divisible by 3 doesn’t change anything so:
= x mod 3

8. Example Still Continued
Induction Step:

9. Example Proof by Induction (Worked)
“A number is divisible by 3 iff the sum of its digits is divisible by three.”
First, some definitions:
Consider a positive integer x to be made up of its n digits: x1x2x3..xn.
For convenience, let’s say SD(x) =
X1x2x3x4…xn SD(x)
x % 3 = SD(x) % 3
tack on digit at end
10 % 3 = 1
Good way to convince them that multiplying by 10 mod 3 is the same as multiplying by 1 mod 3:
10x mod 3 = ( )x mod 3 = 3x + 3x + 3x + x mod 3
Obviously, adding something that’s divisible by 3 doesn’t change anything so:
= x mod 3

10. Example Proof by Induction (Worked)
“A number is divisible by 3 iff the sum of its digits is divisible by three.”
There are many ways to solve this, here’s one.
We’ll prove a somewhat stronger property:
Theorem: for a non-negative integer x with any positive integral number of digits n, SD(x) mod 3 = x mod 3.
X1x2x3x4…xn SD(x)
x % 3 = SD(x) % 3
tack on digit at end
10 % 3 = 1
Good way to convince them that multiplying by 10 mod 3 is the same as multiplying by 1 mod 3:
10x mod 3 = ( )x mod 3 = 3x + 3x + 3x + x mod 3
Obviously, adding something that’s divisible by 3 doesn’t change anything so:
= x mod 3

11. Example Proof by Induction (Worked)
Base case:
Consider any number x with one digit (0-9).
SD(x) = = x1 = x.
So, it’s trivially true that SD(x) mod 3 = x mod 3.
X1x2x3x4…xn SD(x)
x % 3 = SD(x) % 3
tack on digit at end
10 % 3 = 1
Good way to convince them that multiplying by 10 mod 3 is the same as multiplying by 1 mod 3:
10x mod 3 = ( )x mod 3 = 3x + 3x + 3x + x mod 3
Obviously, adding something that’s divisible by 3 doesn’t change anything so:
= x mod 3

12. Example Proof by Induction (Worked)
Induction hypothesis:
Assume for an arbitrary integer n > 0 that for any non-negative integer x with n digits:
SD(x) mod 3 = x mod 3.
X1x2x3x4…xn SD(x)
x % 3 = SD(x) % 3
tack on digit at end
10 % 3 = 1
Good way to convince them that multiplying by 10 mod 3 is the same as multiplying by 1 mod 3:
10x mod 3 = ( )x mod 3 = 3x + 3x + 3x + x mod 3
Obviously, adding something that’s divisible by 3 doesn’t change anything so:
= x mod 3

13. Example Proof by Induction (Worked)
Inductive step:
Consider an arbitrary number y with n + 1 digits.
We can think of y as being made up of its digits: y1y2…ynyn+1. Clearly, y1y2…yn (which we’ll call z) is itself an n digit number; so, the induction hypothesis applies:
SD(z) mod 3 = z mod 3.
X1x2x3x4…xn SD(x)
x % 3 = SD(x) % 3
tack on digit at end
10 % 3 = 1
Good way to convince them that multiplying by 10 mod 3 is the same as multiplying by 1 mod 3:
10x mod 3 = ( )x mod 3 = 3x + 3x + 3x + x mod 3
Obviously, adding something that’s divisible by 3 doesn’t change anything so:
= x mod 3

14. Example Proof by Induction (Worked)
Inductive step continued:
Now, note that y = z*10 + yn+1.
So: y mod 3 = (z*10 + yn+1) mod 3
= (z*9 + z + yn+1) mod 3
z*9 is divisible by 3; so, it has no impact on the remainder of the quantity when divided by 3:
= (z + yn+1) mod 3
X1x2x3x4…xn SD(x)
x % 3 = SD(x) % 3
tack on digit at end
10 % 3 = 1
Good way to convince them that multiplying by 10 mod 3 is the same as multiplying by 1 mod 3:
10x mod 3 = ( )x mod 3 = 3x + 3x + 3x + x mod 3
Obviously, adding something that’s divisible by 3 doesn’t change anything so:
= x mod 3

15. Example Proof by Induction (Worked)
Inductive step continued:
By the IH, we know z mod 3 = SD(z) mod 3.
Continuing the previous slide’s work:
y mod = (z + yn+1) mod 3
= (SD(z) + yn+1) mod 3
= (y1 + y2 + … + yn + yn+1) mod 3
= SD(y) mod 3
QED!
We actually use a property of modular arithmetic that we haven’t proven to do the substitution above (putting SD(z) in for z), but it is a correct property.
Can I really sub SD(z) for z inside the mod, even though they’re only equal mod 3?
Yes… they only differ by a multiple of 3, which cannot affect the “mod 3” sum.

16. Today’s Outline Brief Proof Reminder Asymptotic Analysis, Briefly
Silicon Downs and the SD Cheat Sheet
Asymptotic Analysis, Proofs and Programs
Examples and Exercises
[2010W1: stopped here after 20 mins of lecture.]

17. A Task to Solve and Analyze
Find a student’s name in a class given her student ID
How do we generalize our comparison across computers and input sizes?
How do we compare performance? (Pull out things like wall-clock time, network traffic, etc.)
Issues to address:
Array of all possible IDs
Ordered list of Ids
List of Ids
Let’s try some solutions to this:
Specifically, what does an input size MEAN when there can be many inputs of a given size?

18. Asymptotic Analysis of Algorithms
Analyze the trend in performance of an algorithm as a function of input size, seeking insight into performance/alternative algorithms:
Input size is a number n (but sometimes there are multiple inputs)
We count some stand-in for performance based on input size: comparisons executed, “steps” taken, disk accesses, network messages sent, memory used, …
Running time is a function of n (Z0  R0) such as
T(n) = 4n + 5
T(n) = 0.5 n log n - 2n + 7
T(n) = 2n + n3 + 3n
But...
In general, we’ll use analysis of algorithms to determine just this sort of thing. Time and space tradeoffs.
We’ll also use the analyses to see how we can improve our algorithms.
Here’s a general sketch of what an analasys will look like.

19. Asymptotic Analysis Hacks
We only care about trends; so…
Eliminate low order terms
4n + 5  4n
0.5 n log n - 2n + 7  0.5 n log n
2n + n3 + 3n  2n
Eliminate coefficients
4n  n
0.5 n log n  n log n
n log (n2) = 2 n log n  n log n
We didn’t get very precise in our analysis of the ID info finder; why?
Didn’t know the machine we’d use.
Is this always true? Do you buy that coefficients and low order terms don’t matter?
When might they matter?
(Linked list memory usage)

20. Rates of Growth Suppose a computer executes 1012 ops per second:
On the other hand, the high order terms will tend to dominate given enough input. Here’s a chart of how quickly a computer can finish various algorithms according to their runtime.
lg n is log2 n 104s = 2.8 hrs 1018s = 30 billion years
(Oh, and there are “pie seconds” in a nanocentury)

21. Order Notation
T(n)  O(f(n)) if there are constants c and n0 such that T(n)  c f(n) for all n  n0
We’ll use some specific terminology to describe asymptotic behavior.
There are some analogies here that you might find useful.

22. Order Notation
T(n)  O(f(n)) if there are constants c and n0 such that T(n)  c f(n) for all n  n0
T(n)   (f(n)) if f(n)  O(T(n))
T(n)  (f(n)) if T(n)  O(f(n)) and T(n)   (f(n))
We’ll use some specific terminology to describe asymptotic behavior.
There are some analogies here that you might find useful.
2010W2, dropped these INCORRECT definitions (e.g., n vs. n|sin n|, is not o nor theta)
T(n) o(f(n)) if T(n)  O(f(n)) and T(n)  (f(n))
T(n) (f(n)) if T(n)  O(f(n)) and T(n)  (f(n))
Our “hacks” are provable from these definitions.

23. Examples 10,000 n2 + 25 n  (n2) 10-10 n2  (n2) n log n  O(n2)
n log n  (n)
n3 + 4  O(n4) (but not (n4))
n3 + 4  (n2) (but not (n2))

24. Today’s Outline Programming Project #1 and Forming Teams
Brief Proof Reminder
Asymptotic Analysis, Briefly
Silicon Downs and the SD Cheat Sheet
Asymptotic Analysis, Proofs and Programs
Examples and Exercises

25. Silicon Downs Post #1 n3 + 2n2 n0.1 n + 100n0.1 5n5 n-152n/100 82lg n
mn3
Post #2
100n
log n
2n + 10 log n n!
1000n15
3n7 + 7n
2mn
For each race, which “horse” is “faster”. Note that faster means smaller, not larger!
Left
Right
Tied
It depends
I am opposed to algorithm racing.
Welcome, everyone, to the Silicon Downs. I’m getting race results as we stand here.
Let’s start with the first race. I’ll have the first row bet on race #1. Raise your hand if you bet on function #1 (the jockey is n^0.1)
So on.
Show the race slides after each race.

26. Left
Right
Tied
It depends
Race I
n3 + 2n2
vs.
100n

27. Race II n0.1 vs. log n Left Right Tied It depends
Well, log n looked good out of the starting gate and indeed kept on looking good until about n^17 at which point n^0.1 passed it up forever.
Moral of the story? N^epsilon beats log n for any eps > 0. BUT, which one of these is really better?

28. Race III n + 100n0.1 vs. 2n + 10 log n Left Right Tied It depends
Notice that these just look like n and 2n once we get way out. That’s because the larger terms dominate. So, the left is less, but not asymptotically less.
It’s a TIE!

29. Left
Right
Tied
It depends
Race IV
5n5
vs. n!
N! is BIG!!!

30. Race V n-152n/100 vs. 1000n15 Left Right Tied It depends
No matter how you put it, any exponential beats any polynomial. It doesn’t even take that long here (~250 input size)

31. Race VI 82lg(n) vs. 3n7 + 7n Left Right Tied It depends
We can reduce the left hand term to n^6, so they’re both polynomial and it’s an open and shut case.

32. Race VII mn3 vs. 2mn Left Right Tied It depends
We can reduce the left hand term to n^6, so they’re both polynomial and it’s an open and shut case.

33. Silicon Downs Post #1 n3 + 2n2 n0.1 n + 100n0.1 5n5 n-152n/100 82lg n
mn3
Post #2
100n
log n
2n + 10 log n n!
1000n15
3n7 + 7n
2mn
Winner
O(n2)
O(log n)
TIE O(n)
O(n5)
O(n15)
O(n6)
IT DEPENDS
Welcome, everyone, to the Silicon Downs. I’m getting race results as we stand here.
Let’s start with the first race. I’ll have the first row bet on race #1. Raise your hand if you bet on function #1 (the jockey is n^0.1)
So on.
Show the race slides after each race.

34. Mounties Find Silicon Downs Fixed
The fix sheet (typical growth rates in order)
constant: O(1)
logarithmic: O(log n) (logkn, log n2  O(log n))
poly-log: O(logk n)
linear: O(n)
log-linear: O(n log n)
superlinear: O(n1+c) (c is a constant > 0)
quadratic: O(n2)
cubic: O(n3)
polynomial: O(nk) (k is a constant)
exponential: O(cn) (c is a constant > 1)
Well, it turns out that the old Silicon Downs is fixed. They dope up the horses to make the first few laps interesting, but we can always find out who wins.
Here’s a chart comparing some of the functions. Notice that any exponential beats any polynomial.
Any superlinear beats any poly-log-linear.
Also keep in mind (though I won’t show it) that sometimes the input has more than one parameter. Like if you take in two strings. In that case you need to be very careful about what is constant and what can be ignored.
O(log m + 2n) is not necessarily O(2n)
“tractable”
“intractable”

35. The VERY Fixed Parts
There’s also a notion of asymptotic “dominance”, which means one function as a fraction of another (asymptotically dominant) function goes to zero.
Each line below dominates the one above it:
O(1)
O(logk n), where k > 0
O(nc), where 0 < c < 1
O(n)
O(n logk n), where k > 0
O(n1+c), where 0 < c < 1
O(nk), where k  2 (the rest of the polynomials)
O(cn), where c > 1

36. USE those cheat sheets! Which is faster, n3 or n3 log n?
(Split it up and use the “dominance” relationships.)
People didn’t really get the use of this in 2009W1. Stressing more here.

37. Today’s Outline Programming Project #1 and Forming Teams
Brief Proof Reminder
Asymptotic Analysis, Briefly
Silicon Downs and the SD Cheat Sheet
Asymptotic Analysis, Proofs and Programs
Examples and Exercises

38. Terminology Given an algorithm whose running time is T(n)
T(n)  O(f(n)) if there are constants c and n0 such that T(n)  c f(n) for all n  n0
1, log n, n, 100n  O(n)
T(n)  (f(n)) if there are constants c and n0 such that T(n)  c f(n) for all n  n0
n, n2, n, n3 log n  (n)
T(n)  (f(n)) if T(n)  O(f(n)) and T(n)  (f(n))
n, 2n, 100n, 0.01 n + log n  (n)
2009W1, included:
These are “dominance” comparisons, and many of the properties of normal comparisons hold.
theta --> =
omega --> >
Omega --> >=
o --> <
O --> <=
There’s an analogy here that you might find useful. These terms are sort of like inequalities:
Similarly, omega(n),…
You’ve probably seen O(n) before. Here’s how we define it. There’s some value and some scaling factor after which f(n) beats T(n) forever.
OK, here’s how we’ll talk about these things.
T(n)  (f(n)) if T(n)  O(f(n)) and T(n)  (f(n))
T(n)  o(f(n)) if T(n)  O(f(n)) and T(n)  (f(n))
n1.01, n2, n, n3 log n  (n)
1, log n, n0.99  o(n)

39. Types of analysis Orthogonal axes bound flavor analysis case
upper bound (O)
lower bound ()
asymptotically tight ()
analysis case
worst case (adversary)
average case
best case
“common” case
analysis quality
loose bound (any true analysis)
tight bound (no better (reasonable) bound that is asymptotically different)
We already discussed the bound flavor. All of these can be applied to any analysis case. For example, we’ll later prove that sorting in the worst case takes at least n log n time. That’s a lower bound on a worst case.
Note there’s two senses of tight. I’ll try to avoid the terminology “asymptotically tight” and stick with the lower def’n of tight. O(inf) is not tight!
Why “reasonable”? Consider f(n) = n if n is odd, 0 if n is even. F(n) is itself a “tightest” bound on f(n), but it’s a terrible bounding function. O(n) is the tightest “reasonable” bound.

40. Rules of Thumb: Analyzing Code to Count “Steps”
C++ operations - constant time
consecutive stmts - sum of times
conditionals - sum of branches, condition
loops - sum of iterations
function calls - cost of function body
Alright, I want to do some code examples, but first, how will we examine code?
These are just rules of thumb, not the way you always do it.
Sometimes, for ex, it’s important to keep track of which branch of a conditional is followed when (like when analyzing recursive functions).
Above all, use your head!

41. Analyzing Code Step 1: What’s the input size n? // Linear search
find(key, array)
for i = 1 to length(array) - 1 do
if array[i] == key
return i
return -1
Step 1: What’s the input size n?
How do we analyze this? How long does it take? It depends on the input, right?
Talk about the problems with average case here…
Average case is hard! What does “average” mean. For example, what’s the average case for searching an unordered list (as precise as possible, not asymptotic).
WRONG (maybe)! It’s about n, not 1/2 n. Why? You have to search the whole thing if the elt is not there.

42. Analyzing Code
// Linear search
find(key, array)
for i = 1 to length(array) - 1 do
if array[i] == key
return i
return -1
Step 2: What kind of analysis should we perform? Worst-case? Best-case? Average-case?
Expected-case, amortized, …
How do we analyze this? How long does it take? It depends on the input, right?
Talk about the problems with average case here…
Average case is hard! What does “average” mean. For example, what’s the average case for searching an unordered list (as precise as possible, not asymptotic).
WRONG (maybe)! It’s about n, not 1/2 n. Why? You have to search the whole thing if the elt is not there.

43. Analyzing Code
// Linear search
find(key, array)
for i = 1 to length(array) - 1 do
if array[i] == key
return i
return -1
Step 3: How much does each line cost? (Are lines the right unit?)
How do we analyze this? How long does it take? It depends on the input, right?
Talk about the problems with average case here…
Average case is hard! What does “average” mean. For example, what’s the average case for searching an unordered list (as precise as possible, not asymptotic).
WRONG (maybe)! It’s about n, not 1/2 n. Why? You have to search the whole thing if the elt is not there.

44. Analyzing Code Step 4: What’s T(n) in its raw form? // Linear search
find(key, array)
for i = 1 to length(array) - 1 do
if array[i] == key
return i
return -1
Step 4: What’s T(n) in its raw form?
How do we analyze this? How long does it take? It depends on the input, right?
Talk about the problems with average case here…
Average case is hard! What does “average” mean. For example, what’s the average case for searching an unordered list (as precise as possible, not asymptotic).
WRONG (maybe)! It’s about n, not 1/2 n. Why? You have to search the whole thing if the elt is not there.

45. Analyzing Code
// Linear search
find(key, array)
for i = 1 to length(array) - 1 do
if array[i] == key
return i
return -1
Step 5: Simplify T(n) and convert to order notation. (Also, which order notation: O, , ?)
How do we analyze this? How long does it take? It depends on the input, right?
Talk about the problems with average case here…
Average case is hard! What does “average” mean. For example, what’s the average case for searching an unordered list (as precise as possible, not asymptotic).
WRONG (maybe)! It’s about n, not 1/2 n. Why? You have to search the whole thing if the elt is not there.

46. Analyzing Code
// Linear search
find(key, array)
for i = 1 to length(array) - 1 do
if array[i] == key
return i
return -1
Step 6: Casually name-drop the appropriate terms in order to sound bracingly cool to colleagues: “Oh, linear search? That’s tractable, polynomial time. What polynomial? Linear, duh. See the name?! I hear it’s sub-linear on quantum computers, though. Wild, eh?”
How do we analyze this? How long does it take? It depends on the input, right?
Talk about the problems with average case here…
Average case is hard! What does “average” mean. For example, what’s the average case for searching an unordered list (as precise as possible, not asymptotic).
WRONG (maybe)! It’s about n, not 1/2 n. Why? You have to search the whole thing if the elt is not there.

47. Analyzing Code
// Linear search
find(key, array)
for i = 1 to length(array) - 1 do
if array[i] == key
return i
return -1
Step 7: Prove the asymptotic bound by finding constants c and n0 such that for all n  n0, T(n)  cn.
How do we analyze this? How long does it take? It depends on the input, right?
Talk about the problems with average case here…
Average case is hard! What does “average” mean. For example, what’s the average case for searching an unordered list (as precise as possible, not asymptotic).
WRONG (maybe)! It’s about n, not 1/2 n. Why? You have to search the whole thing if the elt is not there.
You usually won’t do this in practice.

48. Today’s Outline Programming Project #1 and Forming Teams
Brief Proof Reminder
Asymptotic Analysis, Briefly
Silicon Downs and the SD Cheat Sheet
Asymptotic Analysis, Proofs and Programs
Examples and Exercises

49. More Examples Than You Can Shake a Stick At (#0)
// Linear search
find(key, array)
for i = 1 to length(array) - 1 do
if array[i] == key
return i
return -1
Here’s a whack-load of examples for us to:
find a function T(n) describing its runtime
find T(n)’s asymptotic complexity
find c and n0 to prove the complexity
How do we analyze this? How long does it take? It depends on the input, right?
Talk about the problems with average case here…
Average case is hard! What does “average” mean. For example, what’s the average case for searching an unordered list (as precise as possible, not asymptotic).
WRONG (maybe)! It’s about n, not 1/2 n. Why? You have to search the whole thing if the elt is not there.

50. METYCSSA (#1) Time complexity: O(n) O(n lg n) O(n2) O(n2 lg n)
for i = 1 to n do
for j = 1 to n do
sum = sum + 1
Time complexity:
O(n)
O(n lg n)
O(n2)
O(n2 lg n)
None of these
I spent 5 minutes looking at my notes trying to figure out why this 1 was an i when it wasn’t. So, I’m putting variables in blue italics.
This example is pretty straightforward. Each loop goes N times, constant amount of work on the inside.
N*N*1 = O(N^2)

51. METYCSSA (#2) Time complexity: O(n) O(n lg n) O(n2) O(n2 lg n)
while i < n do
for j = i to n do
sum = sum + 1
i++
Time complexity:
O(n)
O(n lg n)
O(n2)
O(n2 lg n)
None of these
There’s a little twist here. J goes from I to N, not 1 to N.
So, let’s do the sums
inside is constant.
Next loop is sum I to N of 1 which equals N - I + 1
Outer loop is sum 1 to N of N - I + 1
That’s the same as
sum N to 1 of I
or N(N+1)/2
or O(N^2)

52. METYCSSA (#3) Time complexity: O(n) O(n lg n) O(n2) O(n2 lg n)
while i < n do
for j = 1 to i do
sum = sum + 1
i += i
Time complexity:
O(n)
O(n lg n)
O(n2)
O(n2 lg n)
None of these
Easy to think this might go n times through the first loop and n times through the second so we get O(n2). No such luck. Of course, O(n2) is correct as an upper-bound. It’s just not tight.
This sum is: S_0^floor(lg (n – 1)) S_1^(2^i) O(1)
Let’s just make O(1) into 1 for now; we can multiply by some constant c later if we want.
S_0^floor(lg(n-1)) 2^i
The sum of powers of 2 from 0 to k turns out to be 2^(k+1) – 1.
== 2^(floor(lg(n-1)) + 1) – 1
We can ditch the floor since this is a O analysis
== 2^lg(n-1)*2 – 1 == (n-1)*2 – 1 = 2n-2 = O(n)
Cool! 
This turns out to be a tight bound.

53. METYCSSA (#4) Conditional Loops if C then S1 else S2 while C do S
OK, so this isn’t exactly an example. Just reiterating the rule.
Time <= time of C plus max of S1 and S2
<= time of C plus S1 plus S2
time <= sum of times of iterations
often #of iterations * time of S (or worst time of S)

54. METYCSSA (#5) Recursion almost always yields a recurrence
Recursive max:
if length == 1: return arr[0]
else: return larger of arr[0] and max(arr[1..length-1])
T(1) <= b
T(n) <= c + T(n - 1) if n > 1
Analysis
T(n) <= c + c + T(n - 2) (by substitution)
T(n) <= c + c + c + T(n - 3) (by substitution, again)
T(n) <= kc + T(n - k) (extrapolating 0 < k  n)
T(n) <= (n – 1)c + T(1) = (n – 1)c + b (for k = n - 1)
T(n) 
[2010W2] De-emphasize this term and leave details and rigorous practice to 320 (but can’t ignore completely since both asympt analysis and recursion are 221’s “job”!)
You may want to take notes on this slide as it just vaguely resembles a homework problem!
Here’s a function defined in terms of itself. You see this a lot with recursion. This one is a lot like the profile for factorial.
WORK THROUGH
Answer: O(n)

55. METYCSSA (#6): Mergesort
Mergesort algorithm
split list in half, sort first half, sort second half, merge together
T(1) <= b
T(n) <= 2T(n/2) + cn if n > 1
Analysis
T(n) <= 2T(n/2) + cn
<= 2(2T(n/4) + c(n/2)) + cn
= 4T(n/4) + cn + cn
<= 4(2T(n/8) + c(n/4)) + cn + cn
= 8T(n/8) + cn + cn + cn
<= 2kT(n/2k) + kcn (extrapolating 1 < k  n)
<= nT(1) + cn lg n (for 2k = n or k = lg n)
T(n) 
This is the same sort of analysis as last slide.
Here’s a function defined in terms of itself.
WORK THROUGH
Answer: O(n log n)
Generally, then, the strategy is to keep expanding these things out until you see a pattern. Then, write the general form. Finally, sub in for the series bounds to make T(?) come out to a known value and solve all the series.
Tip: Look for powers/multiples of the numbers that appear in the original equation.

56. METYCSSA (#7): Fibonacci
Recursive Fibonacci:
int Fib(n)
if (n == 1 or n == 2) return 1
else return Fib(n - 1) + Fib(n - 2)
Lower bound analysis
T(1), T(2)  b
T(n) = T(n - 1) + T(n - 2) + c if n > 2
Analysis
let  be (1 + 5)/2 which satisfies 2 =  + 1
show by induction on n that T(n)  bn - 2
There’s no reason that people should have guessed phi’s involvement, but given these properties, asking about an induction problem like this wouldn’t be unreasonable.
(Should also be doable to develop looser bounds, though I haven’t double-checked that!)

57. Example #7 continued Basis: T(1)  b > b-1 and T(2)  b = b0
Inductive step: Assume T(m)  bm - 2 for all m < n
T(n) = T(n - 1) + T(n - 2) + c
 bn-3 + bn-4 + c
 bn-4( + 1) + c
= bn-42 + c
 bn-2
T(n) 
Why? Same recursive call is made numerous times.

58. Example #7: Learning from Analysis
To avoid recursive calls
store all basis values in a table
each time you calculate an answer, store it in the table
before performing any calculation for a value n
check if a valid answer for n is in the table
if so, return it
This strategy is called “memoization” and is closely related to “dynamic programming”
How much time does this version take?

59. Final Concrete Example (#8): Longest Common Subsequence
Problem: given two strings (m and n), find the longest sequence of characters which appears in order in both strings
lots of applications, e.g., DNA sequencing
Example:
“search me” and “insane method” = “same”
This problem shows what it’s like to have two parameters.
The problem is: STATE
There’s lots of applications, but we’re being theoreticians right now, so we get to not care.
So, what techniques can we use to solve this and how long will they take?
(Just get through brute force at least. Every subsequence of string m against string n:
O(2mn))
(Hm.. That assumes we’re somehow clever about repeats.. I guess we just build a DFA; still O(n) to compare one complete sequence vs. another string.)

60. Abstract Example (#9): It’s Log!
Problem: find an asymptotically tight () bound on T(n) = lg(n!)
Time complexity:
(n)
(n lg n)
(n2)
(n2 lg n)
None of these
Converting to a sum and then doing some clever splitting in half will get us there.
Student also suggested shortcutting the sum conversion and bounding by n^n inside the log. Good approach! Can do (n/2)^(n/2) inside as well.

61. Log Aside
logab means “the exponent that turns a into b” lg x means “log2x” (our usual log in CS) log x means “log10x” (the common log) ln x means “logex” (the natural log) But… O(lg n) = O(log n) = O(ln n) because: logab = logcb / logca (for c > 1) so, there’s just a constant factor between log bases

62. Asymptotic Analysis Summary
Determine what characterizes a problem’s size
Express how much resources (time, memory, etc.) an algorithm requires as a function of input size using O(•), (•), (•)
worst case
best case
average case
common case
overall: which one’s the right one?
Can anyone tell me what the right analysis to use for the overall case is?
IT DEPENDS!

63. Some Well-Known Horses from the Downs
For general problems (not particular algorithms): We can prove lower bounds on any solution. We can give example algorithms to establish “upper bounds” for the best possible solution. (Ideally, an upper bound matching the problem’s lower bound!) Searching an unsorted list using comparisons: provably (n), linear search is O(n). Sorting a list using comparisons: provably (n lg n), mergesort is O(n lg n).

64. Aside: Who Cares About (lg (n!))? Can You Beat O(n lg n) Search?
Chew these over:
How many values can you represent with n bits?
Comparing two values (x < y) gives you one bit of information.
There are n! possible ways to reorder a list. We could number them: 1, 2, …, n!
Sorting basically means choosing which of those reorderings/numbers you’ll apply to your input.
How many comparisons does it take to pick among n! numbers?

65. Some Well-Known Horses from the Downs
Searching and Sorting: polynomial time, tractable
Traveling Salesman Problem: non-deterministic polynomial… can check a guess in polynomial time, maybe exponential time to solve.
Right-hand image from paper: Craig S. Kaplan and Robert Bosch. TSP Art. In Renaissance Banff: Bridges 2005: Mathematical Connections in Art, Music and Science, pages , Fair use (and Craig knows I’m talking about his stuff).
Don’t have source for left-hand 
Are problems in NP really in P? $1M prize to prove yea or nay.

66. Some Well-Known Horses from the Downs
Searching and Sorting numbers: P, tractable
Traveling Salesman Problem: NP, intractable
Kolmogorov Complexity: uncomputable
The Kolmogorov Complexity of a string is its shortest representation in a given language. In other words, it tells us how much the string can possibly be compressed.
Can’t be computed. Pithy but hand-wavy proof: What’s: “The smallest positive integer that cannot be described in fewer than fourteen words.”
The text at the bottom is effectively the Berry Paradox, described by Russell, according to Wikipedia.

67. Learning Goals By the end of this unit, you will be able to…
Define which program operations we measure in an algorithm in order to approximate its efficiency.
Define “input size” and determine the effect (in terms of performance) that input size has on an algorithm.
Give examples of common practical limits of problem size for each complexity class.
Give examples of tractable, intractable, and undecidable problems.
Given code, write a formula which measures the number of steps executed as a function of the size of the input (N).
Continued…

68. Learning Goals By the end of this unit, you will be able to…
Compute the worst-case asymptotic complexity of an algorithm (e.g., the worst possible running time based on the size of the input (N)).
Categorize an algorithm into one of the common complexity classes.
Explain the differences between best-, worst-, and average-case analysis.
Describe why best-case analysis is rarely relevant and how worst-case analysis may never be encountered in practice.
Given two or more algorithms, rank them in terms of their time and space complexity.

69. To Do
Find a teammate for your labs and your assignments (can be different)
Finish HW 1
Finish Project 1
Read Epp / (3rd ed/4th ed) and Koffman 2.6
Prepare for upcoming labs

70. Coming Up Recursion and Induction
Loop Invariants and Proving Program Correctness
Call Stacks and Tail Recursion
First Written Homework due
First Programming Project due
TODO: add due dates for homework/project.