1. CSC 380: Design and Analysis of Algorithms
Dr. Curry Guinn

2. Quick Info Dr. Curry Guinn CIS 2015 guinnc@uncw.edu
Office Hours: MTF: 10:00am-11:00m and by appointment

3. Today Analysis of Algorithms The model Relative rates of growth
Big oh and its kin
Homework 1 due Sunday, Jan 27
Canvas Quiz 3 due by Tuesday, 11:59pm, Jan 29
No late Canvas quizzes
No make up quizzes

4. RAM model Typically use a simple model for basic operation costs
RAM (Random Access Machine) model
RAM model has all the basic operations: +, -, *, / , =, comparisons
fixed sized integers (e.g., 32-bit)
infinite memory
All basic operations take exactly one time unit (one CPU instruction) to execute
analysis= determining run-time efficiency.
model = estimate, not meant to represent everything in real-world

5. Critique of the model Strengths: Weaknesses: simple
easier to prove things about the model than the real machine
can estimate algorithm behavior on any hardware/software
Weaknesses:
not all operations take the same amount of time in a real machine
does not account for page faults, disk accesses, limited memory, floating point math, etc
model = approximation of real world.
can predict run-time of algorithm on machine.

6. Relative rates of growth
Most algorithms' runtime can be expressed as a function of the input size N
Rate of growth: measure of how quickly the graph of a function rises
Goal: distinguish between fast- and slow-growing functions
We only care about very large input sizes (for small sizes, most any algorithm is fast enough)
Motivation:
we usually care only about algorithm performance when there are large number of inputs.
We usually don’t care about small changes in run-time performance. (inaccuracy of estimates
make small changes less relevant). Consider algorithms with slow growth rate better than
those with fast growth rates.

7. Growth rate example Consider these graphs of functions.
Perhaps each one represents an algorithm:
n3 + 2n2
100n
Which grows faster?

8. Growth rate example
How about now?

9. “The fundamental law of computer science: As machines become more powerful, the efficiency of algorithms grows more important, not less.” — Nick Trefethen
An algorithm (or function or technique …) that works well when used with large problems & large systems is said to be scalable.
Or “it scales well”.

10. Big O
Definition: T(N) = O(f(N)) if there exist positive constants c , n0 such that: T(N)  c · f(N) for all N  n0
Idea: We are concerned with how the function grows when N is large. We are not picky about constant factors: coarse distinctions among functions
Lingo: "T(N) grows no faster than f(N)."

11. Big O  c , n0 > 0 such that f(N)  c g(N) when N  n0
f(N) grows no faster than g(N) for “large” N

12. Preferred big-O usage pick tightest bound. If f(N) = 5N, then:
f(N) = O(N5)
f(N) = O(N3)
f(N) = O(N log N)
f(N) = O(N)  preferred
ignore constant factors and low order terms
T(N) = O(N), not T(N) = O(5N)
T(N) = O(N3), not T(N) = O(N3 + N2 + N log N)
remove non-base-2 logarithms
f(N) = O(N log6 N)
f(N) = O(N log N)  preferred

13. Big-O of selected functions

14. Big Omega, Theta
Defn: T(N) = (g(N)) if there are positive constants c and n0 such that T(N)  c g(N) for all N  n0
Lingo: "T(N) grows no slower than g(N)."
Defn: T(N) = (g(N)) if and only if T(N) = O(g(N)) and T(N) = (g(N)).
Big-O, Omega, and Theta establish a relative ordering among all functions of N

15. Intuition about the notations
O (Big-O) 
 (Big-Omega) 
 (Theta) =
o (little-O)
<

16. Big-Omega f(N) grows no slower than g(N) for “large” N
 c , n0 > 0 such that f(N)  c g(N) when N  n0
f(N) grows no slower than g(N) for “large” N

17. Big Theta: f(N) = (g(N))
the growth rate of f(N) is the same as the growth rate of g(N)

18. An O(1) algorithm is constant time.
The running time of such an algorithm is essentially independent of the input.
Such algorithms are rare, since they cannot even read all of their input.
An O(logbn) [for some b] algorithm is logarithmic time.
We do not care what b is.
An O(n) algorithm is linear time.
Such algorithms are not rare.
This is as fast as an algorithm can be and still read all of its input.
An O(n logbn) [for some b] algorithm is log-linear time.
This is about as slow as an algorithm can be and still be truly useful (scalable).
An O(n2) algorithm is quadratic time.
These are usually too slow.
An O(bn) [for some b] algorithm is exponential time.
These algorithms are much too slow to be useful.

19. Hammerin’ the terminolgy
T(N) = O(f(N))
f(N) is an upper bound on T(N)
T(N) grows no faster than f(N)
T(N) = (g(N))
g(N) is a lower bound on T(N)
T(N) grows at least as fast as g(N)
T(N) = o(h(N)) (little-O)
T(N) grows strictly slower than h(N)

20. Notations Asymptotically less than or equal to O (Big-O)
Asymptotically greater than or equal to  (Big-Omega)
Asymptotically equal to  (Big-Theta)
Asymptotically strictly less o (Little-O)

21. Facts about big-O
If T(N) is a polynomial of degree k, then: T(N) = (Nk)
example: 17n3 + 2n2 + 4n + 1 = (n3)

22. Hierarchy of Big-O Functions, ranked in increasing order of growth: 1
log n n
n log n n2
n2 log n n3
... 2n n! nn

23. Various growth rates

24. Techniques for Determining Which Grows Faster
Evaluate:
limit is
Big-Oh relation
f(N) = o(g(N))
c  0
f(N) = (g(N)) 
g(N) = o(f(N))
no limit
no relation

25. Techniques, cont'd L'Hôpital's rule: If and , then
example: f(N) = N, g(N) = log N
Use L'Hôpital's rule
f'(N) = 1, g'(N) = 1/N
 g(N) = o(f(N))

26. Program loop runtimes
for (int i = 0; i < n; i += c) // O(n)
statement(s);
Adding to the loop counter means that the loop runtime grows linearly when compared to its maximum value n.
Loop executes its body exactly n / c times.
for j in range(0, n, c): // O(n)
statement(s)
Or if myList contains n elements
for item in myList: // O(n)

27. More loop runtimes Nesting loops multiplies their runtimes.
for (int i = 0; i < n; i += c) { //O(n2)
for (int j = 0; j < n; i += c) {
statement;
} }
for j in range(0, n, c): // O(n2)
for k in range(0, n, c):
statement(s)
Or if myList contains n elements
for item in myList: // O(n2)
for item in myList:

28. The loop maximum is n2, so the runtime is quadratic.
for (int i = 0; i < n * n; i += c) // O(n2)
statement(s);
The loop maximum is n2, so the runtime is quadratic.
Loop executes its body exactly (n2 / c) times.
for j in range(0, n*n, c): // O(n2)
statement(s)

29. for (int i = 1; i <= n; i *= c) // O(log n)
statement(s);
Multiplying the loop counter means that the maximum value n must grow exponentially to linearly increase the loop runtime; therefore, it is logarithmic.
Loop executes its body exactly logc n times.
j = 1
while j <= n: // O(log n)
statement(s)
j *= c

30. for (int i = n; i >= 1; i /= c) // O(log n)
statement(s);
Multiplying the loop counter means that the maximum value n must grow exponentially to linearly increase the loop runtime; therefore, it is logarithmic.
Loop executes its body exactly logc n times.
while n >= 1: // O(log n)
statement(s)
n /= c

31. Loops in sequence add together their runtimes, which means the loop set with the larger runtime dominates.
for (int i = 0; i < n; i += c) { // O(n)
statement; }
// O(nlog n)
for (int i = 0; i < n; i += c) {
for (int j = 0; j < n; i *= c) {
} }

32. Types of runtime analysis
Express the running time as f(N), where N is the size of the input
worst case: your enemy gets to pick the input
average case: need to assume a probability distribution on the inputs
However, even with input size N, cost of an algorithm could vary on different input.

33. Example: Sequential search
Worst case
Best case
Average case
A. Levitin “Introduction to the Design & Analysis of Algorithms,” 3rd ed., Ch. 2 ©2012 Pearson Education, Inc. Upper Saddle River, NJ. All Rights Reserved.

34. Some rules When considering the growth rate of a function using Big-O
Ignore the lower order terms and the coefficients of the highest-order term
No need to specify the base of logarithm
Changing the base from one constant to another changes the value of the logarithm by only a constant factor
If T1(N) = O(f(N) and T2(N) = O(g(N)), then
T1(N) + T2(N) = max(O(f(N)), O(g(N))),
T1(N) * T2(N) = O(f(N) * g(N))

35. For Next Class, Friday
Homework 1 is due Sunday, 11:59pm, Jan 27, 11:59pm
Canvas Quiz 3 due by Tuesday, 11:59pm, Jan 29
No late Canvas quizzes
No make up quizzes