1. CSE 830: Design and Theory of Algorithms
Dr. Eric Torng
TA: Carl Bussema
Many slides adapted from those used by Dr. Charles Ofria
Welcome to CSE 830, the Design and Theory of Algorithms.
My name is Charles, and I will be your instructor for this class. The goal of the class is to teach you the fundamentals of creating and analyzing computer algorithms.
Before we get to the actual material, however, I need to go through a few administrative details. While I take attendance, I have some handouts for you to be reading over.

2. Outline Definitions Course Objectives Administrative stuff …
Algorithms
Problems
Course Objectives
Administrative stuff …
Analysis of Algorithms

3. What is an Algorithm?
Algorithms are the ideas behind computer programs.
An algorithm is the thing that stays the same whether the program is in C++ running on a Cray in New York or is in BASIC running on a Macintosh in Alaska!
To be interesting, an algorithm has to solve a general, specified problem.

4. What is a problem? Definition Problem Specification Example: Sorting
A mapping/relation between a set of input instances (domain) and an output set (range)
Problem Specification
Specify what a typical input instance is
Specify what the output should be in terms of the input instance
Example: Sorting
Input: A sequence of N numbers a1…an
Output: the permutation (reordering) of the input sequence such that a1  a2  …  an .

5. Types of Problems Search: find X in the input satisfying property Y
Structuring: Transform input X to satisfy property Y
Construction: Build X satisfying Y
Optimization: Find the best X satisfying property Y
Decision: Does X satisfy Y?
Adaptive: Maintain property Y over time.

6. Two desired properties of algorithms
Correctness
Always provides correct output when presented with legal input
Efficiency
Computes correct output quickly given input

7. Correctness Example: Traveling Salesperson Problem (TSP)
Input: A sequence of N cities with the distances dij between each pair of cities
Output: a permutation (ordering) of the cities <c1’, …, cn’> that minimizes the expression
Σj =1 to n-1 dj’,j’+1 + dn’,1’
Which of the following algorithms is correct?
Nearest neighbor: Initialize tour to city 1. Extend tour by visiting nearest unvisited city. Finally return to city 1.
All tours: Try all possible orderings of the points selecting the ordering that minimizes the total length:

8. Efficiency Example: Odd Number Problem Input: A number n
Output: Yes if n is odd, no if n is even
Which of the following algorithms is most efficient?
Count up to that number from one and alternate naming each number as odd or even.
Factor the number and see if there are any twos in the factorization.
Keep a lookup table of all numbers from 0 to the maximum integer.
Look at the last bit (or digit) of the number.

9. Outline Definitions Course Objectives Administrative stuff …
Algorithms
Problems
Course Objectives
Administrative stuff …
Analysis of Algorithms

10. Course Objectives Details of classic algorithms
Methods for designing algorithms
Validate/verify algorithm correctness
Analyze algorithm efficiency
Prove (or at least indicate) no correct, efficient algorithm exists for solving a given problem
Writing clear algorithms and proofs

11. Classic Algorithms
Lots of wonderful algorithms have already been developed
I expect you to learn most of this from reading, though we will reinforce in lecture

12. Algorithm design methods
Something of an art form
Cannot be fully automated
We will describe some general techniques and try to illustrate when each is appropriate

13. Algorithm correctness
Proving an algorithm generates correct output for all inputs
One technique covered in textbook
Loop invariants
We will do some of this in the course, but it is not emphasized as much as other objectives

14. Analyzing algorithms
The “process” of determining how much resources (time, space) are used by a given algorithm
We want to be able to make quantitative assessments about the value (goodness) of one algorithm compared to another
We want to do this WITHOUT implementing and running an executable version of an algorithm

15. Proving hardness results
We believe that no correct and efficient algorithm exists that solves many problems such as TSP
We define a formal notion of a problem being hard
We develop techniques for proving hardness results

16. Clear Writing Methods for Expressing Algorithms
Implementations
Pseudo-code
English
Writing clear and understandable proofs
My main concern is not the specific language used but the clarity of your algorithm/proof
In this class, most algorithms will be expressed in plain English with pseudo-code used to clear up any ambiguities.

17. Outline Definitions Course Objectives Administrative stuff …
Algorithms
Problems
Course Objectives
Administrative stuff …
Analysis of Algorithms

18. Algorithm Analysis Overview
RAM model of computation
Concept of input size
Measuring complexity
Best-case, average-case, worst-case
Asymptotic analysis
Asymptotic notation

19. The RAM Model
RAM model represents a “generic” implementation of the algorithm
Each “simple” operation (+, -, =, if, call) takes exactly 1 step.
Loops and subroutine calls are not simple operations, but depend upon the size of the data and the contents of a subroutine. We do not want “sort” to be a single step operation.
Each memory access takes exactly 1 step.
Basically, any basic operation that takes a small, fixed amount of time we assume to take just one step.
We measure the run time of an algorithm by counting the number of steps it takes.
Why does this work? For the same reason that the “Flat Earth” model works. In our day-to-day lives we assume the Earth to be flat!
Now, lets look at how we can use this.

20. Input Size
In general, larger input instances require more resources to process correctly
We standardize by defining a notion of size for an input instance
Examples
What is the size of a sorting input instance?
What is the size of an “Odd number” input instance?

21. Algorithm Analysis Overview
RAM model of computation
Concept of input size
Measuring complexity
Best-case, average-case, worst-case
Asymptotic analysis
Asymptotic notation

22. Measuring Complexity
The running time of an algorithm is the function defined by the number of steps (or amount of memory) required to solve input instances of size n
F(1) = 3
F(2) = 5
F(3) = 7 …
F(n) = 2n+1
Problem: Inputs of the same size may require different numbers of steps to solve

23. 3 different analyses
The worst case running time of an algorithm is the function defined by the maximum number of steps taken on any instance of size n.
The best case running time of an algorithm is the function defined by the minimum number of steps taken on any instance of size n.
The average-case running time of an algorithm is the function defined by an average number of steps taken on any instance of size n.
Which of these is the best to use?

24. Average case analysis Drawbacks
Based on a probability distribution of input instances
The distribution may not be appropriate
Provides little consolation if we have a worst-case input
More complicated to compute than worst case running time
Worst case running time is often comparable to average case running time (see next graph)
Counterexamples to above point:
Quicksort
simplex method for linear programming

25. Best, Worst, and Average Case

26. Worst case analysis
Typically much simpler to compute as we do not need to “average” performance on many inputs
Instead, we need to find and understand an input that causes worst case performance
Provides guarantee that is independent of any assumptions about the input
Often reasonably close to average case running time
The standard analysis performed

27. Algorithm Analysis Overview
RAM model of computation
Concept of input size
Measuring complexity
Best-case, average-case, worst-case
Asymptotic analysis
Asymptotic notation

28. Motivation for Asymptotic Analysis
An exact computation of worst-case running time can be difficult
Function may have many terms:
4n2 - 3n log n n - 43 n⅔ + 75
An exact computation of worst-case running time is unnecessary
Remember that we are already approximating running time by using RAM model

29. Simplifications Ignore constants Asymptotic Efficiency
4n2 - 3n log n n - 43 n⅔ + 75 becomes
n2 – n log n + n - n⅔ + 1
Asymptotic Efficiency
n2 – n log n + n - n⅔ + 1 becomes n2
End Result: Θ(n2)

30. Why ignore constants? RAM model introduces errors in constants
Do all instructions take equal time?
Specific implementation (hardware, code optimizations) can speed up an algorithm by constant factors
We want to understand how effective an algorithm is independent of these factors
Simplification of analysis
Much easier to analyze if we focus only on n2 rather than worrying about 3.7 n2 or 3.9 n2

31. Asymptotic Analysis
We focus on the infinite set of large n ignoring small values of n
Usually, an algorithm that is asymptotically more efficient will be the best choice for all but very small inputs. 

32. “Big Oh” Notation O(g(n)) =
{f(n) : there exist positive constants c and n0 such that " n≥n0, 0 ≤ f(n) ≤ c g(n) }
What are the roles of the two constants?
n0: c:  n0
f(n) ≤ c g(n)

33. Set Notation Comment O(g(n)) is a set of functions.
However, we will use one-way equalities like
n = O(n2)
This really means that function n belongs to the set of functions O(n2)
Incorrect notation: O(n2) = n
Analogy
“A dog is an animal” but not “an animal is a dog”

34. Three Common Sets
f(n) = O(g(n)) means c  g(n) is an Upper Bound on f(n)
f(n) = (g(n)) means c  g(n) is a Lower Bound on f(n)
f(n) = (g(n)) means c1  g(n) is an Upper Bound on f(n)
and c2  g(n) is a Lower Bound on f(n)
These bounds hold for all inputs beyond some threshold n0.
Asymptotic or Big-O notation.
O Omega Sigma

35. O(g(n))

36. (g(n))

37. (g(n))

38. O(f(n)) and (g(n))

39. Example Function
f(n) = 3n n + 6

40. Quick Questions c n0 3n2 - 100n + 6 = O(n2) 3n2 - 100n + 6 = O(n3)
[ Do on blackboard!!!!! ]
f(n) = 3n n + 6

41. Common Complexity Functions
n 110-5 sec 210-5 sec 310-5 sec 410-5 sec 510-5 sec 610-5 sec
n sec sec sec sec sec sec
n sec sec sec sec sec sec
n sec sec sec min min min
2n sec sec min days years cent
3n 0.59sec min years cent 2108cent 1013cent
log2 n 310-6 sec 410-6 sec 510-6 sec 510-6 sec 610-6 sec 610-6 sec
n log2 n 310-5 sec 910-5 sec sec sec sec sec

42. Example Problems 1. What does it mean if:
f(n)  O(g(n)) and g(n)  O(f(n)) ?
2. Is 2n+1 = O(2n) ?
Is 22n = O(2n) ?
3. Does f(n) = O(f(n)) ?
4. If f(n) = O(g(n)) and g(n) = O(h(n)),
can we say f(n) = O(h(n)) ?

43. Extra Slides Slides illustrating TSP algorithms
Case study with Insertion sort for Best, Average, Worst case analysis

44. Possible Algorithm: Nearest neighbor

45. Not Correct!

46. A Correct Algorithm
Try all possible orderings of the points selecting the ordering that minimizes the total length:
d = 
For each of the n! permutations, Pi of the n points,
if cost(Pi) < d then
d = cost(Pi)
Pmin = Pi
return Pmin

47. Case study: Insertion Sort
Count the number of times each line will be executed:
Num Exec.
for i = 2 to n
key = A[i]
j = i - 1
while j > 0 AND A[j] > key
A[j+1] = A[j]
j = j -1
A[j+1] = key
n-1 (or n) for all but while loop
varies for while loop