Sample Questions Task: sorting n elements How fast can you do it? –in worst case –on average How much more memory is needed? Data assumptions? –in memory or on disk? –nearly sorted? –duplicates retained? –.... Different algorithms for different conditions
How can we tell how well an algorithm works? 1. Experimentally, I.E. Run algorithm on data, report cpu time –depends on coding –depends on machine –depends on compiler – little generality + can do it + expensive run multiple times, show average performance guess relevant variables and vary them What’s relevant for sorting?
2. Theoretical Analysis Give upper and lower bounds for complexity + demonstrates an understanding of algorithm + shows relevant variables + shows importance of variables + independent of machine, language, coding - analysis is sometimes difficult - usually only works for simple algorithms - often doesn’t include important constants Same questions for time can be done for memory usage
Language of Analysis big O notation A function f(n) is O(g(n)) iff there are positive constants c and N such than f(n) N. Intuitively this says that f is eventually less than a constant times g. Examples –17n 2 +1000n+15 is O(n 2 ). generalize –10n+nlog 10 n is O(nlg(n)) where lg = log 2 It turns out that sorting is O(n log(n)). What about finding the maximum in an array? What about finding the k-th largest in an array?
Squeezing the complexity: lower bounds When upper bounds equal lower bounds, then you know you have the best possible algorithm. A function f(n) is big-omega(g(n)) iff there are positive constants c and N such that f(n)> c*g(n) for n > N. Intuitively, this says f is eventually larger than g. example: n 3 -1000n 2 is big-omega of n 3. Theoreticians work on closing the gap between the upper and lower bounds. It turns out that sorting is big-omega(n*log(n)), hence sorting is solved. –caveat: this assumes no special properties of the data. –sometimes constants count –sometimes eventually is not good enough
Useful Theorems Let f and g be function of n If f <= g, then O(f+g) = O(g). O(constant*f) = O(f). O(f*f) = [O(f)]^2. O(Polynomial(n) ) = O(n max degree ) O(f/g).. Can be anything. Style: write the simplest and smallest O-notation. –eg. do not write O(n 2 + 3n), but O(n^2). –Do not write O(3*n^3), but O(n^3).
Proving O theorems To prove: if f1 and f2 are O(g(n)) then f1*f2 is O(g(n)*g(n)). Proof: by your nose. –Ask: what do I need to do? By definition find a constant c and integer N such that f1(n)*f2(n) N –Ask: what do I know? There are constants c1,c2, N1,N2 such that f1(n) N1 f2(n) N2 –Aren’t c and N obvious now? Can you do this?
Another Proof To prove: if f1 and f2 are O(g(n)) then f1+f2 is O(g(n)). By your nose Since f1 is O(g(n)), f1(n) N1. Since f2 is O(g(n)), f2(n) N2. Then f1(n)+f2(n) max(N1,N2). Hint: go back to these proofs in a few days and see if you can prove them without looking.
More notation A function f(n) is big-theta of g(n) iff it is both big-O and big-omega of g(n). Intuitively, they have same growth rate. A function f(n) is little-o of g(n) iff it is big-O of g(n), but not big-omega of g(n). TSP is big-O(2 n ). Why? –No one knows if it is big-omega(2 n ). Intuitively this means there is some doubt. These results require proofs. One can also get estimates by running careful experiments. But experiments can’t prove anything.
Traveling Salesman Problem Given: n cities and distances between them Find: tour of minimum length. –A tour is path that visits every city How many tours are possible? –n*(n-1)...*1 = n! –n! > 2*2……*1 = 2^(n-1) –So n! is omega of 2^n. ( lower bound) No one knows if there is a polynomial time algorithm to find minimum length tour. There are many very good heuristic algorithms.
Boolean Satisfiability Suppose we have a boolean formula on n-variables –e.g. (x1 or not x2 or x3)&(x2 or not x5 or x7)&(...) This is a conjunct of disjuncts. If restricted to at most 3 literals per disjunct then 3-sat. Theorem: 3-Sat and general Boolean satisfiability are equivalently difficult. Does this have a solution? –i.e. assignment of true or false to the variables so that the entire formula is true. When the formula has at most 3 literals (variable or its negation) in each clause, the problem is called 3-sat. If we just checked every possibility, how many cases? –2 n, so exhaustive search is O(2 n )
Satisfiability, the value No one knows if this can be done in polynomial time. Nobel prize level problem, but none given in CS. One Million $ prize Application: Given 2 circuits, are they equivalent? –Suppose they each have n inputs. –Let the two circuits, or formulas, be f1 and f2. –Consider: XOR(f1, f2). –This is a boolean formula on n-variables. –If it has a solution S, then f1(S) != f2(S) and the two circuits are different.
A problem we can do: Linear Lookup/Search Task: determine if value is in array (n items) Algorithm For i = 1 to n if (a[i] = element) return true else return false In worst case, time O(n). In best case, O(1). What about space? O(1)
Space Analysis If program has a bad time complexity, you can wait. If program has a bad space complexity, the program bombs. SPACE is critical. Space utilization occurs in two places: –1. Every time you call new. Analysis is like time complexity –2. Hidden usage: function calls Depth of function calls is critical here. Not a problem except when using recursion.
Find most similar student records Assume we have n records Record[I] Each record has k grades Algorithm Sketch – for j = 1…n – for h = j+1…n – compute similarity of records j and h – if similarity better than best, save j,h and score end h –end j –return j and h. Time complexity: O(n^2) Space complexity: O(1).
Divide and Conquer General Problem Solving method Top-down division of labor General Algorithm: Solution = Solve(Problem) Solve(Problem) = if Problem easy, return basicSolve(Problem) else Pieces = split(Problem) PartSolutions =Solve(Pieces) return merge(PartSolutions) Examples: mergesort, quicksort, binary search
General analysis of Divide and Conquer Suppose splitting yields a subproblems Suppose size of subproblems is N/b Suppose merging cost is O(n^k) Then T(N) = if a>b^k O(N^(log b a)) if a = b^k O(N^k * log(N) ) if a < b^k O(N^k) For example, in mergesort where N = b^m. A=2, b=2, k = 1 Therefore: O(N*log(N)) You should know how to use this theorem (you don’t need to memorize result).
Binary Search Binary search (assumes ordered) BinarySearch(a, l, u) –mid = (l+u)/2 –if ( l==u ) return a[mid] == element; –if (element = a[mid]) return mid; –else if (element <a[mid]) return BinarySearch(a,l,mid); –else if (element >a[mid]) return BinarySearch(a,mid,u); Time Analysis –f(n) = c+f(n/2) –f(1) = c. –Hence: f(n) = c*lg(n). –Expand out to see this, with n a power of 2.
Binary Search Space Analysis –Only new memory counts –New memory occurs when constructors are called recursion occurs –Back to code: –What is max depth of stack? O(log n)
Maximum contiguous subsequence problem Given a…a[n] are integers, find i, j such that sum of a[k] from i to j is maximum. Algorithmic Approaches –Exhaustive, dumb but a start optimize via problem constraints –Divide and Conquer requires breaking problem in pieces whose solutions can be merged –Dynamic Programming requires merging subsolutions of smaller problems into full solution –Be Clever: this is for the theoreticians –Transform a known problem solution
Exhaustive Best = 0, begin = 0, end= 0 For i = 1…n for j = i+1 … n temp = 0 for k = i …j temp += a[k] if (temp > best) best = temp update begin, end Time complexity: O(N^3) Space complexity: O(1) see text for Java code
Partition trick For i = 1…n/3 > n/3 times for j = 2n/3 … n > n/3 times temp = 0 for k = i …j > n/3 temp += a[k] if (temp > best) best = temp update begin, end Time complexity: O(N^3) N^3/9 More painful analysis in the text.
Optimize (exhaustive) Best = 0, begin = 0, end= 0 For i = 1…n temp = 0 for j = i+1 … n temp += a[k] if (temp > best) best = temp update begin, end Time: O(n^2) Space: O(1)
Dynamic Programming General Idea: use results of smaller problem Decomposition: finding relationship between problem and smaller ones Idea Compute S[i][j] = sum a[i]+… a[j] Keep track of best Note: S[i][i] = a[i], boundary condition S[i][k] = S[i][k-1] + a[k] See Matrix Time: O(n^2) Space: O(n^2)
Fibonacci via DP Definition of f(k) –if (k <2) return 1 else return f(k-1)+f(k-2 Expand: f(4) = f(3) + f(2) = f(2)+f(1) + f(1)+f(0) = f(1)+f(0)+ f(1)+f(1)+f(0) Redundant computation: exponential! (see text) Dynnamic programming: bottom-up instead of top- down. StraightForward: use array: f(k) depends on two previous values. Optimise: don’t really need array, just last 2 values. T(N) = T(N-1)+T(N-2) +2
Dynamic Programming (example 2) Define Comb(k, n) = n! / (k!*(n-k)!) Why doesn’t this compute well? Goal: find definition in terms of smaller k,n Note: Comb(k,n) = Comb(k-1, n-1)+ Comb(k,n-1) Ok, so dynamic programming will work where S[k][n] is determine by element above and to left time analysis: O(n^2) space analysis: O(n^2) Why is this useful?
Divide and Conquer Previous examples –MergeSort, QuickSort, BinarySearch Divide array in half, say left and right Suppose solutions to left and right How to combine? –From each half need two solutions interior solution (away from boundary) boundary solution –Solution is best of: interior left, interior right, sum of (boundary solutions) See text for code.
Clever Look at properties of answer Note: if a[i]+…+a[j] is a solution then a[i]+…+a[p] must be positive. Best = 0, temp = 0 For j = 1…n temp += a[j] if (temp > best) best = temp else if (temp <0) temp = 0 Time: O(n)
Transform Problem Problem: what is minimum subsequence sum? –Replace each a[i] by -a[i]. Done. Problem. Assume each ai is positive. What is maximum product? –Replace each a[i] by log(a[i]). – Note xy > uv iff log(xy) >log(uv) –But log(xy) = log(x)+ log(y) Transforms not always easy to find.
Summary Memory and time complexity measures Critical for some applications Sometimes the constants count Be aware of your computational expenses Document non-O(1) memory or time costs Consider problem decomposition, either dynamic programming or divide and conquer