1. Theory of Algorithms: Fundamentals of the Analysis of Algorithm Efficiency James Gain and Edwin Blake {jgain | edwin} @cs.uct.ac.za Department of Computer Science University of Cape Town August - October 2004

2. Objectives lTo outline a general Analytic Framework lTo introduce the conceptual tools of:  Asymptotic Notations  Base Efficiency Classes lTo cover Techniques for the Analysis of Algorithms:  Mathematical (non-recursive and recursive)  Empirical  Visualisation

3. Analysis of Algorithms lIssues:  Correctness (Is it guaranteed to produce a correct correspondence between inputs and outputs?)  Time efficiency (How fast does it run?)  Space efficiency (How much extra space does it require?)  Optimality (Is this provably the best possible solution?) lApproaches:  Theoretical analysis  Empirical analysis  Visualisation

4. Theoretical analysis of time efficiency lTime efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input size lBasic operation: the operation that contributes most towards the running time of the algorithm. l T(n) ≈ c op C(n) running time execution time for basic operation Number of times basic operation is executed input size

5. Input size and basic operations Problem Input size measure Basic operation Search for key in list of n items Number of items in list n Key comparison Multiply two matrices of floating point numbers Dimensions of matrices Floating point multiplication Compute a n n Floating point multiplication Graph problem #vertices and/or edges Visiting a vertex or traversing an edge

6. Best-, Average- and Worst-cases lFor some algorithms efficiency depends on type of input: lWorst case: W(n) – max over inputs of size n lBest case: B(n) – min over inputs of size n lAverage case: A(n) – “avg” over inputs of size n  Number of times the basic operation will be executed on typical input  NOT the average of worst and best case  Expected number of basic operations repetitions considered as a random variable under some assumption about the probability distribution of all possible inputs of size n

7. Amortized Efficiency lFrom a data structures perspective the total time of a sequence of operations is important lReal-time apps are an exception lAmortized efficiency: time of an operation averaged over a worst- case sequence of operations lDesire “self-organizing” data structures that adapt to the context of use, e.g. red-black trees

8. Exercise: Sequential search lProblem: Given a list of n elements and a search key K, find an element equal to K, if any. lAlgorithm: Scan the list and compare its successive elements with K until either a matching element is found (successful search) of the list is exhausted (unsuccessful search) lCalculate the:  Worst case?  Best case?  Average case?

9. Types of Formulas for Operation Counts lExact formula  e.g., C(n) = n(n-1)/2 lFormula indicating order of growth with specific multiplicative constant  e.g., C(n) ≈ 0.5 n 2 lFormula indicating order of growth with unknown multiplicative constant  e.g., C(n) ≈ cn 2

10. Order of Growth lMost important: Order of growth within a constant multiple as n  ∞ lExample: How much faster will algorithm run on computer that is twice as fast? How much longer does it take to solve problem of double input size?

11. Asymptotic Notations lA way of comparing functions that ignores constant factors and small input sizes lO(g(n)):  class of functions f(n) that grow no faster than g(n)  f(n) ≤ c g(n) for all n ≥ n 0 lΘ(g(n)):  class of functions f(n) that grow at same rate as g(n)  c 2 g(n) ≤ f(n) ≤ c 1 g(n) for all n ≥ n 0 l Ω (g(n)):  class of functions f(n) that grow at least as fast as g(n)  f(n) ≥ c g(n) for all n ≥ n 0

12. Big-Oh: t(n)  O(g(n))

13. Big-Omega: t(n)  Ω(g(n))

14. Big-Theta: t(n)  Θ(g(n))

15. Establishing Rate of Growth: using Limits llim n→∞ t(n)/g(n) =  0 implies order of t(n) < order of g(n)  c implies order of t(n) = order of g(n)  ∞ implies order of t(n) > order of g(n) Example:  10n vs. 2n 2  lim n→∞ 10n/2n 2 = 5 lim n→∞ 1/n = 0 lExercises:  n(n+1)/2 vs. n 2  log b n vs. log c n

16. L’Hôpital’s rule lIf lim n→∞ f(n) = lim n → ∞ g(n) = ∞ and the derivatives f ', g' exist, Then lim n→∞ f(n) / g(n) = lim n → ∞ f '(n) / g'(n) lExample:  log n vs. n  lim n→∞ log n / n = lim n→∞ 1/n log e = log e lim n→∞ 1/n = 0 log n < order n  So, order log n < order n

17. Establishing Rate of Growth: using Definitions lf(n) is O(g(n)) if order of growth of f(n) ≤ order of growth of g(n) (within constant multiple) lThere exist positive constant c and non-negative integer n 0 such that  f(n) ≤ c g(n) for every n ≥ n 0 lThis needs to be mathematically provable lExamples:  10n is O(2n 2 ) because 10n 5  5n+20 is O(10n) because 5n+20 4

18. Basic Asymptotic Efficiency Classes 1constantOutside best-case, few examples log nlogarithmicAlgorithms that decrease by a constant nlinearAlgorithms that scan an n-sized list n log n Algorithms that divide and conquer, e.g. quicksort n2n2 quadraticTypically two embedded loops n3n3 cubicTypically three embedded loops 2n2n exponentialAlgorithms that generate all subsets of an n-element list n!factorialAlgorithms that generate all permutations of an n-element list

19. Time Efficiency of Non-recursive Algorithms lSteps in mathematical analysis of nonrecursive algorithms: 1.Decide on parameter n indicating input size 2.Identify algorithm’s basic operation 3.Determine worst, average, and best case for input of size n 4.Set up summation for C(n) reflecting algorithm’s loop structure 5.Simplify summation using standard formulas (see Appendix A of textbook)

20. Examples: Analysing non-recursive algorithms lMatrix multiplication  Multiply two square matrices of order n using dot product of rows and columns lSelection sort  Find smallest element in remainder of list and swap with current element lInsertion sort  Assume sub-list is sorted an insert current element lMystery Algorithm  Exercise

21. Matrix Multiplication 1.n = matrix order 2.Basic op = multiplication or addition 3.Best case = worst case = average case 4.

22. Selection Sort ln = number of list elements lbasic op = comparison lbest case = worst case = average case l

23. Insertion Sort ln = number of list elements lbasic op = comparision lbest case != worst case != average case lbest case: A[j] > v executed only once on each iteration

24. Exercise: Mystery Algorithm lWhat does this algorithm compute? lWhat is its basic operation? lWhat is its efficiency class? lSuggest an improvement and analyse your improvement // Input: A matrix A[0..n-1, 0..n-1] of real numbers for i  0 to n - 1 do for j  i to n-1 do if A[i,j]  A[j,i] return false return true

25. Factorial: a Recursive Function lRecursive Evaluation of n! lDefinition:  n ! = 1*2*…*(n-1)*n lRecursive form for n!:  F(n) = F(n-1) * n, F(0) = 1 lAlgorithm:  IF n=0 THEN F(n)  1 ELSE F(n)  F(n-1) * n RETURN F(n) lNeed to find the number of multiplications M(n) to compute n!

26. Recurrence Relations lDefinition:  An equation or inequality that describes a function in terms of its value on smaller inputs lRecurrence:  The recursive step  E.g., M(n) = M(n-1) + 1 for n > 0 lInitial Condition:  The terminating step  E.g., M(0) = 0 [call stops when n = 0, no mults when n = 0] lMust differentiate the recursive function (factorial calculation) from the recurrence relation (number of basic operations)

27. Recurrence Solution for n! lNeed an analytic solution for M(n) lSolved by Method of Backward Substitution: M(n) = M(n-1) + 1 Substitute M(n-1) = M(n-2) + 1 = [M(n-2) + 1] +1 = M(n-2) + 2 Substitute M(n-2) = M(n-3) + 1 = [M(n-3) + 1] + 2 = M(n-3) + 3 lPattern: M(n) = M(n-i) + i lUltimately: M(n) = M(n-n)+n = M(0) + n = n

28. Plan for Analysis of Recursive Algorithms lSteps in mathematical analysis of recursive algorithms: 1.Decide on parameter n indicating input size 2.Identify algorithm’s basic operation 3.Determine worst, average, and best case for input of size n 4.(a) Set up a recurrence relation and initial condition(s) for C(n)-the number of times the basic operation will be executed for an input of size n OR (b) Alternatively, count recursive calls 5.(a) Solve the recurrence to obtain a closed form OR (b) Estimate the order of magnitude of the solution (see Appendix B)

29. Iterative Methods for Solving Recurrences lMethod of Forward Substitution:  Starting from the initial condition generate the first few terms  Look for a pattern expressible as a closed formula  Check validity by direct substitution or induction  Limited because pattern is hard to spot lMethod of Backward Substitution:  Express x(n-1) successively as a function of x(n-2), x(n-3), …  Derive x(n) as a function of x(n-i)  Substitute n-i = base condition  Surprisingly successful

30. Templates for Solving Recurrences lDecrease by One Recurrence lOne (constant) operation reduces problem size by one  Examples: Factorial  T(n) = T(n-1) + c T(1) = d  Solution: T(n) = (n-1)c + d  Linear Order lA pass through input reduces problem size by one  Examples: Insertion Sort  T(n) = T(n-1) + cn T(1) = d  Solution: T(n) = [n(n+1)/2 – 1] c + d  Quadratic Order

31. More Templates for Solving Recurrences lDecrease by a Constant Factor lOne (constant) operation reduces problem size by half  Example: binary search  T(n) = T(n/2) + c T(1) = d  Solution: T(n) = c lg n + d  Logarithmic Order lA pass through input reduces problem size by half  Example: Merge Sort  T(n) = 2T(n/2) + cn T(1) = d  Solution: T(n) = cn lg n + d n  n log n order

32. Master Method for Solving Recurrences lDivide and Conquer Style Recurrence lT(n) = aT(n/b) + f (n) where  f (n)  .(n k ) is the time spent in dividing and merging  a >= 1, b >= 2 1.a < b k T(n)  .(n k ) 2.a = b k T(n)  .(n k lg n ) 3.a > b k T(n)  .( n log b a ) lNote: the same results hold with O instead of .

33. Example: Fibonacci Numbers lThe Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, … lDescribes the growth pattern of Rabbits. Just short of exponential, ask Australia! lFibonacci recurrence: F(n) = F(n-1) + F(n-2) F(0) = 0 F(1) = 1 lAnother example: A(n) = 3A(n-1) - 2A(n-2) A(0) = 1 A(1) = 3 2nd order linear homogeneous recurrence relation with constant coefficients

34. Computing Fibonnaci Numbers Algorithm Alternatives: 1.Definition based recursive algorithm 2.Nonrecursive brute-force algorithm 3.Explicit formula algorithm F(n) = 1/√5  n where  is the golden ratio (1 + √5) / 2 4.Logarithmic algorithm based on formula: F(n-1) F(n) F(n) F(n+1) 0 1 1 = n for n≥1, assuming an efficient way of computing matrix powers.

35. Empirical Analysis of Time Efficiency lSometimes a mathematical analysis is difficult for even simple algorithms (limited applicability) lAlternative is to measure algorithm execution lPLAN: 1.Understand the experiment’s purpose  Are we checking the accuracy of a theoretical result, comparing efficiency of different algorithms or implementations, hypothesising the efficiency class? 2.Decide on an efficiency measure  Use physical units of time (e.g., milliseconds) OR  Count actual number of basic operations

36. Plan for Empirical Analysis 3.Decide on Characteristics of the Input Sample  Can choose Pseudorandom inputs, Patterned inputs or a Combination  Certain problems already have benchmark inputs 4.Implement the Algorithm 5.Generate a Sample Set of Inputs 6.Run the Algorithm and Record the Empirical Results 7.Analyze the Data  Regression Analysis is often used to fit a function to the scatterplot of results

37. Algorithm Visualisation lDefinition:  Algorithm Visualisation seeks to convey useful algorithm information through the use of images lFlavours:  Static algorithm visualisation  Algorithm animation lSome new insights: e.g., odd and even disks in Towers of Hanoi lAttributes of Information Visualisation apply - zoom and filter, detail on demand, etc. lWebsource:  The Complete Collection of Algorithm Animation  http://www.cs.hope.edu/~alganim/ccaa/ http://www.cs.hope.edu/~alganim/ccaa/