CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Analysis of Algorithms

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Analysis of Algorithms Coming up Asymptotic performance, Insertion Sort A formal introduction to asymptotic notation (Chap , Chap 3.1)

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Asymptotic performance “How does the algorithm behave as the problem size gets very large?” Running time Memory/storage requirements Bandwidth/power requirements/logic gates/etc. In analysis of algorithms, we care most about asymptotic performance

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Asymptotic performance Assume: an algorithm can solve a problem of size n in f(n) microseconds (10 -6 seconds). f(n)n=20n=40n=60 Log 2 n Sqrt(n) n n log 2 n n2n2 n4n4 2n2n n! 5.32 * sec4.32 * sec5.91 * sec 6.32 * sec4.47 * sec7.75 * sec 40 * sec20 * sec60 * sec 213 * sec86 * sec354 * sec 1600 * sec400 * sec3600 * sec 2.56 sec0.16 sec12.96 sec

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Asymptotic performance Assume: an algorithm can solve a problem of size n in f(n) microseconds (10 -6 seconds).

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Input Size Time and space complexity is generally a function of the input size E.g., sorting, multiplication How we characterize input size depends: Sorting: number of input items Multiplication: total number of bits Graph algorithms: number of nodes & edges

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Insertion Sort

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Insertion Sort To sort A[1..n] in place: Steps: Pick element A[j] Move A[j-1..1] to the right until proper position for A[j] is found. Currently sorted part Currently unsorted part j1…j+1..n

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Insertion Sort

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Correctness of Insertion Sort To prove Insertion Sort is correct, we state the loop invariant: “At start of each iteration of the for loop, A[1..j-1] consists of the elements originally in A[1..j-1] but in sorted order.” We observe 3 properties: Initialization: It is true prior to the first iteration of the loop Maintenance: If it is true before an iteration, it remains true before next iteration. Termination: When the loop terminates, the invariant gives a useful property that helps show the algorithm is correct. Can you state these properties for Insertion Sort?

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Correctness of Insertion Sort loop invariant “At start of each iteration of the for loop, A[1..j-1] consists of the elements originally in A[1..j-1] but in sorted order.” Initialization Before the first iteration, j=2. => A[1.. j-1] contains only A[1]. => Loop invariant holds prior to the first iteration. Maintenance The outer loop moves A[j-1],A[j-2],A[j-3].. to the right until the proper position for A[j] is founded. Then A[j] is inserted. => if the loop invariant is true before an iteration, it remains true before next iteration. Termination The outer loop ends with j=n+1. Substituting n+1 for j in the loop invariant, we get “A[1..n] consists of the n sorted elements.”

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Analyzing Insertion Sort times n n-1  j=2..n t j  j=2..n (t j -1) n-1 Cost c 1 c 2 0 c 4 c 5 c 6 c 7 c 8 The running time T(n) = c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n t j )+c 6 *(  j=2..n (t j -1))+c 7 *(  j=2..n (t j -1))+c 8 *(n-1) t j = no. of times that line 5 is executed, for each j.

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Analyzing Insertion Sort Noting :  j=2..n j = __________  j=2..n (j-1) = _______ T(n) = c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n t j )+ c 6 *(  j=2..n (t j -1))+c 7 *(  j=2..n (t j -1))+c 8 *(n-1) Worse case: Reverse sorted  inner loop body executed for all previous elements.  t j =j.  T(n) = c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n j)+ c 6 *(  j=2..n (j-1))+c 7 *(  j=2..n (j-1))+c 8 *(n-1)  T(n) = An 2 +Bn+C  T(n) = c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n j)+ c 6 *(  j=2..n (j-1))+c 7 *(  j=2..n (j-1))+c 8 *(n-1)

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Analyzing Insertion Sort T(n)=c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n t j )+c 6 *(  j=2..n (t j -1))+c 7 *(  j=2..n (t j -1))+c 8 *(n-1) Worst caseReverse sorted => inner loop body executed for all previous elements. So, t j =j. => T(n) is quadratic: T(n)=An 2 +Bn+C Average caseHalf elements in A[1..j-1] are less than A[j]. So, t j = j/2 => T(n) is also quadratic: T(n)=An 2 +Bn+C Best case Already sorted => inner loop body never executed. So, t j =1. => T(n) is linear: T(n)=An+B

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Kinds of Analysis (Usually) Worst case: T(n) = max time on any input of size n Knowing it gives us a guarantee about the upper bound. In some cases, worst case occurs fairly often Average case is often as bad as worst case. (Sometimes) Average case: T(n) = average time over all inputs of size n (Rarely) Best case: Cheat with slow algorithm that works fast on some input. Good only for showing bad lower bound.

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Random-Access Machine Analysis is performed with respect to a computational model We usually use a generic uniprocessor random-access machine (RAM) All memory equally expensive to access No concurrent operations All reasonable instructions take unit time (Except, of course, function calls) Constant word size

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Order of Growth  Ignore machine-dependent constants  Look at growth of T(n) as n ->   Drop low-order terms, Ignore leading constants Eg. worse case of insertion sort, T(n) = An 2 +Bn+C Order of Growth = n 2 “T(n) is in  (n 2 )” For convenience, we usually say “T(n) is  (n 2 )” and “T(n) =  (n 2 )” An algorithm is more efficient if its worst-case running time has a lower order of growth.  (n 2 ) is a set of functions that relates to n 2 in some way. (We’ll define later)

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Asymptotic Notations 3 major notations for describing algorithm complexities: Asymptotic Tight Bound:  Intuitively like “=” Asymptotic Upper Bound:  Intuitively like “  ” Asymptotic Lower Bound:  Intuitively like “  ” Other notations: o,  Intuitively like “ ” Eg., Insertion Sort’s worse case running time is  (n 2 ). Insertion Sort’s best case running time is  (n). Insertion Sort’s running time is  (n). Insertion Sort’s running time is O(n 2 )... running time is in  (n 2 ).

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Asymptotic Notations Very often the algorithm complexity can be observed directly from simple algorithms: The levels of nested loops. O(n 2 )  (n)

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Definitions of O, , and  O(g(n)) = {f(n): there exist positive constants c and n 0 such that 0  f(n)  cg(n) for all n  n 0 } f(n) c 1 g(n) c 2 g(n) n0n0 f(n) is  (g(n)) f(n) cg(n) n0n0 f(n) is O(g(n)) f(n) cg(n) n0n0 f(n) is  (g(n))  (g(n)) = {f(n): there exist positive constants c and n 0 such that 0  cg(n)  f(n) for all n  n 0 }  (g(n)) = {f(n): there exist positive constants c 1, c 2, n 0 such that 0  c 1 g(n)  f(n)  c 2 g(n) for all n  n 0 }  (g(n)) O(g(n)) and  (g(n))

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Example To do so, we must determine positive constants c 1, c 2, n 0 such that c 1 n 2  0.5n 2 -3n  c 2 n 2, for all n  n 0. Dividing by n 2 yields: c 1  /n  c 2. c 1  0.5-3/n holds for any value of n  7 by choosing c 1  1/ /n  c 2 holds for any value n  1 by choosing c 2  0.5 Hence by choosing c 1 =1/14, c 2 =0.5, and n 0 =7, we can verify that 0.5n 2 - 3n =  (n 2 ). According to the formal definition of , prove that 0.5n 2 - 3n =  (n 2 )

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Example (cont’d) We have shown that 0.5n 2 - 3n =  (n 2 ). Similar prove can be applied to show An 2 +Bn+C =  (n 2 ) and An+B =  (n). For insertion sort: Worse case running time: T(n) = An 2 +Bn+C => T(n) =  (n 2 ) Average case running time: T(n) = An 2 +Bn+C => T(n) =  (n 2 ) Best case running time: T(n) = An+B => T(n) =  (n) Recall that, informally, the  () term can be obtained by observing the highest order term and drop the leading constant.

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Example According to the formal definition of O, prove that 6n 3  O(n 2 ) Prove by contradiction: Suppose c, n 0 exist such that 6n 3  cn 2, for all n  n 0. => n  c 2 /6, which is impossible for arbitrarily large n, since c is a constant. => 6n 3  cn 2 is not correct => 6n 3  O(n 2 )

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Points to Note We can write 2n 2 +3n+1 = 2n 2 +  (n). f(n) = n 2 + O(n) means f(n) = n 2 + h(n) for some h(n)  O(n) O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n). Why? It is also correct to write n 2 =O(n 3 ), and n 2 =  (n). Proof?

CS3381 Des & Anal of Alg ( SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms Analysis of Algorithms Summary Algorithm, input, output, instance, “Correct Algorithm” (recall Introduction) Asymptotic Performance, Input Size Insertion Sort Proof: Insertion Sort is correct (Loop Invariant + 3 properties) Analysis of Insertion Sort, Worse/Average/Best case Order of Growth Asymptotic Notations