1. Analysis of Algorithms Asymptotic Performance

2. Review: Asymptotic Performance Asymptotic performance: How does algorithm behave as the problem size gets very large? oRunning time oMemory/storage requirements  Remember that we use the Hypothetical machine (Random Access Machine): oAll memory equally expensive to access oNo concurrent operations oAll reasonable instructions take unit time (c i )

3. Review: Running Time Number of primitive steps that are executed  Except for time of executing a function call most statements roughly require the same amount of time  We can be more exact if need be

4. Example int a, b, larger;c1 scanf(“%d %d”, &a, &b);c2 larger = b;c3 if (a > b)c4 larger = a;c5 printf(“Larger number is %d\n”, larger);c6 Running time T=c1+c2+c3+c4+c5+c6=C (constant)

5. Sum of first N natural numbers void main() {c1 int N, count, sum;c2 scanf (“%d”, &N) ;c3 sum = 0;c4 count = 1;c5 while (count <= N) {c6 sum = sum + count;c7 count = count + 1;c8 } printf (“Sum = %d\n”, sum) ;c9 } T(n)=c1+c2+c3+c4+c5+c6*(n+1)+c7*n+c8*n+c9 =C’+c6*(n+1)+c7*n+c8*n

6. 2-D Figure: with for loop Print * * * * *.... for (row=1; row<=n; ++row) { (n+1) for (col=1; col<=n; ++col) { n*(n+1) printf(“* ”); n*n } printf(“\n”); n }

7. Problem A Algorithm 1Algorithm 2 T1(n)=n+n  n+…… …. T2(n)=n+n 2 +6/n…… …. Which algorithm is better to solve problem A?

8. T(n)=> unknown function Map this to a known function g(n) Behavior of g(n) is known to us

9. Known function g(n) n^3 2^n n^2 nlogn n logn Growth function g(n)

10. Upper Bound Notation Read O as “Big-O” (you’ll also hear it as “order”) In general a function  f(n) is O(g(n)) if there exist positive constants c and n 0 such that f(n)  c  g(n) for all n  n 0 Formally  O(g(n)) = { f(n):  positive constants c and n 0 such that f(n)  c  g(n)  n  n 0

11. 11 f(n) = O( g(n)) n n0n0 f(n) cg(n) There exist positive constants c such that there is a positive constant n 0 such that …

12. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } }

13. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i =  j =  key =  A[j] =  A[j+1] = 

14. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30104020 1234 i = 2j = 1key = 10 A[j] = 30 A[j+1] = 10

15. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

16. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

17. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 30

18. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 30 4020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 30

19. David Luebke 19 1/13/2016 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 2j = 0key = 10 A[j] =  A[j+1] = 10

20. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 0key = 10 A[j] =  A[j+1] = 10

21. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 0key = 40 A[j] =  A[j+1] = 10

22. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 0key = 40 A[j] =  A[j+1] = 10

23. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

24. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

25. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

26. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 2key = 40 A[j] = 30 A[j+1] = 40

27. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

28. David Luebke 28 1/13/2016 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

29. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

30. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10304020 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

31. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

32. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

33. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

34. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

35. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 103040 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

36. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

37. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

38. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

39. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 1030 40 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

40. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10203040 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20

41. An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } 10203040 1234 i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20 Done!

42. Analysis of Insertion Sort INSERTION-SORT(A) cost times 1. for j = 2 to length[A] c 1 n 2. do key  A[j]c 2 n-1 3. //insert A[j] to sorted sequence A[1..j-1] 0n-1 4. i  j-1 c 4 n-1 5. while i >0 and A[i]>key c 5  j=2 n t j 6. do A[i+1]  A[i] c 6  j=2 n (t j –1) 7. i  i-1 c 7  j=2 n (t j –1) 8. A[i+1]  key c 8 n –1 (t j is the number of times the while loop test in line 5 is executed for that value of j) The total time cost T(n) = sum of cost  times in each line =c 1 n + c 2 (n-1) + c 4 (n-1) + c 5  j=2 n t j + c 6  j=2 n (t j -1)+ c 7  j=2 n (t j -1)+ c 8 (n-1)

43. Analyzing Insertion Sort What can T be?  Best case -- inner loop body never executed  Worst case -- inner loop body executed for all previous elements  Average case o???

44. 44 Analysis of Insertion Sort (cont.) Best case cost: already ordered numbers  t j =1, and line 6 and 7 will be executed 0 times  T(n) = c 1 n + c 2 (n-1) + c 4 (n-1) + c 5 (n-1) + c 8 (n-1) =(c 1 + c 2 + c 4 + c 5 + c 8 )n – (c 2 + c 4 + c 5 + c 8 ) = cn + c‘ Worst case cost: reverse ordered numbers  t j =j,  so  j=2 n t j =  j=2 n j =n(n+1)/2-1, and  j=2 n (t j –1) =  j=2 n (j –1) = n(n-1)/2, and  T(n) = c 1 n + c 2 (n-1) + c 4 (n-1) + c 5 (n(n+1)/2 -1) + + c 6 (n(n-1)/2 -1) + c 7 (n(n-1)/2)+ c 8 (n-1) =((c 5 + c 6 + c 7 )/2)n 2 +(c 1 + c 2 + c 4 +c 5 /2-c 6 /2- c 7 /2+c 8 )n-(c 2 + c 4 + c 5 + c 8 ) =an 2 +bn+c Average case cost: random numbers  i n average, t j = j/2. T(n) will still be in the order of n 2, same as the worst case.

45. Insertion Sort Is O(n 2 ) Proof  Suppose runtime is an 2 + bn + c oIf any of a, b, and c are less than 0 replace the constant with its absolute value  an 2 + bn + c  (a + b + c)n 2 + (a + b + c)n + (a + b + c)   3(a + b + c)n 2 for n  1  Let c’ = 3(a + b + c) and let n 0 = 1

46. Big O Fact A polynomial of degree k is O(n k ) Proof:  Suppose f(n) = b k n k + b k-1 n k-1 + … + b 1 n + b 0 oLet a i = | b i |  f(n)  a k n k + a k-1 n k-1 + … + a 1 n + a 0

47. 1. g(n) = n and f(n) = 2n + 3 f(n)=O(n)? c=3, n 0 =3 2. 100n = O(n 2 )? c=1, n 0 =100 3. n 2 =O(100n)? n 2 <c*100n 100c>n can not fix c!!!

48. Linear search // Searches an unordered array of integers int search(int data[], int n, int value){ / for(int index = 0; index < n; index++){ if(data[index] == value) return index; } return -1; } T(n)=O(n)

49. Bubble sort void bubbleSort ( int A[], int n ) { for (i=0; i<n; i++) { for (j=i+1; j<n; ++j) [n, n-1,….,2] { if (A[j] > A[j+1]) [n-1, n-2,….,1] { t = A[j]; A[j] = A[j+1]; A[j+1] = t; } T(n)=c1[n+(n-1)+(n-2)+…2]+c2[(n-1)+(n- 2)..+1]=an 2 +bn+c=O(n 2 )

50. Binary Search 82134657109111214130 64141325335143538472939597966 lo hi mid Ex. Binary search for 33.

51. Binary Search 82134657109111214130 64141325335143538472939597966 lo hi Ex. Binary search for 33.

52. Binary Search 82134657109111214130 64141325335143538472939597966 lo midhi Ex. Binary search for 33.

53. Binary Search 82134657109111214130 64141325335143538472939597966 lohi

54. Binary Search 82134657109111214130 64141325335143538472939597966 lohimid

55. Binary Search 82134657109111214130 64141325335143538472939597966 lo hi

56. Binary Search 82134657109111214130 64141325335143538472939597966 lo hi mid

57. Binary Search 82134657109111214130 64141325335143538472939597966 lo hi mid

58. Non recursive int binSearch ( int A[], int n, int x ) { int L, R, M; L = 0; R = n-1; c1 while (L < R) (k+1)*c2 { M = (L + R) / 2; c3*k if (x > A[M]) c4*k L = M+1; c5*k else R = M; c6*k } return (A[L] == x); c7 } n=2 k T(n)=c1+(k+1)*c2+(c3+c4+c5+c6)*k+c7 =k*C’+C’’=C’*log n+C’’=O(log n)

59. Recursive function Running time can be expressed as the recurrence relation First compute the recurrence relation Next, solve the recurrence relation using substitution method

60. Recursive int binary(int a[],int n,int m,int l,int u){ int mid,c=0; if(l<=u){ mid=(l+u)/2; if(m==a[mid]){ c=1; } else if(m<a[mid]){ return binary(a,n,m,l,mid-1); } else return binary(a,n,m,mid+1,u); } else return c; } T(n)=T(n/2)+c Recurrence relation

61. Solving recurrences  Substitution method T(n/2)=T(n/4)+c T(n/4)=T(n/8)+c ………….. T(n/2 (k-1) )=T(n/2 k )+c=T(1)+c T(n)=T(n/2)+c n=2 k T(n)=c+c+c…….+c + T(1) =kc+c1=c*log(n)+c1=O(log(n)) k times Recursive Recurrence relation

62. Factorial int factorialIter ( int n ) cost times { int prod, i; c1 1 if (n <= 1) c2 1 return 1; c3 1 prod = 1; c4 1 for (i=2; i<=n; ++i) c5 n prod *= i; c6 n-1 return prod; c7 1 } T(n)=c1+c2+..+n*c5+(n-1)*c6+c7=an+b=O(n)

63. Recursive version int factorialRec ( int n ) { int x; if (n <= 1) return 1; x=n * factorialRec(n-1); return x } T(0)=C1; T(1)=C1 T(n)= C2+T(n-1) n>1 Recurrence relation

64. Solving recurrences T(n)= C2+T(n-1) T(n-1)=C2+T(n-2) T(n)=C2+C2+T(n-2) T(n-2)=C2+T(n-3) T(n)=C2+C2+T(n-2) =C2+C2+C2+T(n-3) After n-1 number of steps T(n)=(n-1)C2+T(1) =(n-1)C2+C1 =an+b =O(n)

65. Mergesort

66. Basic Idea Divide the array into two halves Sort the two sub-arrays Merge the two sorted sub-arrays into a single sorted array Step 2 (sorting the sub-arrays) is done recursively (divide in two, sort, merge) until the array has a single element (base condition of recursion)

67. 67 Merging Two Sorted Arrays 34789 257 2 34789 257 23 34789 257 234 34789 257 Problem : Two sorted arrays A and B are given. We are required to produce a final sorted array C which contains all elements of A and B.

68. 68 2345 34789 257 23457 34789 257 234577 34789 257 2345778 34789 257

69. 69 Merge Code 23457789 34789 257 void merge (int A[], int B[], int C[], int m,int n) { int i=0,j=0,k=0; while (i<m && j<n) { if (A[i] < B[j]) C[k++] = A[i++]; else C[k++] = B[j++]; } while (i<m) C[k++] = A[i++]; while (j<n) C[k++] = B[j++]; }

70. 70 Merge Sort: Sorting an array recursively void mergesort (int A[], int n) { int i, j, B[max]; if (n <= 1) return; i = n/2; mergesort(A, i); mergesort(A+i, n-i); merge(A, A+i, B, i, n-i); for (j=0; j<n; j++) A[j] = B[j]; free(B); } T(1) = C1, T(n) = 2 T(n/2) + cn + d Recurrence relation

71. Solving recurrences T(n) = 2 T(n/2) + cn + d T(n/2)=2T(n/4)+cn/2+d T(n)=2(2T(n/4)+cn/2+d)+cn+d =4T(n/4)+cn+2d+cn+d =4T(n/4)+2cn+3d After k steps T(n)=2 k T(n/2 k ) + kcn+C’d =nT(1)+nlog(n)+C2’ =C1*n+nlog(n)+C2’=O(nlogn) n=2 k