Analysis of Algorithms Asymptotic Performance

Review: Asymptotic Performance Asymptotic performance: How does algorithm behave as the problem size gets very large? oRunning time oMemory/storage requirements  Remember that we use the Hypothetical machine (Random Access Machine): oAll memory equally expensive to access oNo concurrent operations oAll reasonable instructions take unit time (c i )

Review: Running Time Number of primitive steps that are executed  Except for time of executing a function call most statements roughly require the same amount of time  We can be more exact if need be

Example int a, b, larger;c1 scanf(“%d %d”, &a, &b);c2 larger = b;c3 if (a > b)c4 larger = a;c5 printf(“Larger number is %d\n”, larger);c6 Running time T=c1+c2+c3+c4+c5+c6=C (constant)

Sum of first N natural numbers void main() {c1 int N, count, sum;c2 scanf (“%d”, &N) ;c3 sum = 0;c4 count = 1;c5 while (count <= N) {c6 sum = sum + count;c7 count = count + 1;c8 } printf (“Sum = %d\n”, sum) ;c9 } T(n)=c1+c2+c3+c4+c5+c6*(n+1)+c7*n+c8*n+c9 =C’+c6*(n+1)+c7*n+c8*n

2-D Figure: with for loop Print * * * * *.... for (row=1; row<=n; ++row) { (n+1) for (col=1; col<=n; ++col) { n*(n+1) printf(“* ”); n*n } printf(“\n”); n }

Problem A Algorithm 1Algorithm 2 T1(n)=n+n  n+…… …. T2(n)=n+n 2 +6/n…… …. Which algorithm is better to solve problem A?

T(n)=> unknown function Map this to a known function g(n) Behavior of g(n) is known to us

Known function g(n) n^3 2^n n^2 nlogn n logn Growth function g(n)

Upper Bound Notation Read O as “Big-O” (you’ll also hear it as “order”) In general a function  f(n) is O(g(n)) if there exist positive constants c and n 0 such that f(n)  c  g(n) for all n  n 0 Formally  O(g(n)) = { f(n):  positive constants c and n 0 such that f(n)  c  g(n)  n  n 0

11 f(n) = O( g(n)) n n0n0 f(n) cg(n) There exist positive constants c such that there is a positive constant n 0 such that …

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } }

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i =  j =  key =  A[j] =  A[j+1] = 

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 1key = 10 A[j] = 30 A[j+1] = 10

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 0key = 10 A[j] =  A[j+1] = 30

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 0key = 10 A[j] =  A[j+1] = 30

David Luebke 19 1/13/2016 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 2j = 0key = 10 A[j] =  A[j+1] = 10

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 0key = 10 A[j] =  A[j+1] = 10

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 0key = 40 A[j] =  A[j+1] = 10

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 0key = 40 A[j] =  A[j+1] = 10

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 40 A[j] = 30 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

David Luebke 28 1/13/2016 An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20

An Example: Insertion Sort InsertionSort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20 Done!

Analysis of Insertion Sort INSERTION-SORT(A) cost times 1. for j = 2 to length[A] c 1 n 2. do key  A[j]c 2 n-1 3. //insert A[j] to sorted sequence A[1..j-1] 0n-1 4. i  j-1 c 4 n-1 5. while i >0 and A[i]>key c 5  j=2 n t j 6. do A[i+1]  A[i] c 6  j=2 n (t j –1) 7. i  i-1 c 7  j=2 n (t j –1) 8. A[i+1]  key c 8 n –1 (t j is the number of times the while loop test in line 5 is executed for that value of j) The total time cost T(n) = sum of cost  times in each line =c 1 n + c 2 (n-1) + c 4 (n-1) + c 5  j=2 n t j + c 6  j=2 n (t j -1)+ c 7  j=2 n (t j -1)+ c 8 (n-1)

Analyzing Insertion Sort What can T be?  Best case -- inner loop body never executed  Worst case -- inner loop body executed for all previous elements  Average case o???

44 Analysis of Insertion Sort (cont.) Best case cost: already ordered numbers  t j =1, and line 6 and 7 will be executed 0 times  T(n) = c 1 n + c 2 (n-1) + c 4 (n-1) + c 5 (n-1) + c 8 (n-1) =(c 1 + c 2 + c 4 + c 5 + c 8 )n – (c 2 + c 4 + c 5 + c 8 ) = cn + c‘ Worst case cost: reverse ordered numbers  t j =j,  so  j=2 n t j =  j=2 n j =n(n+1)/2-1, and  j=2 n (t j –1) =  j=2 n (j –1) = n(n-1)/2, and  T(n) = c 1 n + c 2 (n-1) + c 4 (n-1) + c 5 (n(n+1)/2 -1) + + c 6 (n(n-1)/2 -1) + c 7 (n(n-1)/2)+ c 8 (n-1) =((c 5 + c 6 + c 7 )/2)n 2 +(c 1 + c 2 + c 4 +c 5 /2-c 6 /2- c 7 /2+c 8 )n-(c 2 + c 4 + c 5 + c 8 ) =an 2 +bn+c Average case cost: random numbers  i n average, t j = j/2. T(n) will still be in the order of n 2, same as the worst case.

Insertion Sort Is O(n 2 ) Proof  Suppose runtime is an 2 + bn + c oIf any of a, b, and c are less than 0 replace the constant with its absolute value  an 2 + bn + c  (a + b + c)n 2 + (a + b + c)n + (a + b + c)   3(a + b + c)n 2 for n  1  Let c’ = 3(a + b + c) and let n 0 = 1

Big O Fact A polynomial of degree k is O(n k ) Proof:  Suppose f(n) = b k n k + b k-1 n k-1 + … + b 1 n + b 0 oLet a i = | b i |  f(n)  a k n k + a k-1 n k-1 + … + a 1 n + a 0

1. g(n) = n and f(n) = 2n + 3 f(n)=O(n)? c=3, n 0 = n = O(n 2 )? c=1, n 0 = n 2 =O(100n)? n 2 <c*100n 100c>n can not fix c!!!

Linear search // Searches an unordered array of integers int search(int data[], int n, int value){ / for(int index = 0; index < n; index++){ if(data[index] == value) return index; } return -1; } T(n)=O(n)

Bubble sort void bubbleSort ( int A[], int n ) { for (i=0; i<n; i++) { for (j=i+1; j<n; ++j) [n, n-1,….,2] { if (A[j] > A[j+1]) [n-1, n-2,….,1] { t = A[j]; A[j] = A[j+1]; A[j+1] = t; } T(n)=c1[n+(n-1)+(n-2)+…2]+c2[(n-1)+(n- 2)..+1]=an 2 +bn+c=O(n 2 )

Binary Search lo hi mid Ex. Binary search for 33.

Binary Search lo hi Ex. Binary search for 33.

Binary Search lo midhi Ex. Binary search for 33.

Binary Search lohi

Binary Search lohimid

Binary Search lo hi

Binary Search lo hi mid

Binary Search lo hi mid

Non recursive int binSearch ( int A[], int n, int x ) { int L, R, M; L = 0; R = n-1; c1 while (L < R) (k+1)*c2 { M = (L + R) / 2; c3*k if (x > A[M]) c4*k L = M+1; c5*k else R = M; c6*k } return (A[L] == x); c7 } n=2 k T(n)=c1+(k+1)*c2+(c3+c4+c5+c6)*k+c7 =k*C’+C’’=C’*log n+C’’=O(log n)

Recursive function Running time can be expressed as the recurrence relation First compute the recurrence relation Next, solve the recurrence relation using substitution method

Recursive int binary(int a[],int n,int m,int l,int u){ int mid,c=0; if(l<=u){ mid=(l+u)/2; if(m==a[mid]){ c=1; } else if(m<a[mid]){ return binary(a,n,m,l,mid-1); } else return binary(a,n,m,mid+1,u); } else return c; } T(n)=T(n/2)+c Recurrence relation

Solving recurrences  Substitution method T(n/2)=T(n/4)+c T(n/4)=T(n/8)+c ………….. T(n/2 (k-1) )=T(n/2 k )+c=T(1)+c T(n)=T(n/2)+c n=2 k T(n)=c+c+c…….+c + T(1) =kc+c1=c*log(n)+c1=O(log(n)) k times Recursive Recurrence relation

Factorial int factorialIter ( int n ) cost times { int prod, i; c1 1 if (n <= 1) c2 1 return 1; c3 1 prod = 1; c4 1 for (i=2; i<=n; ++i) c5 n prod *= i; c6 n-1 return prod; c7 1 } T(n)=c1+c2+..+n*c5+(n-1)*c6+c7=an+b=O(n)

Recursive version int factorialRec ( int n ) { int x; if (n <= 1) return 1; x=n * factorialRec(n-1); return x } T(0)=C1; T(1)=C1 T(n)= C2+T(n-1) n>1 Recurrence relation

Solving recurrences T(n)= C2+T(n-1) T(n-1)=C2+T(n-2) T(n)=C2+C2+T(n-2) T(n-2)=C2+T(n-3) T(n)=C2+C2+T(n-2) =C2+C2+C2+T(n-3) After n-1 number of steps T(n)=(n-1)C2+T(1) =(n-1)C2+C1 =an+b =O(n)

Mergesort

Basic Idea Divide the array into two halves Sort the two sub-arrays Merge the two sorted sub-arrays into a single sorted array Step 2 (sorting the sub-arrays) is done recursively (divide in two, sort, merge) until the array has a single element (base condition of recursion)

67 Merging Two Sorted Arrays Problem : Two sorted arrays A and B are given. We are required to produce a final sorted array C which contains all elements of A and B.

69 Merge Code void merge (int A[], int B[], int C[], int m,int n) { int i=0,j=0,k=0; while (i<m && j<n) { if (A[i] < B[j]) C[k++] = A[i++]; else C[k++] = B[j++]; } while (i<m) C[k++] = A[i++]; while (j<n) C[k++] = B[j++]; }

70 Merge Sort: Sorting an array recursively void mergesort (int A[], int n) { int i, j, B[max]; if (n <= 1) return; i = n/2; mergesort(A, i); mergesort(A+i, n-i); merge(A, A+i, B, i, n-i); for (j=0; j<n; j++) A[j] = B[j]; free(B); } T(1) = C1, T(n) = 2 T(n/2) + cn + d Recurrence relation

Solving recurrences T(n) = 2 T(n/2) + cn + d T(n/2)=2T(n/4)+cn/2+d T(n)=2(2T(n/4)+cn/2+d)+cn+d =4T(n/4)+cn+2d+cn+d =4T(n/4)+2cn+3d After k steps T(n)=2 k T(n/2 k ) + kcn+C’d =nT(1)+nlog(n)+C2’ =C1*n+nlog(n)+C2’=O(nlogn) n=2 k