1. CSS342: Algorithm Analysis1 Professor: Munehiro Fukuda

2. CSS342: Algorithm Analysis2 Today’s Topics 1.Mathematical analysis of algorithm complexity 2.Examples of algorithm complexity 3.Algorithm improvement from O(N 3 ) to O(N) 4.Efficiency of search algorithms

3. CSS342: Algorithm Analysis3 Measuring Algorithm Efficiency Cost of a computer program –Human cost: Developing good problem-solving skills and programming style –Execution cost: Algorithm efficiency Then, what in efficiency should we focus? Clever trick in coding?Execution time? How should we compare programs? Based on a particular computer? What data should be given? Mathematical Analysis

4. CSS342: Algorithm Analysis4 Counting an Algorithm’s operations for ( i = 1; i <= n; i++ ) for ( j = 1; j <= i; j++ ) for ( k = 1; k <= 5; k++ ) task( ); Task:t operations ( or may take t seconds) Loop on k:5 * t Loop on j:5 * t (i = 1), 5 * t * 2 (i = 2), … 5 * t * n (i = n) n Loop on i:∑(5 * t * i) = 5 * t * n * (n + 1) / 2 i=1 Mathematical Analysis

5. CSS342: Algorithm Analysis5 Algorithm Growth Rates Nested loop i/j/k requires 5n(n+1)/2 time units to process a problem of size n. –Units may be seconds, minutes, and even dependent on what machines we use! Nested loop i/j/k requires time proportional to 5n(n+1)/2 –Focus on how quickly the algorithm’s time requirement grows as a function of the problem size. Growth rate Mathematical Analysis

6. CSS342: Algorithm Analysis6 Order-of-Magnitude Analysis Let Algorithm A require time proportional to f(n). A is of order at most g(n), O(g(n)) if f(n) n 0 A is of order at least g(n), Ω(g(n)) if f(n) >= k * g(n), for a positive constant k and n > n 0 A is of order g(n), Θ(g(n)) if f(n) = O(g(n)) and f(n) = Ω(g(n)) Mathematical Analysis

7. CSS342: Algorithm Analysis7 Example of O(g(n)), Ω(g(n)),andΘ(g(n)) Suppose Algorithm A requires time proportional to f(n) = n 2 – 3n + 10 f(n) = 2, and thus f(n) = O(n 2 ) f(n) > k * n 2, if k = 0.5 n>= 0, and thus f(n) = Ω(n 2 ) Therefore f(n) = Θ(n 2 ) 27104.53 12822 380.51 01000 3n 2 N 2 -3N+100.5n 2 n Mathematical Analysis

8. CSS342: Algorithm Analysis8 Example of Program Analysis j = n while ( j >= 1 ) { for ( i = 1; i <= j; i++ ) x = x + 1;// How many times is x=x+1 executed? j = j / 2; } T(n): # x=x+1 executed In the 1 st while-loop, j = n. for-loop executes x=x+1 n times. T(n)=Ω(n) In the 2 nd while-loop, j <= n/2,n/2 times In the 3 rd while-loop, j <= n/4,n/4 times In the k th while-loop, j <= n/2 k-1,n/2 k-1 times Mathematical Analysis

9. CSS342: Algorithm Analysis9 Example of Program Analysis Cont’d Geometric Sum: a + ar 1 + ar 2 + …. + ar n = a(r n+1 – 1)/(r – 1), where r != 1 (Proof is given in the week 4, “Induction”) Thus, T(n)<= n + n/2 + n/4 + … + n/2 k-1 = n + n*(1/2) + n*(1/2) 2 + … + n * (1/2) (k-1) = n((1/2) k -1)/(1/2 – 1) = n(1-1/2 k )/(1-1/2) = 2n(1-1/2 k ) <= 2n T(n) = O(n) Therefore, T(n) = Θ(n) Mathematical Analysis

10. CSS342: Algorithm Analysis10 Focusing on Big O Notation Upper-bound information is the main concern In most cases, O(g(n))=Ω(g(n)) =Θ(g(n)) O(1) < O(log 2 n) < O(n) < O(n*log 2 n) < O(n 2 ) < O(n 3 ) < O(2 n ) Focus on the dominant factor: –Low-order terms can be ignored. O(n 3 +4n) = O(n 3 ) –Multiplicative constant in the high-order term can be ignored. O(3n 3 +4) = O(n 3 ) O(f(n)) + O(g(n)) = O(f(n) + g(n)) If f(n) = O(g(n)) and g(n) = O(h(n)), then f(n)=O(h(n)) Mathematical Analysis

11. CSS342: Algorithm Analysis11 Intuitive interpretation of growth-rate functions Increase too rapidly to be practicalExponentialO(2 n ) Ex. Three nested loopsCubicO(n 3 ) Ex. Two nested loopsQuadraticO(n 2 ) Increase more rapidly than a liner algorithm. Ex. Merge sort O(nlog 2 n) Increase directly with sizeLinearO(n) Increase slowly as size increases. Ex. Binary search LogarithmicO(log 2 n) Independent of problem sizeConstantO(1) Mathematical Analysis

12. CSS342: Algorithm Analysis12 Examples of Algorithm Complexity Minimum Element in an Array Given an array of N items, find the smallest item. What order (in big O) will this problem be bound to? 391852074610-2 N 1 Examples

13. CSS342: Algorithm Analysis13 Examples of Algorithm Complexity Closest Points in the Plane Given N points in a plane (that is, an x-y coordinate system), find the pair of points that are closest together. What order (in big O) will this problem be bound to? 2,1 x y 1,4 1,6 4,7 4,5 5,4 5,6 7,1 8,4 6,7 6,3 sqrt( (x 2 – x 1 ) 2 + (y 2 – y 1 ) 2 ) Examples

14. CSS342: Algorithm Analysis14 Examples of Algorithm Complexity Colinear Points in the Plane Given N points in a plane (that is, an x-y coordinate system), determine if any tree form a straight line. What order (in big O) will this problem be bound to? 2,1 x y 1,4 1,6 4,7 4,5 5,4 5,6 7,1 8,4 6,7 6,3 Find y = ax + b for two points. Check if the 3 rd point satisfies this equation Examples

15. CSS342: Algorithm Analysis15 The Maximum Contiguous Subsequence Sum Problem Given a sequence of integers I 1..I N, find and identify the sub sequence corresponding to the maximum value of ∑ k=i j A k. Algorithm Improvement -211-413-521-34-26-2 N 20 1

16. CSS342: Algorithm Analysis16 O(N 3 ) Algorithm Algorithm Improvement -211-413-521-34-26-2 N 1 ij ∑ k=i j A k The most inner loop The intermediate loop j++ The most outer loop i++ int maxSum = 0; for ( int i = 0; i < n; i++ ) for ( int j = i; j < n; j++ ) { int thisSum = 0; for ( int k = i; k <= j; k++ ) thisSum += a[k]; if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } We don’t need this loop! Simply increment j and add a new integer to the sum.

17. CSS342: Algorithm Analysis17 O(N 2 ) Algorithm Algorithm Improvement -211-413-521-34-26-2 N 1 ij ∑ j=i j A j The intermediate loop j++ The most outer loop i++ int maxSum = 0; for ( int i = 0; i < n; i++ ) { int thisSum = 0; for ( int j = i; j < n; j++ ) { thisSum += a[j]; if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; }

18. CSS342: Algorithm Analysis18 O(N) Algorithm Algorithm Improvement N1 ∑ < 0 Ignore this part From here, restart the summation. ∑ j=i n A j Generally, the summation increases but if it gets below 0 int thisSum = 0; int maxSum = 0; for ( int i = 0, j = 0; j < n; j++ ) { thisSum += a[j]; if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } else if ( thisSun < 0 ) { i = j + 1; thisSum = 0; }

19. CSS342: Algorithm Analysis19 Worst, Best, and Average-case Analysis Worst-case analysis: –Algorithm A requires no more than k * f(n) time units. –It might happen rarely, if at all, in practice. Best-case analysis: –Like worst-case analysis, it might happen rarely. Average-case analysis: –A requires no more than k * f(n) time units for all but a finite number of values of n –Difficulties:determining probability and distribution of size n and input data Efficiency of Search Algorithms

20. CSS342: Algorithm Analysis20 Efficiency of Sequential Search 29101413374652755436921188 Best case O(1) Hit! Worst case Hit! O(n) Average case Hit! O(n/2) = O(n) (Assume desired items are uniformly distributed.) Efficiency of Search Algorithms

21. CSS342: Algorithm Analysis21 Efficiency of Binary Search 291014133746527554369211889991 Efficiency of Search Algorithms template int binarySearch( const vector &a, const Comparable &x ) { int low = 0; int high = a.size( ) – 1; int mid; while ( low <= high ) { mid = ( low + high ) / 2; if ( a[mid] < x ) low = mid + 1; else if ( a[mid] > x ) high = mid – 1; else return mid; } return NOT_FOUND; // NOT_FOUND = -1 } low =0 high=15 mid = (15 + 0)/2 = 7 10 x a new high=7–1=6

22. CSS342: Algorithm Analysis22 Efficiency of Binary Search N = 2K = 1 N = 4K = 2 N = 8K = 3 N = 16K = 4 N = 2 k K = K 2 K-1 < N < 2 K K – 1 < log 2 N < K K < 1 + log 2 N < K + 1K = O(log 2 N) 29101413374652755436921188 1 st step 9991 2 nd step 3 rd step 4 th step Hit! Efficiency of Search Algorithms

23. CSS342: Algorithm Analysis23 Efficiency of Interpolation Search A way you start your search for “Hank Aaron” from the first 20 pages in a 500-page white page: next = low + ┌ (x – a[low]) / (a[high] – a[low]) * (high – low – 1)┐ Example: Assume low = 0, high = 999, a[low] = 1000, a[high]=1000000. If you search for the value 12000: next = 0 + ┌(12000 – 1000)/(1000000 – 1000) * (999 – 0 – 1) ┐ = 10 Two assumptions: –Each access must be very expensive like a disk access. –Data must be uniformly distributed Complexity –Average: O(log log N) –Worst: O(N) Efficiency of Search Algorithms

24. CSS342: Algorithm Analysis24 Some Note for Algorithm Analysis Checking an algorithm analysis 1.Empirically observe running time as increasing N. 2.Divide the time by O(f(N)). 3.The dividend should converge to a positive constant. Limitations of Big-Oh Analysis –Does not work for a small N –Worst-case analysis may be uncommon –Average-case analysis is more difficult than worse case –Empirically observation depends on memory hierarchy and multiprogramming level, etc.. Limitations

25. CSS342: Algorithm Analysis25 Intractable, Unsolvable, and NP problems Good algorithms: –Their worst-case time is proportional to a polynomial. Intractable problems: –Their worst-case time cannot be bounded to a polynomial. Unsolvable algorithms: –They have no algorithms at all. Ex. Halting problem NP(Non-Polynomial) problems: –They are thought to be intractable, but not yet proved. –Ex1. Traveling salesperson problem –Ex2. Hamiltonian cycle Limitations