Algorithm Efficiency: Searching and Sorting Algorithms Recursion and Searching Problems Execution time of Algorithms Algorithms growth rate Order-of-Magnitude Analysis and Big O Notation The efficiency of Searching Algorithms Sorting Algorithms and their Efficiency

Recursion and Search Problems Problem: Look for the word ‘vademecum’ in a dictionary! Solution 1: Start at the beginning of the dictionary and look at every word in order until you find ‘vademecum’ -- sequential search You want a faster way to perform the search? Solution 2: Open the dictionary probably to a point near its middle and glance at the page, determine which “half” of the dictionary contains the desired word -- binary search.

Pseudocode: A binary search of a dictionary //search a dictionary for a word by using a recursive binary search if( the dictionary contains only one page) { scan the page for the word } else { open the dictionary to a point near the middle Determine which half of the dictionary contains the word if (the word is in the first half of the dictionary) { Search the first half of the dictionary for the word Search the second half of the dictionary for the word } // end if } // end if

Missing from our solution … How do you scan a page? How do you find the middle of the dictionary? Once the middle is found, how do you determine which half contains the word? Search dictionary Search first half of dictionary Search second half of dictionary Recursive Solution

A binary search uses divide-and-Conquer strategy After you have divided the dictionary so many times that you are left with only single page, halving ceases Now the problem is sufficiently small that you can solve it directly by scanning the single page that remains for the word – base case This strategy is one of Divide-and-conquer: You solve the dictionary search problem by first dividing the dictionary into two halves and then conquering the appropriate half

A more rigorous formulation search(theDictionary, aWord) if( theDictionary is one page in size) { scan the page for aWord } else { Open theDictionary to a point near the middle Determine which half of theDictionary contains aWord if (aWord is in the first half of theDictionary) { search(first half of theDictionary, aWord) search(second half of theDictionary, aWord) } // end if } // end if

Determining the Efficiency of Algorithms Comparison of Algorithms is a topic that is central to computer Science The choice of algorithm for a given application often has a great impact Responsive word processors, automatic teller machines, video games, and life support systems all depend on efficient algorithms Consider efficiency when selecting algorithms The analysis of algorithms is an area of computer science that provide tools for contrasting the efficiency of different methods of solutions An analysis should focus on gross differences in the efficiency of algorithms that are likely to dominate the overall cost of a solution

Determining the Efficiency of Algorithms The efficiency of both time and memory is important, but the emphasis will be on time. Three difficulties with comparing programs instead of algorithms How are the algorithms coded? What computer should you use? What data the programs use? Algorithm analysis should be independent of specific implementations, computers, and data Computer scientists employ mathematical techniques that analyze algorithms independently of specific implementations, computers, or data. Begin the analysis by counting the number of significant operations in a particular solution

The Execution time of Algorithms An algorithm’s execution time is related to the number of operations it requires Counting an algorithm’s operations – if possible - is a way to assess its efficiency Traversal of a linked list: Displaying contents of linked list Node curr = head;  1 assignment while (curr != null) {  n +1 comparisons System.out.println(curr.getItem());  n writes Curr.setNext(curr.getNext());  n assignments } // end while

The Execution time of Algorithms If each assignment, comparison, and write operation requires, respectively a, c, and w time units, the statements require (n+1) * (a + c) + n * w time units. Displaying the data in a linked list of n nodes requires time proportional to n. Intuitively it takes longer to display, or traverse, a linked list of 100 items than it does a liked list of 10 items

The Execution time of Algorithms Nested loops: for (i = 1 through n) { for ( j = 1 through i) { for (k = 1 through 5) { Task T } // end for If task T requires t time units, the innermost loop on k requires 5 * t time units. The loop on j requires 5 * t * i time units, and the outermost loop on i requires ∑ni=1 (5*t*i) = 5 *t*(1+2+…+n)=5*t*n*(n+1)/2 time units.

Algorithm Growth Rates Measure an algorithm's time requirements as a function of the problem size Problem size examples: number of nodes in a linked list, the number of disks in the Tower of Hanoi problem, the size of an array, the number of items in a stack, etc. Thus, we reached conclusions such as Algorithm A requires n2/5 time units to solve a problem of size n Algorithm B requires 5 * n time units to solve a problem of size n The time units in these statements must be the same before you can compare the efficiency of the two algorithms. Perhaps we should have written Algorithm A requires n2/5 seconds to solve a problem of size n

Algorithm Growth Rates But, preceding statement is inherent to the following difficulties On what computer does the algorithm require n2/5 seconds? What implementation of the algorithm requires n2/5 seconds? What data caused the algorithm to require n2/5 seconds? Most important thing to learn is how quickly the algorithm’s time requirement grows as a function of the problem size Statements such as Algorithm A requires time proportional to n2 Algorithm B requires time proportional to n Each express an algorithm’s proportional time requirement, or growth rate, and enable you to compare algorithm A with another algorithm B

Algorithm Growth Rates Compare algorithm efficiencies for large problems Although you cannot determine the exact time requirement for either algorithm A or algorithm B from these statements, you can determine that for large problems, B will require significantly less time than A. B’s time requirement as a function of the problem size n increases at a slower rate than A’s time requirement, because n increases at a slower rate than n2 Even if B actually requires 5*n seconds and actually A requires n2/5 seconds, B eventually will require significantly less time than A, as n increases.

The time requirements as function of the problem size n Algorithm A requires n2/5 seconds Algorithm B requires 5*n seconds seconds n 25 Note: A’s time requirement does not exceed B’s until n exceeds 25!

Order-of-Magnitude Analysis and Big O notation If Algorithm A requires time proportional to f(n), Algorithm A is said to be order f(n), which is denoted as O(f(n)). The function f(n) is called the algorithm’s growth-rate function Because the notation uses the capital letter O to denote order, it is called the Big O notation If a problem of size n requires time that is directly proportional to n, the problem is O(n)—that is, order n. If the time requirement is directly proportional to n2, the problem is O(n2), and so on.

Order-of-Magnitude Analysis and Big O notation Definition of the Order of an Algorithm: An algorithm A is order f(n)—denoted O(f(n))—if constants k and n0 exist such that A requires no more than k*f(n) time units to solve a problem of size n ≥ n0 The requirement n ≥ n0 in the definition of O(f(n)) formalizes the notion of sufficiently large problems. In general, many values of k and n can satisfy the definition

Definition of the Order of an Algorithm: Examples Suppose that an algorithm requires n2–3 * n + 10 seconds to solve a problem of size n. If constants k and n0 exist such that k*n2 > n2 -3*n +10 for all n ≥ n0 the algorithm is order n2. In fact, if k is 3 and n0 is 2, 3*n2 > n2-3*n+10 for all n ≥ 2 3*n2 n2 – 3*n + 10 Seconds When n ≥ 2, 3*n2 exceeds n2 – 3*n + 10; thus the algorithm requires no more than k*n2 time units for n > n0, and so is O(n2). 1 2 3 n

Definition of the Order of an Algorithm: Examples We found that displaying a linked list’s first n items requires (n+1)*(a+c) + n*w time units. Since 2*n ≥ n +1 for n ≥ 1, (2*a + 2*c + w)*n ≥ (n+1)*(a+c) + n*w for n ≥1 Thus, this task is O(n). Here k is 2*a + 2*c + w, and n0 is 1

Order of growth of some common functions O(1) < O(log2n) < O(n) < O(n*log2n) < O(n2) < O(n3) < O(2n) Function 10 100 1,000 10,000 100,000 1,000,000 1 log2n 3 6 9 13 16 19 n 102 103 104 105 106 n*log2n 30 664 9,965 107 n2 108 1010 1012 n3 109 1015 1018 2n 1030 10301 103,101 1030,103 10301,030

Graphical comparison of growth rate function (f(n)=1 is ommitted) Value of growth-rate function 2n 100 n3 n2 75 n*log2n 50 n 25 log2n 1 5 10 15 20 n

Order of Growth of some Common Functions The table demonstrates the relative speed at which the values of the function grow The growth-rate functions are also depicted graphically If algorithm A requires time that is proportional to function f and algorithm B requires time that is proportional to a slower-growing function g, it is apparent that B will always be significantly more efficient than A for large enough problems For large enough problems, the proportional growth rate dominates all other factors in determining an algorithm’s efficiency

Order of Growth of some Common Functions Some properties of growth-rate functions: You can ignore low-order terms in an algorithm’s growth-rate function. E.g. if an algorithm is O(n3+4*n2+3), it is also O(n3). You can ignore a multiplicative constant in the high-order term of an algorithm’s growth-rate function. e.g. if an algorithm is O(5*n3), it is also O(n3) O(f(n)) + O(g(n)) = O(f(n) + g(n)). E.g. if an algorithm is O(n2) + O(n), it is also O(n2 + n), which is simply O(n2).

Worst-case and Average-case Analyses Algorithms can require different times to solve different problems of the same size E.g. the time that an algorithm requires to search n items might depend on the nature of the items Maximum amount of time that an algorithm can require to solve a problem of size n—is the worst case An average-case analysis attempts to determine the average amount of time that an algorithm requires to solve problems of size n. Worst-case analysis is easier to calculate and is thus more common

The efficiency of Searching Algorithms Order-of-magnitude analysis: - Efficiency of Sequential Search and Binary Search of an array Sequential search:- to search from an array of n items, you look each item in turn, starting with the first one, until either you find the desired item or you search to the end of the data collection Worst case: O(n) Average case: O(n) Best case: O(1) Does the algorithm order depend on whether or not the initial data is sorted?

The efficiency of Searching Algorithms Binary search:-Searches a sorted array for a particular item by repeatedly dividing the array in half—the binary search algorithm searches successively smaller arrays. The size of a given array is approximately one-half the size of the array previously searched At each division, the algorithm makes a comparison. The number of comparison is equal to the number of times that the algorithm divides the array in half

The efficiency of Searching Algorithms – Binary Search Suppose that n = 2k for some k The search requires the following steps: Inspect the middle item of an array of size n Inspect the middle item of an array of size n/2 Inspect the middle item of an array of size n/22, and so on, until only one item remains. You will have performed k divisions (n/2k = 1) In the worst case, the algorithm performs k divisions and, therefore, k comparisons. Because n=2k, k = log2n. Thus, the algorithm is O(log2n) in the worst case when n = 2k.

The efficiency of Searching Algorithms – Binary Search What if n is not a power of 2? Find the smallest k such that 2k-1 < n < 2k The algorithm still requires at most k divisions to obtain a sub-array with one item. It follows that k-1 < log2n < k k < 1 + log2n < k+1 K = 1 + log2n rounded down Thus the algorithm is still O(log2n) in the worst case when n ≠2k. How does binary search compare to sequential search? E.g. log21,000,000 = 19, so one million sorted items can require one million comparisons with SS but at most 20 with BS! Note: Maintaining the array in sorted order requires an overhead cost, which can be substantial!!