1. Example Algorithms CSE 2320 – Algorithms and Data Structures Alexandra Stefan University of Texas at Arlington Last updated 1/31/2016 1

2. Summary Search (sequential and binary) - Chapter 2.6. – See the notation conventions (e.g. log 2 N = lg N) Sorting algorithms – Properties of sorting algorithms – Data abstraction - briefly: Item, key(), less(),.. All algorithms can benefit from this, but may require different interface – Selection sort – Chapter 6.2. – Insertion sort – Chapter 6.3 – Merge sort – Chapter 6.3 Union-Find – Chapter 1. 2

3. Searching 3

4. Unsorted array – Sequential search Sorted arrays – Sequential search (section 12.3) – Binary search (section 12.4) – Later on: Range search (section 12.4) Interpolation search (section 12.4) 4

5. Membership Search (Chapter 2.6) We have a set S of N objects. Given an object v, we want to determine if v is an element of S. For simplicity, now we will only handle the case where objects are integers. – It will become apparent soon that the solution actually works for much more general types of objects. Can anyone think of a simple solution for this problem? 5

6. Membership Search We have a set S of N objects. Given an object v, we want to determine if v is an element of S. 6

7. Sequential Search - in unsorted array We have a set S of N objects (unsorted). Given an object v, we want to determine if v is an element of S. Sequential search: – Compare v with every element of S. How long does this take? 7

8. Sequential Search - in unsorted array We have a set S of N objects (unsorted). Given an object v, we want to determine if v is an element of S. Sequential search: – Compare v with every element of S. How long does this take? – If v is in S, we need on average to compare v with |S|/2 objects assuming uniform distribution of objects – If v is not in S, we need compare v with all |S| objects. 8

9. Sequential Search - in sorted array Assume that S is sorted in ascending order (this is an assumption that we did not make before). Sequential search, version 2: – Compare v with every element of S, till we find the first element s such that s >= v. – Then, if s != v we can safely say that v is not in S. How long does this take? 9

10. Sequential Search - in sorted array Assume that S is sorted in ascending order (this is an assumption that we did not make before). Sequential search, version 2: – Compare v with every element of S, till we find the first element s such that s >= v. – Then, if s != v we can safely say that v is not in S. How long does this take? – We need on average to compare v with |S|/2 objects, regardless of whether v is in S or not. A little bit better than when S was not sorted, but only by a factor of 2, only when v is not in S (assuming that v can be between any 2 consecutive elements in S). 10

11. Binary Search Again, assume that S is sorted in ascending order. If v is in S, v can appear in any position, from 0 to N-1 (where N =|S|). Let's call – left the leftmost position where v may be, and – right the rightmost position where v may be. Initially: – left = 0 – right = N - 1 If we compare v with S[N/2]. – Note: if N/2 is not an integer, round it down. – Where should we search for v next? (left=? and right=?) 11

12. Binary Search Initially: – left = 0 – right = N – 1 – m = (left+right)/2 (middle index, between left and right) Now, suppose we compare v with S[m]. – If v == S[m], we found v, so we are done. – If v < S[m], then right = m - 1. – If v > S[m], then left = m + 1. Importance: We have reduced our search range in half, with a single comparison. See animation: https://www.cs.usfca.edu/~galles/visualization/Search.html https://www.cs.usfca.edu/~galles/visualization/Search.html – The array stretches on 2 or more lines 12

13. Binary Search - Code /* Determines if v is an element of S. If yes, returns the position of v in S. If not, returns -1. N is the size of S.*/ int search(int S[], int N, int v){ int left, right; left = 0; right = N-1; while (right >= left) { int m = (left+right)/2; if (v == S[m]) return m; if (v < S[m]) right = m-1; else left = m+1; } return -1; } 13 How many times will the while loop repeat for N = 32 ? Best case: 1 iteration (less) (first comparison) Worst case: 6 iterations (not found) Candidates (=right–left+1): 32 16 8 4 2 1 0 (indexes cross over) => 6 iterations => Verify with order cases, e.g. N: 17, 31, 32

14. Verifying the formula Max new candidates (N): – (N-1) b.c. middle position is removed Examples verifying the formula: 14 N: 17 8 4 2 1 0 – stop 5 iter = 4 +1 N: 31 15 7 3 1 0 – stop 5 iter = 4 +1 N: 32 16 8 4 2 1 0 – stop 6 iter = 5 +1

15. Binary Search - Recursive /* Adapted from Sedgewick */ int search(int S[], int left, int right, int v) { int m = (left+right)/2; if (left > right) return -1; if (v == S[m]) return m; if (left == right) return -1; if (v < S[m]) return search(S, left, m-1, v); // recursive call else return search(S, m+1, right, v); // recursive call } - How many recursive calls? - Any correspondence between the previous code (while loop) and this one (recursive)? 15

16. Binary Search – recursive, abstract /* Sedgewick See this slide after data abstraction discussed. Here the array is called st (not S) and it is global. Returns the element, not the position. */ Item search(int l, int r, Key v) { int m = (l+r)/2; if (l > r) return NULLitem; if eq(v, key(st[m])) return st[m]; if (l == r) return NULLitem; if less(v, key(st[m])) return search(l, m-1, v); else return search(m+1, r, v); } 16

17. Time Analysis of Binary Search How many elements do we need to compare v with, if S contains N objects? At most log 2 (N). This is what we call logarithmic time complexity. While constant time is the best we can hope, we are usually very happy with logarithmic time. 17

18. Sorting 18

19. Sorting Suppose that we have an array, a, of items (numbers, strings, etc.), that we want to sort. Why sort it? – To use in binary search. – To compute rankings, statistics (top-10, top-100, median). Sorting is one of the most common operations in software. In this course we will do several different sorting algorithms, with different properties. Today we will look at a few: selection sort, insertion sort, merge sort. 19

20. Properties of sorting Sedgewick 6.1 Stable: – It does not change the relative order of items whose keys are equal. Adaptive: – “performs different sequences of operations depending on the outcome of comparisons ( less operations)”. – The runtime will depend on the input – see later insertion sort vs selection sort. Memory used – In place methods: constant extra memory – N pointers (e.g. linked lists or indirect access) – double: require a complete copy of the data Access of items – Direct: move entire items when needed – Indirect: move pointers to items. Can keep the key with pointer or not. 20

21. Stable sorting In this example, an item consists of an int (e.g. GPA) and a string (e.g. name). The GPA will be the key for sorting. Stable sort (Tom before Jane and Bob before Anna) : Unstable sort (violation: Anna is now before Bob) : 21 4 Bob 3 Tom 4 Anna 3 Jane 1 Henry 1 Henry 3 Tom 3 Jane 4 Bob 4 Anna 1 Henry 3 Tom 3 Jane 4 Anna 4 Bob

22. Applications of stable sorting Sorting by 2 criteria, – E.g.: 1 st by GPA, 2 nd by name: When the GPA is the same, have data in order of names – Solution 1: First sort by name Next, with a stable sort, sort by GPA – Solution 2: write a more complex comparison function. Part of other sorting methods – See later: count sort used as part of LSD radix sort 22

23. Data abstraction Different types of data and different comparisons: – Integers – Doubles – Strings – Records containing multiple pieces of data E.g. student records: name, ID, GPA, major,scans,… Sort by any piece of data from that record: – strcmp,, combination, new measure (video simmilarity) Same algorithm for all of the above if: – Abstract over: type, key, comparison and Any other operations on the data 23

24. The Item Type In general, the array or list we want to sort contains items of some type, that we will call Item. For example, Item can be defined to be: – int – double – char * – a pointer to some structure. – flexibility. Most of the sorting algorithms we study work on any Item type. In our code, we will specify the Item type as needed using typedef. For example: typedef int Item; // working with integers 24

25. The Key If Item is a pointer to a structure, then we need to specify which member variable of that structure will be used for sorting. This member variable is called the key of the object. For example, we could have a structure for a person, that includes age, weight, height, address, phone number. – If we want to sort based on age, then the key of the structure is age. – If we want to sort based on height, then the key of the structure is height. For simple data types (numbers, strings), the key of the object can be itself. In our code, we will specify the key as needed using a #define macro. For example: #define key(A) (A) 25

26. The Comparison Operator We are sorting in some kind of order. The comparison operator C is what defines this order. The comparison operator C is a function that takes two objects (of type Item) as arguments, and returns True or False. A sequence A 1, A 2, A 3,..., A K is sorted if, for all i between 1 and K-1: – C(A i, A i+1 ) is True, OR A i is equal to A i+1. 26

27. The Comparison Operator Examples of comparison operators:, strcmp(). However, they can be more complicated: – Case-sensitive alphabetical ordering. – Case-insensitive alphabetical ordering. –... The textbook code (unfortunately) calls every such operator less, even if it is set to >. We define less as needed using a #define macro. For example: #define less(A, B) (key(A) < key(B)) 27

28. Additional Notation To save lines of code and make the algorithm more clear, the textbook defines the following macros: exch(A, B): swaps the values of A and B. #define exch(A, B) { Item t = A; A = B; B = t; } compexch(A, B): swap values of A and B ONLY IF less(B, A). #define compexch(A, B) if (less(B, A)) exch(A, B) 28

29. Summary of Definitions and Macros In a specific implementation, we must define: – Item – Key – less For example: typedef int Item; #define key(A) (A) #define less(A, B) (key(A) < key(B)) In all implementations, we use macros exch and compexch (most of the times you would not need to redefine these): #define exch(A, B) { Item t = A; A = B; B = t; } #define compexch(A, B) if (less(B, A)) exch(A, B) 29

30. Selection sort 30

31. Selection Sort Given unsorted array, a. From left to right, put the remaining smallest element on it’s final position – Find the smallest element, and exchange it with element at position 0. – Find the second smallest element, and exchange it with element at position 1. – … – Find the i-th smallest element, and exchange it with element at position i-1. – If we do this |a|-1 times, then a will be sorted. See animation at: https://www.cs.usfca.edu/~galles/visualization/ComparisonSort.html https://www.cs.usfca.edu/~galles/visualization/ComparisonSort.html 31

32. Selection Sort - Code For simplicity, we only handle the case where the items are integers. /* sort array a in ascending order. N is the number of elements in a. */ void selection(int a[], int N) { int i, j, temp; for (i = 0; i < N-1; i++) { int min = i; //min: index of the smallest remaining element for (j = i+1; j < N; j++) if (a[j] < a[min]) min = j; // swap elements: temp = a[min]; a[min] = a[i]; a[i] = temp; } 32

33. Selection Sort – data abstraction For simplicity, we only handle the case where the items are integers. /* sort array a in ascending order. N is the number of elements in a. */ void selection(Item a[], int N) { int i, j, temp; for (i = 0; i < N; i++) { int min = i; //min is the index of the smallest element for (j = i+1; j < N; j++) if ( less(a[j],a[min]) ) min = j; // swap elements exch(a[i], a[min]); } 33

34. Selection Sort - Time Analysis Find the smallest element, and exchange it with element at position 0. – N-1 comparisons. Find the 2 nd smallest element, and exchange it with element at position 1. – N-2 comparisons. Step i: find the i-th smallest element, and exchange it with element at position i-1. – N-i comparisons. Total: (N-1) + (N-2) + (N-3) + … + 1 = [(N-1)*N]/2 = (N 2 –N)/2 – about N 2 /2 comparisons. (dominant term: N 2 ) 34

35. Selection Sort - Time Analysis Total: (N-1) + (N-2) + (N-3) + … + 1 = about 0.5 * N 2 comparisons. Quadratic time complexity: the dominant term is N 2. Commonly used sorting algorithms are a bit more complicated, but have N * lg(N) time complexity, which is much better (as N gets large). – See merge sort 35

36. Insertion sort 36

37. Insertion sort Process array from left to right. Step i: – elements a[0],[a[1],…a[i-1] are already sorted – insert element a[i] in it’s place among a[0],..a[i-1] See animation: https://www.cs.usfca.edu/~galles/visualization/ComparisonSort.html https://www.cs.usfca.edu/~galles/visualization/ComparisonSort.html 37

38. Insertion sort void insertion(Item a[], int l, int r) { int i; // place the smallest item on the leftmost position: sentinel for (i = l+1; i <= r; i++) compexch(a[l], a[i]); for (i = l+2; i <= r; i++) { int j = i; Item v = a[i]; while (less(v, a[j-1])) { a[j] = a[j-1]; j--; } a[j] = v; } 38

39. Insertion sort - Properties Is this particular implementation stable? – Give an example array and show the problem. Can you give a different implementation (of the same algorithm) that is stable? – With sentinel – Without sentinel 39

40. Insertion sort – time complexity // Sort array, a, between positions left and r. void insertion(Item a[], int left, int r) { int i; // place the smallest item on the leftmost position: sentinel for (i = left+1; i <= r; i++) compexch(a[left], a[i]); for (i = left+2; i <= r; i++) { int j = i; Item v = a[i]; while (less(v, a[j-1])) { a[j] = a[j-1]; j--; } a[j] = v; } 40 How many time does the while loop execute for a particular i ? Best case: Worst case: Average case: Total runtime: Best: Worst: Average:

41. Time complexity Insertion sort is adaptive: – If a is sorted, the runtime is proportional to N 41

42. Merge sort and Union-Find See merge sort slides. Skip the time complexity analysis. See the Union-Find slides 42