Chapter 9 sorting

Insertion Sort I The list is assumed to be broken into a sorted portion and an unsorted portion The list is assumed to be broken into a sorted portion and an unsorted portion Keys will be inserted from the unsorted portion into the sorted portion. Keys will be inserted from the unsorted portion into the sorted portion. Sorted Unsorted

Insertion Sort II For each new key, search backward through sorted keys For each new key, search backward through sorted keys Move keys until proper position is found Move keys until proper position is found Place key in proper position Place key in proper position Moved

Insertion Sort: Code template void insertionSort( vector & a ) { for( int p = 1; p < a.size( ); p++ ) { Comparable tmp = a[ p ]; int j; for( j = p; j > 0 && tmp < a[ j - 1 ]; j-- ) a[ j ] = a[ j - 1 ]; a[ j ] = tmp; } Fixed n-1 iterations Worst case i-1 comparisons Move current key to right Insert the new key to its proper position Searching for the proper position for the new key Moved

Insertion Sort: Analysis Worst Case: Keys are in reverse order Worst Case: Keys are in reverse order Do i-1 comparisons for each new key, where i runs from 2 to n. Do i-1 comparisons for each new key, where i runs from 2 to n. Total Comparisons: … + n-1 Total Comparisons: … + n-1 Comparison

Optimality Analysis I To discover an optimal algorithm we need to find an upper and lower asymptotic bound for a problem. To discover an optimal algorithm we need to find an upper and lower asymptotic bound for a problem. An algorithm gives us an upper bound. The worst case for sorting cannot exceed  (n 2 ) because we have Insertion Sort that runs that fast. An algorithm gives us an upper bound. The worst case for sorting cannot exceed  (n 2 ) because we have Insertion Sort that runs that fast. Lower bounds require mathematical arguments. Lower bounds require mathematical arguments.

Optimality Analysis II Making mathematical arguments usually involves assumptions about how the problem will be solved. Making mathematical arguments usually involves assumptions about how the problem will be solved. Invalidating the assumptions invalidates the lower bound. Invalidating the assumptions invalidates the lower bound. Sorting an array of numbers requires at least  (n) time, because it would take that much time to rearrange a list that was rotated one element out of position. Sorting an array of numbers requires at least  (n) time, because it would take that much time to rearrange a list that was rotated one element out of position.

Rotating One Element 2nd 3rd 4th nth 1st 2nd 3rd n-1st nth 1st n keys must be moved  (n) time Assumptions: Keys must be moved one at a time All key movements take the same amount of time The amount of time needed to move one key is not dependent on n.

Other Assumptions The only operation used for sorting the list is swapping two keys. The only operation used for sorting the list is swapping two keys. Only adjacent keys can be swapped. Only adjacent keys can be swapped. This is true for Insertion Sort and Bubble Sort. This is true for Insertion Sort and Bubble Sort.

Inversion Not an InversionInversions Suppose we are given a list of elements L, of size n. Suppose we are given a list of elements L, of size n. Let i, and j be chosen so 1  i<j  n. Let i, and j be chosen so 1  i<j  n. If L[i]>L[j] then the pair (i,j) is an inversion. If L[i]>L[j] then the pair (i,j) is an inversion.

Maximum Inversions The total number of pairs is: The total number of pairs is: This is the maximum number of inversions in any list. This is the maximum number of inversions in any list. Exchanging adjacent pairs of keys removes at most one inversion. Exchanging adjacent pairs of keys removes at most one inversion.

Swapping Adjacent Pairs Swap Red and Green The relative position of the Red and blue areas has not changed. No inversions between the red key and the blue area have been removed. The same is true for the red key and the orange area. The same analysis can be done for the green key. The only inversion that could be removed is the (possible) one between the red and green keys.

Lower Bound Argument A sorted list has no inversions. A sorted list has no inversions. A reverse-order list has the maximum number of inversions,  (n 2 ) inversions. A reverse-order list has the maximum number of inversions,  (n 2 ) inversions. A sorting algorithm must exchange  (n 2 ) adjacent pairs to sort a list. A sorting algorithm must exchange  (n 2 ) adjacent pairs to sort a list. A sort algorithm that operates by exchanging adjacent pairs of keys must have a time bound of at least  (n 2 ). A sort algorithm that operates by exchanging adjacent pairs of keys must have a time bound of at least  (n 2 ).

Lower Bound For Average I There are n! ways to rearrange a list of n elements. There are n! ways to rearrange a list of n elements. Recall that a rearrangement is called a permutation. Recall that a rearrangement is called a permutation. If we reverse a rearranged list, every pair that used to be an inversion will no longer be an inversion. If we reverse a rearranged list, every pair that used to be an inversion will no longer be an inversion. By the same token, all non-inversions become inversions. By the same token, all non-inversions become inversions.

Lower Bound For Average II There are n(n-1)/2 inversions in a permutation and its reverse. There are n(n-1)/2 inversions in a permutation and its reverse. Assuming that all n! permutations are equally likely, there are n(n-1)/4 inversions in a permutation, on the average. Assuming that all n! permutations are equally likely, there are n(n-1)/4 inversions in a permutation, on the average. The average performance of a “swap- adjacent-pairs” sorting algorithm will be  (n 2 ). The average performance of a “swap- adjacent-pairs” sorting algorithm will be  (n 2 ).

Shell Sort With insertion sort, each time we insert an element, other elements get nudged one step closer to where they ought to be With insertion sort, each time we insert an element, other elements get nudged one step closer to where they ought to be What if we could move elements a much longer distance each time? What if we could move elements a much longer distance each time? We could move each element: We could move each element: A long distance A long distance A somewhat shorter distance A somewhat shorter distance A shorter distance still A shorter distance still This approach is what makes shellsort so much faster than insertion sort This approach is what makes shellsort so much faster than insertion sort

Sorting nonconsecutive subarrays  Consider just the red locations  Suppose we do an insertion sort on just these numbers, as if they were the only ones in the array?  Now consider just the yellow locations  We do an insertion sort on just these numbers  Now do the same for each additional group of numbers   The resultant array is sorted within groups, but not overall Here is an array to be sorted (numbers aren’t important)

Doing the 1-sort In the previous slide, we compared numbers that were spaced every 5 locations In the previous slide, we compared numbers that were spaced every 5 locations This is a 5-sort This is a 5-sort Ordinary insertion sort is just like this, only the numbers are spaced 1 apart Ordinary insertion sort is just like this, only the numbers are spaced 1 apart We can think of this as a 1-sort We can think of this as a 1-sort Suppose, after doing the 5-sort, we do a 1-sort? Suppose, after doing the 5-sort, we do a 1-sort? In general, we would expect that each insertion would involve moving fewer numbers out of the way In general, we would expect that each insertion would involve moving fewer numbers out of the way The array would end up completely sorted The array would end up completely sorted

Example of shell sort original sort sort sort

Diminishing gaps For a large array, we don’t want to do a 5- sort; we want to do an N-sort, where N depends on the size of the array For a large array, we don’t want to do a 5- sort; we want to do an N-sort, where N depends on the size of the array N is called the gap size, or interval size N is called the gap size, or interval size We may want to do several stages, reducing the gap size each time We may want to do several stages, reducing the gap size each time For example, on a 1000-element array, we may want to do a 364-sort, then a 121-sort, then a 40-sort, then a 13-sort, then a 4-sort, then a 1-sort For example, on a 1000-element array, we may want to do a 364-sort, then a 121-sort, then a 40-sort, then a 13-sort, then a 4-sort, then a 1-sort Why these numbers? Why these numbers?

Increment sequence No one knows the optimal sequence of diminishing gaps No one knows the optimal sequence of diminishing gaps This sequence is attributed to Donald E. Knuth: This sequence is attributed to Donald E. Knuth: Start with h = 1 Start with h = 1 Repeatedly compute h = 3*h + 1 Repeatedly compute h = 3*h + 1 1, 4, 13, 40, 121, 364, , 4, 13, 40, 121, 364, 1093 This sequence seems to work very well This sequence seems to work very well Another increment sequence mentioned in the textbook is based on the following formula: Another increment sequence mentioned in the textbook is based on the following formula: start with h = the half of the container’s size start with h = the half of the container’s size h i = floor (h i-1 / 2.2) h i = floor (h i-1 / 2.2) It turns out that just cutting the array size in half each time does not work out as well It turns out that just cutting the array size in half each time does not work out as well

Analysis What is the real running time of shellsort? What is the real running time of shellsort? Nobody knows! Nobody knows! Experiments suggest something like O(n 3/2 ) or O(n 7/6 ) Experiments suggest something like O(n 3/2 ) or O(n 7/6 ) Analysis isn’t always easy! Analysis isn’t always easy!

Merge Sort If List has only one Element, do nothing If List has only one Element, do nothing Otherwise, Split List in Half Otherwise, Split List in Half Recursively Sort Both Lists Recursively Sort Both Lists Merge Sorted Lists Merge Sorted Lists Mergesort(A, l, r) if l < r then q = floor((l+r)/2) mergesort(A, l, q) mergesort(A, q+1, r) merge(A, l, q, r)

The Merge Algorithm Assume we are merging lists A and B into list C. Ax := 1; Bx := 1; Cx := 1; while Ax  n and Bx  n do if A[Ax] < B[Bx] then C[Cx] := A[Ax]; Ax := Ax + 1; else C[Cx] := B[Bx]; Bx := Bx + 1; endif Cx := Cx + 1; endwhile while Ax  n do C[Cx] := A[Ax]; Ax := Ax + 1; Cx := Cx + 1; endwhile while Bx  n do C[Cx] := B[Bx]; Bx := Bx + 1; Cx := Cx + 1; endwhile

auxiliary array smallest AGLORHIMST Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done. A

auxiliary array smallest AGLORHIMST A G Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done.

auxiliary array smallest AGLORHIMST AG Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done. H

auxiliary array smallest AGLORHIMST AGH Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done. I

auxiliary array smallest AGLORHIMST AGHI Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done. L

auxiliary array smallest AGLORHIMST AGHIL Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done. M

auxiliary array smallest AGLORHIMST AGHILM Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done. O

auxiliary array smallest AGLORHIMST AGHILMO Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done. R

auxiliary array first half exhausted smallest AGLORHIMST AGHILMOR Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done. S

auxiliary array first half exhausted smallest AGLORHIMST AGHILMORS Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done. T

auxiliary array first half exhausted second half exhausted AGLORHIMST AGHILMORST Merging Merge. Merge. Keep track of smallest element in each sorted half. Keep track of smallest element in each sorted half. Insert smallest of two elements into auxiliary array. Insert smallest of two elements into auxiliary array. Repeat until done. Repeat until done.

Merging numerical example

Merge Sort: Analysis Sorting requires no comparisons Sorting requires no comparisons Merging requires n-1 comparisons in the worst case, where n is the total size of both lists (n key movements are required in all cases) Merging requires n-1 comparisons in the worst case, where n is the total size of both lists (n key movements are required in all cases) Recurrence relation: Recurrence relation:

Merge Sort: Space Merging cannot be done in place Merging cannot be done in place In the simplest case, a separate list of size n is required for merging In the simplest case, a separate list of size n is required for merging It is possible to reduce the size of the extra space, but it will still be  (n) It is possible to reduce the size of the extra space, but it will still be  (n)

Quick Sort I Split List into “Big” and “Little” keys Put the Little keys first, Big keys second Recursively sort the Big and Little keys LittleBig Pivot Point Quicksort( A, l, r) if l < r then q = partition(A, l, r) quicksort(A, l, q-1) quicksort(A, q+1, r)

Quicksort II Big is defined as “bigger than the pivot point” Big is defined as “bigger than the pivot point” Little is defined as “smaller than the pivot point” Little is defined as “smaller than the pivot point” The pivot point is chosen “at random”. In the following example, we pick up the middle element as the pivot. The pivot point is chosen “at random”. In the following example, we pick up the middle element as the pivot.

Partitioning Pick pivot == 37

Partitioning Step 1: Move pivot to end of array

Partitioning Step 2: set i == 0 and j == array.length - 1 ij

Partitioning Step 3: move i right until value larger than the pivot is found ij

Partitioning Step 4: move j left until value less than the pivot is found ij

Partitioning Step 5: swap elements at positions i and j ij

Partitioning ij Step 6: move i right until value larger than the pivot is found

Partitioning ij Step 7: move j left until value less than the pivot is found

Partitioning ij Step 8: swap elements at positions i and j

Partitioning i j Step 9: move i left until it hits j

Partitioning i j Step 10: put pivot in correct spot

Quicksort III Pivot point may not be the exact median Pivot point may not be the exact median Finding the precise median is hard Finding the precise median is hard If we “get lucky”, the following recurrence applies (n/2 is approximate) If we “get lucky”, the following recurrence applies (n/2 is approximate)

Quicksort IV If the keys are in order, “Big” portion will have n-1 keys, “Small” portion will be empty. If the keys are in order, “Big” portion will have n-1 keys, “Small” portion will be empty. T(N) = T(N-1) + O(N) = O(N 2 ) T(N) = T(N-1) + O(N) = O(N 2 ) N-1 comparisons are done for first key N-1 comparisons are done for first key N-2 comparisons for second key, etc. N-2 comparisons for second key, etc. Result: Result:

A Better Lower Bound The  (n 2 ) time bound does not apply to Quicksort, Mergesort. The  (n 2 ) time bound does not apply to Quicksort, Mergesort. A better assumption is that keys can be moved an arbitrary distance. A better assumption is that keys can be moved an arbitrary distance. However, we can still assume that the number of key-to-key comparisons is proportional to the run time of the algorithm. However, we can still assume that the number of key-to-key comparisons is proportional to the run time of the algorithm.