1. 1 CSE 326: Data Structures: Sorting Lecture 16: Friday, Feb 14, 2003

2. 2 Review: QuickSort procedure quickSortRecursive (Array A, int left, int right) if (left == right) return; int pivot = choosePivot(A, left, right); /* partition A s.t.: A[left], A[left+1], …, A[i]  pivot A[i+1], A[i+2], …, A[right]  pivot */ quickSortRecursive(A, left, i); quickSortRecursive(A, i+1, right); } procedure quickSortRecursive (Array A, int left, int right) if (left == right) return; int pivot = choosePivot(A, left, right); /* partition A s.t.: A[left], A[left+1], …, A[i]  pivot A[i+1], A[i+2], …, A[right]  pivot */ quickSortRecursive(A, left, i); quickSortRecursive(A, i+1, right); }

3. 3 Review: The Partition i = left; j = right; repeat { while (A[i] < pivot) i++; while (A[j] > pivot) j--; if (i<j) { swap (A[i], A[j]); i++; j++;} else break; } i = left; j = right; repeat { while (A[i] < pivot) i++; while (A[j] > pivot) j--; if (i<j) { swap (A[i], A[j]); i++; j++;} else break; } There exists a sentinel A[k]  pivot A[left]…A[i-1]A[i]…A[j]…A[right]  pivot  pivot Why do we need i++, j++ ?

4. 4 Review: The Partition  pivot A[left]…A[j]…A[i]…A[right]  pivot At the end: Q: How are these elements ? A: They are = pivot ! quickSortRecursive(A, left, j); quickSortRecursive(A, i, right); quickSortRecursive(A, left, j); quickSortRecursive(A, i, right);

5. 5 Why is QuickSort Faster than Merge Sort? Quicksort typically performs more comparisons than Mergesort, because partitions are not always perfectly balanced –Mergesort – n log n comparisons –Quicksort – 1.38 n log n comparisons on average Quicksort performs many fewer copies, because on average half of the elements are on the correct side of the partition – while Mergesort copies every element when merging –Mergesort – 2n log n copies (using “temp array”) n log n copies (using “alternating array”) –Quicksort – n/2 log n copies on average

6. 6 Stable Sorting Algorithms Typical sorting scenario: Given N records: R[1], R[2],..., R[N] They have N keys: R[1].A,..., R[N].A Sort the records s.t.:R[1].A  R[2].A ...  R[N].A A sorting algorithm is stable if: If i < j and R[i].A = R[j].A then R[i] comes before R[j] in the output

7. 7 Stable Sorting Algorithms Which of the following are stable sorting algorithms ? Bubble sort Insertion sort Selection sort Heap sort Merge sort Quick sort

8. 8 Stable Sorting Algorithms Which of the following are stable sorting algorithms ? Bubble sortyes Insertion sortyes Selection sortyes Heap sortno Merge sortno Quick sortno We can always transform a non-stable sorting algorithm into a stable one How ?

9. 9 Detour: Computing the Median The median of A[1], A[2], …, A[N] is some A[k] s.t.: –There exists N/2 elements  A[k] –There exists N/2 elements  A[k] Think of it as the perfect pivot ! Very important in applications: –Median income v.s. average income –Median grade v.s. average grade To compute: sort A[1], …, A[N], then median=A[N/2] –Time O(N log N) Can we do it in O(N) time ?

10. 10 Detour: Computing the Median int medianRecursive(Array A, int left, int right) { if (left==right) return A[left];... Partition... if N/2  j return medianRecursive(A, left, j); if N/2  i return medianRecursive(A, i, right); return pivot } Int median(Array A, int N) { return medianRecursive(A, 0, N-1); } int medianRecursive(Array A, int left, int right) { if (left==right) return A[left];... Partition... if N/2  j return medianRecursive(A, left, j); if N/2  i return medianRecursive(A, i, right); return pivot } Int median(Array A, int N) { return medianRecursive(A, 0, N-1); }  pivot A[left]…A[j]…A[i]…A[right]  pivot Why ?

11. 11 Detour: Computing the Median Best case running time: T(N) = T(N/2) + cN = T(N/4) + cN(1 + 1/2) = T(N/8) + cN(1 + 1/2 + 1/4) =... = T(1) + cN (1 + 1/2 + 1/4 + … 1/2 k ) = O(N) Worst case = O(N 2 ) Average case = O(N) Question: how can you compute the median in O(N) worst case time ? Note: it’s tricky.

12. 12 Back to Sorting Naïve sorting algorithms: –Bubble sort, insertion sort, selection sort –Time = O(N 2 ) Clever sorting algorithms: –Merge sort, heap sort, quick sort –Time = O(N log N) I want to sort in O(N) ! –Is this possible ?

13. 13 Could We Do Better? Consider any sorting algorithm based on comparisons Run it on A[1], A[2],..., A[N] –Assume they are distinct At each step it compares some A[i] with some A[j] –If A[i] < A[j] then it does something... –If A[i] > A[j] then it does something else...  Decision Tree !

14. 14 Decision tree to sort list A,B,C Every possible execution of the algorithm corresponds to a root-to-leaf path in the tree.

15. 15 Max depth of the decision tree How many permutations are there of N numbers? How many leaves does the tree have? What’s the shallowest tree with a given number of leaves? What is therefore the worst running time (number of comparisons) by the best possible sorting algorithm?

16. 16 Max depth of the decision tree How many permutations are there of N numbers? N! How many leaves does the tree have? N! What’s the shallowest tree with a given number of leaves? log(N!) What is therefore the worst running time (number of comparisons) by the best possible sorting algorithm? log(N!)

17. 17 Stirling’s approximation At least one branch in the tree has this depth

18. 18 If you forget Stirling’s formula... Theorem: Every algorithm that sorts by comparing keys takes  (n log n) time

19. 19 Bucket Sort Now let’s sort in O(N) Assume: A[0], A[1], …, A[N-1]  {0, 1, …, M-1} M = not too big Example: sort 1,000,000 person records on the first character of their last names: –Hence M = 128 (in practice: M = 27)

20. 20 Bucket Sort int bucketSort(Array A, int N) { for k = 0 to M-1 Q[k] = new Queue; for j = 0 to N-1 Q[A[j]].enqueue(A[j]); Result = new Queue; for k = 0 to M-1 Result = Result.append(Q[k]); return Result; } int bucketSort(Array A, int N) { for k = 0 to M-1 Q[k] = new Queue; for j = 0 to N-1 Q[A[j]].enqueue(A[j]); Result = new Queue; for k = 0 to M-1 Result = Result.append(Q[k]); return Result; } Stable sorting !

21. 21 Bucket Sort Running time: O(M+N) Space: O(M+N) Recall that M << N, hence time = O(N) What about the Theorem that says sorting takes  (N log N) ?? This is not real sorting, because it’s for trivial keys

22. 22 Radix Sort I still want to sort in time O(N): non-trivial keys A[0], A[1], …, A[N-1] are strings –Very common in practice Each string is: c d-1 c d-2 …c 1 c 0, where c 0, c 1, …, c d-1  {0, 1, …, M-1} M = 128 Other example: decimal numbers

23. 23 RadixSort Radix = “The base of a number system” (Webster’s dictionary) –alternate terminology: radix is number of bits needed to represent 0 to base-1; can say “base 8” or “radix 3” Used in 1890 U.S. census by Hollerith Idea: BucketSort on each digit, bottom up.

24. 24 The Magic of RadixSort Input list: 126, 328, 636, 341, 416, 131, 328 BucketSort on lower digit: 341, 131, 126, 636, 416, 328, 328 BucketSort result on next-higher digit: 416, 126, 328, 328, 131, 636, 341 BucketSort that result on highest digit: 126, 131, 328, 328, 341, 416, 636

25. 25 Inductive Proof that RadixSort Works Keys: d-digit numbers, base B –(that wasn’t hard!) Claim: after i th BucketSort, least significant i digits are sorted. –Base case: i=0. 0 digits are sorted. –Inductive step: Assume for i, prove for i+1. Consider two numbers: X, Y. Say X i is i th digit of X: X i+1 < Y i+1 then i+1 th BucketSort will put them in order X i+1 > Y i+1, same thing X i+1 = Y i+1, order depends on last i digits. Induction hypothesis says already sorted for these digits because BucketSort is stable

26. 26 Radix Sort int radixSort(Array A, int N) { for k = 0 to d-1 A = bucketSort(A, on position k) } int radixSort(Array A, int N) { for k = 0 to d-1 A = bucketSort(A, on position k) } Running time: T = O(d(M+N)) = O(dN) = O(Size)

27. 27 Radix Sort 3535535353525232322525 Q[0]Q[1]Q[2]Q[3]Q[4]Q[5]Q[6]Q[7]Q[8]Q[9] 53533353552525 5252323253533353552525 52523232 Q[0]Q[1]Q[2]Q[3]Q[4]Q[5]Q[6]Q[7]Q[8]Q[9] 3232335352525525253535 2525323233535525253535 A=

28. 28 Running time of Radixsort N items, D digit keys of max value M How many passes? How much work per pass? Total time?

29. 29 Running time of Radixsort N items, D digit keys of max value M How many passes? D How much work per pass? N + M –just in case M>N, need to account for time to empty out buckets between passes Total time? O( D(N+M) )

30. 30 Radix Sort What is the size of the input ? Size = DN Radix sort takes time O(Size) !! c D-1 c D-2 …c0c0 A[0]‘S’‘m’‘i’‘t’‘h’ A[1]‘J’‘o’‘n’‘e’‘s’ … A[N-1]

31. 31 Radix Sort Variable length strings: Can adapt Radix Sort to sort in time O(Size) ! –What about our Theorem ?? A[0] A[1] A[2] A[3] A[4]

32. 32 Radix Sort Suppose we want to sort N distinct numbers Represent them in decimal: –Need D=log N digits Hence RadixSort takes time O(DN) = O(N log N) The total Size of N keys is O(N log N) ! No conflict with theory

33. 33 Sorting HUGE Data Sets US Telephone Directory: –300,000,000 records 64-bytes per record –Name: 32 characters –Address: 54 characters –Telephone number: 10 characters –About 2 gigabytes of data –Sort this on a machine with 128 MB RAM… Other examples?

34. 34 Merge Sort Good for Something! Basis for most external sorting routines Can sort any number of records using a tiny amount of main memory –in extreme case, only need to keep 2 records in memory at any one time!

35. 35 External MergeSort Split input into two “tapes” (or areas of disk) Merge tapes so that each group of 2 records is sorted Split again Merge tapes so that each group of 4 records is sorted Repeat until data entirely sorted log N passes

36. 36 Better External MergeSort Suppose main memory can hold M records. Initially read in groups of M records and sort them (e.g. with QuickSort). Number of passes reduced to log(N/M)