2. Sorting algorithms General problem description: Given array A[0..n  1] with n elements that can be compared directly (i.e. for each i and j either A[i] < A[j], or A[i] = A[j], or A[i] > A[j]), rearrange elements of A in such a way that A[0]  A[1]  A[2]  …  A[n  1].

3. Sorting algorithms To understand algorithms it is sufficient to assume that elements of array A are just integers. In practice elements of array A usually will be more complex and will have to be sorted according to some particular property (Key), e.g. (in C++) they may be defined as classes class Item { int Key;  };

4. Sorting algorithms If elements of A are complex (use large amount of memory) it may be more efficient to construct new array B of keys, sort it, and then rearrange items in initial array according to the order of keys. Actually elements of B must contain also the information about the initial position of keys, i.e. they should be pairs (Key,Position).

5. Sorting algorithms procedure InsertionSort(array A[0..n  1]): for i from 1 to n  1 do j  i x  A[i] while j  1 and A[j  1] > x do A[j]  A[j  1] j  j  1 A[j]  x Insertion Sort

6. Sorting algorithms 82497 24789 24897 24897 28497 Insertion Sort - Example

7. Sorting algorithms Insertion Sort - Example 2 [Adapted from T.Niemann]

8. Sorting algorithms procedure InsertionSort(array A[0..n  1]): for i from 1 to n  1 do j  i x  A[i] while j  1 and A[j  1] > x do A[j]  A[j  1] j  j  1 A[j]  x n  1 0..i Insertion Sort - Complexity

9. Sorting algorithms Time (number of operations) – worst case: T(n) = (c 1 + c 2 ) + (c 1 + 2c 2 ) + … + (c 1 + (n  1) c 2 ) T(n) = n c 1 + (1/2 n (n  1)) c 2  T(n) =  (n 2 ) Space (memory) – worst case: M(n) = n + const Insertion Sort - Complexity

10. Sorting algorithms Average complexity? For space – obviously the same: M(n) = n + const For time – it can be shown that  (n 2 ) operations are required also in average case. Insertion Sort - Complexity

11. Sorting algorithms Insertion Sort - Complexity (empirical) [Adapted from M.Lamont]

12. Sorting algorithms procedure BubbleSort(array A[0..n  1]): for i from 1 to n do for j from 1 to n  1 do if A[j] > A[j  1] then A[j]  A[j  1] Bubble Sort (version 1)

13. Sorting algorithms 82497 24879 24789 24789 24789 24789 Bubble Sort - Example

14. Sorting algorithms procedure BubbleSort(array A[0..n  1]): for i from 1 to n do for j from 1 to n  1 do if A[j] > A[j  1] then A[j]  A[j  1] n n1n1 Bubble Sort - Complexity

15. Sorting algorithms Worst case: T(n) = n(n –1) const =  (n 2 ) M(n) = n + const Average complexities are the same as in worst case Bubble Sort Complexity

16. Sorting algorithms Bubble Sort Complexity (empirical) [Adapted from M.Lamont]

17. Sorting algorithms procedure BubbleSort(array A[0..n  1]): stop = 0 while stop = 0 do stop = 1 for j from 1 to n  1 do if A[j] > A[j  1] then stop = 0 A[j]  A[j  1] Bubble Sort (version 2)

18. Sorting algorithms Simple sorting algorithms - empirical complexity [Adapted from M.Lamont]

19. Sorting algorithms Merge Sort - Idea [Adapted from H.Lang]

20. Sorting algorithms procedure MergeSort(array A[l..r], B[l..r]): if l < r then m   l + (r  l)/2  MergeSort(A[l..m], B[l..m]) MergeSort(A[m+1..r], B[m+1..r]) Merge(B[l..r], A[l..m], A[m+1..r]) A[l..r]  B[l..r] Merge Sort

21. Sorting algorithms procedure Merge(array B[l..r], A[l..m], A[m+1..r]) apos  l; bpos  l; cpos  m + 1 while bpos  m and cpos  r do if A[bpos] < A[cpos] then B[apos]  A[bpos]; bpos  bpos + 1; apos  apos + 1 else B[apos]  A[cpos]; cpos  cpos + 1; apos  apos + 1 while bpos  m do B[apos]  A[pbos]; bpos  bpos + 1; apos  apos + 1 while cpos  r do B[apos]  A[cbos]; cpos  cpos + 1; apos  apos + 1 Merge Sort

22. Sorting algorithms 82497 82497 824 97 7 9 9 7 79 42 8 47928 24789 Merge Sort - Example

23. Sorting algorithms Worst case: Merge procedure: T m (n) = n const =  (n) MergeSort procedure: T(n) = 2 T(n/2) + const + T m (n) = 2 T(n/2) +  (n) =  (n log n) Merge Sort - Complexity

24. Sorting algorithms Worst case: MergeSort procedure: T(n) = 2 T(n/2) + const + T m (n) = 2 T(n/2) +  (n) T(n) =  (n log n) M(n) = 2 n + const (can be impoved up to 3/2 n + const) Average complexities are the same as in worst case Merge Sort - Complexity

25. Sorting algorithms Quick Sort Sort - Idea [Adapted from H.Lang]

26. Sorting algorithms procedure QuickSort(array A[0..n  1]): if n > 1 then x  A[0] (or i  Random(n); x  A[i]) bpos  0; cpos  0 for i from 0 to n  1 do if x > A[i] then B[bpos] = A[i]; bpos  bpos + 1 if x  A[i] then C[cpos] = A[i]; cpos  cpos + 1 QuickSort(B[0..bpos  1]); QuickSort(C[0..cpos  1]) Append(x,A[0..n  1], B[0..bpos  1], C[0..cpos  1]) Quick Sort (not very efficient version)

27. Sorting algorithms procedure Append(x,A[0..n  1], B[0..bpos  1], C[0..cpos  1]) for i from 0 to bpos  1 do A[i]  B[i] A[bpos] = x for i from 0 to cpos  1 do A[bpos+i+1]  B[i] Quick Sort (not very efficient version)

28. Sorting algorithms Quick Sort - Example [Adapted from T.Niemann]

29. Sorting algorithms 82497 2479 7 7 47 247 24789 8 47 2 4 4 2 9 8 Quick Sort - Example 2

30. Sorting algorithms Worst case: Append procedure: T a (n) = n const =  (n) QuickSort procedure: T(n) = T(bpos) + T(cpos) +  (n) + T a (n) = = T(bpos) + T(cpos) +  (n) Quick Sort - Complexity

31. Sorting algorithms QuickSort procedure: T(n) = T(bpos) + T(cpos) +  (n) Case 1: bpos = cpos T(n) = 2 T(n/2) +  (n) Case 2: bpos = 1, cpos = n – 1 T(n) = T(1) + T(n – 1) +  (n) =  (n log n) =  (n 2 ) Quick Sort - Complexity

32. Sorting algorithms It can be shown that the worst situation is in Case 2 (or with bpos and cpos reversed), thus QuickSort worst case time complexity is T(n) =  (n 2 ) Quick Sort - Complexity

33. Sorting algorithms Good news: Average time complexity for QuickSort is  (n log n) Quick Sort - Complexity This (inefficient) version uses 2n + const of space, but that can be improved to n + const On average QuickSort is practically the fastest sorting algorithm

34. Sorting algorithms Quick Sort - Complexity [Adapted from K.Åhlander]

35. Sorting algorithms Quick Sort - Complexity [Adapted from K.Åhlander]

36. Sorting algorithms Quick Sort - Complexity [Adapted from K.Åhlander]

37. Sorting algorithms Quick Sort - Complexity [Adapted from K.Åhlander]

38. Sorting algorithms Quick Sort - Complexity [Adapted from K.Åhlander]

39. Sorting algorithms procedure QuickSort(array A[l..r]): if l < r then i  l; j  r + 1; x  A[l] (or select x randomly) while i < j do i  i + 1 while i  r and A[i] < x do i  i + 1 j  j – 1 while j  l and A[j] > x do j  j – 1 A[i]  A[j] A[i]  A[j]; A[j]  A[l] QuickSort(A[l..j–1]) QuickSort(A[j+1..r]) Quick Sort (efficient version)

40. Sorting algorithms ij After selecting of x move pointers i and j from start and array correspondingly and swap all pairs of elements for which A[i] > x and A[j] < x Quick Sort (efficient version)

41. Sorting algorithms Keys:9 111171318412145 >><>>< after 1  while:9151713184121411 ><<<><<< after 2  while:9154131817121411 <><< after 3  while: 9151341817121411 2 extra swaps:4159131817121411 sort sub-arrays:1459111213141718 Positions:0123456789 Quick Sort (efficient version) - example

42. Sorting algorithms Complexity comparison Tw(n)Tw(n)Ta(n)Ta(n)M(n) InsertionSort (n2)(n2) (n2)(n2) n BubbleSort (n2)(n2) (n2)(n2) n MergeSort  (n log n) 3/2 n QuickSort (n2)(n2)  (n log n) n

43. Sorting algorithms h 1 = 1 < h 2 < h 3 < h 4 < h 5 <  - an increasing sequence of integers For given n select some h k < n. Repeatedly for inc = h k, h k-1, , h 1 = 1 and for s from 0 to inc – 1 InsertionSort the sequences A[s], A[s + inc],A[s + 2 inc], A[s + 3 inc],  Shell Sort - Idea

44. Sorting algorithms Shell Sort - Idea [Adapted from H.Lang]

45. Sorting algorithms procedure ShellSort(array A[0..n–1]): inc  InitialInc(n) while inc  1 do for i from inc to n – 1 do j  i x  A[i] while j  inc and A[j–inc] > x do A[j]  A[j–inc] j  j – inc A[j]  x inc  NextInc(inc,n) Shell Sort

46. Sorting algorithms Shell Sort - Example [Adapted from T.Niemann]

47. Sorting algorithms Keys:9 111171318412145 sort for inc = 5:91111713 18 412145 9111145 18 4121713 sort for inc = 3:9111145 18 4121713 411195 17 13121814 sort for inc = 1: 411195 17 13121814 145911 12 13141718 Positions:0123456789 Shell Sort - Example 2

48. Sorting algorithms sequence 1, 2, 4, 8,  (h i = 2 i-1 ) is not particularly good Shell Sort - Good increment sequences? the “best” increment sequence is not known sequence 1, 4, 13, 40, 121,  (h i = (3 i – 1)/2 ) is one (of several known) that gives good results in practice

49. Sorting algorithms Shell Sort - Complexity for sequence 1, 2, 4, 8,  we have T(n) =  (n 2 ) it is known that for sequence 1, 4, 13, 40, 121,  T(n) = O(n 1.5 ), but in practice it tends to perform comparably with n log n time algorithms there exists sequences for which it has been proven that T(n) = O(n (log n) 2 ) in general the finding of time complexity for a given increment sequence is very difficult

50. Sorting algorithms Heap Sort - procedure "Heapify" [Adapted from T.Giang]

51. Sorting algorithms Heap Sort - procedure "Heapify" [Adapted from T.Giang]

52. Sorting algorithms procedure Heapify(array A[i..j]): if RC(j)  i and A[RC(j)]  A[LC(j)] and A[RC(j)] < A[j] then A[j]  A[RC(j)] Heapify(A[i.. RC(j)]) else if LC(j)  i and A[LC(j)] < A[j] then A[j]  A[LC(j)] Heapify(A[i.. LC(j)]) Here LC(j) = 2j – n (left child) and RC(j) = 2j – n – 1 (right child) Heap Sort - procedure "Heapify"

53. Sorting algorithms Heap Sort - building a heap [Adapted from T.Giang]

54. Sorting algorithms Heap Sort - building a heap [Adapted from T.Giang]

55. Sorting algorithms procedure InitialiseHeap(array A[0..n–1]): for i from 1 to n – 1 do Heapify(A[0..i]) Heap Sort - building a heap

56. Sorting algorithms procedure HeapSort(array A[0..n–1]): InitialiseHeap(A[0..n–1]) for i from 0 to n – 2 do A[i]  A[n–1] Heapify(A[i+1..n–1]) Heap Sort

57. Sorting algorithms Keys:9 111171318412145 InitialiseHeap:1814111713951241 1  Heapify:1141817139111254 2  Heapify:1418171314111295 3  Heapify:1451713141812119 4  Heapify:1459131718121411 5  Heapify:1459111718131412 6  Heapify:1459111218171413 Positions:0123456789 Heap Sort - Example

58. Sorting algorithms 7  Heapify:1459111213171814 8  Heapify:1459111213141817 9  Heapify:1459111213141718 6  Heapify:1459111218171413 Heap Sort - Example

59. Sorting algorithms Worst case: Heapify procedure: T h (n) =  (log n) InitialiseHeap procedure: T i (n)  (n – 1) T h (n)  T i (n) = O(n log n) Heap Sort - Complexity

60. Sorting algorithms InitialiseHeap procedure - a closer look: T i (n) =  h = 1..log n h (n/2 h )   n  h = 1..  (h/2 h ) Heap Sort - Complexity

61. Sorting algorithms 1  1/2 + 2  1/4 +3  1/8 + 4  1/16 + 5  1//32 +  = = 1/2 + 1/4 +1/8+ 1/16+ 1/32+ 1/64 +  + 1/4 +1/8+ 1/16+ 1/32+ 1/64 +  +1/8+ 1/16+ 1/32+ 1/64 +  + 1/16+ 1/32+ 1/64 +  + 1/32+ 1/64 +  Heap Sort - Complexity = 1 = 1/2 = 1/4 = 1/8 = 1/16  = 2

62. Sorting algorithms InitialiseHeap procedure - a closer look: T i (n) =  h = 1..log n h (n/2 h )   n  h = 1..  (h/2 h ) Heap Sort - Complexity = 2 n T i (n) > 1/2 n T i (n) =  (n)

63. Sorting algorithms HeapSort procedure: T(n)  T i (n) + ((n – 1)  (log n) = =  (n) +  (n log n) =  (n log n) M(n) = n + const Heap Sort - Complexity

64. Sorting algorithms Complexity comparison Tw(n)Tw(n)Ta(n)Ta(n)M(n) InsertionSort (n2)(n2) (n2)(n2) n BubbleSort (n2)(n2) (n2)(n2) n MergeSort  (n log n) 3/2 n QuickSort (n2)(n2)  (n log n) n ShellSort  (n log 2 n) n HeapSort  (n log n) n

65. Sorting algorithms Sorting algorithms - empirical complexity [Adapted from M.Lamont]

66. Sorting algorithms Best complexity? Is it possible to design a sorting algorithm with time complexity T(n) = o(n log n)? Apparently T(n) =  (n) Can we have T(n) =  (n)?

67. Sorting algorithms Complexity - the theoretical lower bound Theorem Any comparison- based sorting algorithm has the worst case time complexity T(n) =  (n log n).

68. Sorting algorithms Complexity - the theoretical lower bound

69. Sorting algorithms Complexity - the theoretical lower bound There are n! possible different permutation of n elements (if all of them are different). There are 2 k possible outcomes of k comparisons, i.e. with k comparisons we can distinguish up to 2 k different permutations. Therefore we must have 2 k  n! k =  (n log n)  T(n) =  (n log n)

70. Sorting algorithms Digital sorting In this case there is some additional information available about the data to be sorted (in addition to the information that will be extracted by comparisons) CountingSort -keys to be sorted are integers from the range 0..k – 1 RadixSort - keys to be sorted are integers of restricted length, i.e. they are using no more than m “units” of memory each

71. Sorting algorithms Counting Sort procedure CountingSort(array A[0..n–1], B[0..n–1], C[0..k–1], k): for i from 0 to k – 1 do C[i]  0 for i from 0 to n – 1 do C[A[i]]  C[A[i]] + 1 for i from 1 to k – 1 do C[i]  C[i] + C[i – 1] for i from n – 1 downto 0 do B[C[A[i]] – 1]  A[i] C[A[i]]  C[A[i]] – 1

72. Sorting algorithms Counting Sort - Example k = 3 A B C 21201 21201000 21201122 21201135 2120101122 135

73. Sorting algorithms Counting Sort - Complexity T(n) =  (k) +  (n) +  (k) +  (n) =  (k + n) M(n) = 2 n + k + const CountingSort is stable – if two keys are equal, they will be in the same order after sorting in which they were before

74. Sorting algorithms Counting Sort - Complexity (empirical) [Adapted from V.Zabrodsky]

75. Sorting algorithms Radix Sort - where it came from [Adapted from D.Luebke]

76. Sorting algorithms Radix Sort - Idea [Adapted from K.Houten]

77. Sorting algorithms Radix Sort procedure RadixSort(array A[0..n–1], m): for i from 1 to m do StableSort array A[0..n–1] by using as keys i-th lowest “units” of memory

78. Sorting algorithms Radix Sort - Example 12 84 82 51 23 34 12 23 34 51 82 84 51 12 82 23 84 34

79. Sorting algorithms Radix Sort - Example 2 [Adapted from K.Houten]

80. Sorting algorithms Radix Sort needs stable sorting 12 84 82 51 23 34 12 23 34 51 84 82 51 12 82 23 84 34

81. Sorting algorithms Bucket sort Problem Sort n numbers that are uniformly distributed over the interval [0,1). The method for solving this problem can be also applied to sorting uniformly distributed integers from interval [1,k] (consider integer i as a number i/k).

82. Sorting algorithms Bucket sort - Idea [Adapted from P.Berenbrink]

83. Sorting algorithms Bucket sort - Idea A  [0,1) B 1  [0,1/k) B 2  [1/k,2/k) B k  [k–1/k,1)

84. Sorting algorithms Bucket Sort procedure BucketSort(array A[0..n–1]): for i from 0 to n–1 do insert A[i] into list B[  n A[i]  ] for i from 0 to n–1 do sort B[i] with InsertionSort concatenate the lists B[0], B[1],..., B[n–1] together in order

85. Sorting algorithms Bucket Sort - Example [Adapted from P.Berenbrink]

86. Sorting algorithms Bucket Sort - Complexity Worst case: T(n) =  (n 2 ) Average case: T(n) = n  (E[n i 2 ]) E[n i 2 ] = ? E[n i 2 ] = D[n i ] + E 2 [n i ] E 2 [n i ] = 1 D[n i ] = ?

87. Sorting algorithms Bucket Sort - Binomial distribution The probability that a particular bucket will contain k numbers corresponds to that given by binomial distribution (where p = 1/n): [Adapted from T.Cormen, C.Leiserson, R. Rivest]

88. Sorting algorithms Bucket Sort - Binomial distribution For p = 1/n we have E[X] = n  1/n = 1 [Adapted from T.Cormen, C.Leiserson, R. Rivest]

89. Sorting algorithms Bucket Sort - Binomial distribution X i - a random variable describing the success of i-th trial (success of i-th digit falling in this particular bucket) X i have values 0 or 1 E[X i ] = p and E[X i 2 ] = p D[X i ] = E[X i 2 ] – E 2 [X i ] = p – p 2 = p(1–p)

90. Sorting algorithms Bucket Sort - Binomial distribution X i - a random variable describing the success of i-th trial (success of i-th digit falling in this particular bucket) D[X i ] = p(1–p) Here Var[X] denotes the same as D[X], q = 1–p For p = 1/n we have D[X] = n  1/n  (1–1/n) = 1–1/n [Adapted from T.Cormen, C.Leiserson, R. Rivest]

91. Sorting algorithms Bucket Sort - Complexity Worst case: T(n) =  (n 2 ) Average case: T(n) = n  (E[n i 2 ]) E[n i 2 ] = D[n i ] + E 2 [n i ] = 1– 1/n + 1 2 = 2 – 1/n T(n) = n  (2 – 1/n) = n  (1) =  (n)