1. Sorting
… and Insertion Sort

2. Sorting techniques
Ideally, when we set up a new data collection, we ensure that it is (somehow) ordered:
Initially (easy if it is empty or contains one item)
Each addition ensures that the ordering is preserved (e.g. by inserting in the correct place)
We also need a sorting technique to put jumbled data into order
The data may be completely randomly ordered, or we may know that some order already exists
Different sorting algorithms may perform better in different circumstances

3. Sorting Algorithms
Problem: Given a sequence of N values, rearrange them so that they are in non-decreasing order.
E.g. ascending numerical order, or alphabetical order
‘non-decreasing’ allows for repeat/duplicate values
We restrict ourselves to arrays of numbers
Algorithms for sorting lists:
“Naïve”: Bubble sort, Selection sort
Cleverer: Quick sort
There are many others: Insert sort, Merge sort,...
We'll look at a few of these,
and understand their complexity analysis 1

4. Things to keep in mind…
The data to be sorted may vary in type and be simple or more complex objects
The sorting techniques remain the same.
The result of sorting is simply the rearranged list
No value is "returned", and no exception can be thrown
We will assume the problem consists of:
data is size integers, in elements indexed 0 to (size-1) of array numbers

5. Merge sort

6. Insertion Sort

7. Insertion Sort Descending order.

8. the fourth iteration of this loop is shown here
Insertion Sort
GREEN– sorted, RED unsorted.
while some elements unsorted:
Using linear search, find the location in the sorted portion where the 1st element of the unsorted portion should be inserted
Move all the elements after the insertion location up one position to make space for the new element 45 38 60 60 66 66 45 79 47 13 74 36 21 94 22 57 16 29 81
the fourth iteration of this loop is shown here 38 45 60 66 60 45 66 79 47 13 74 36 21 94 22 57 16 29 81

9. Insert sort in action
Start of algorithm. Sorted portion of list will be marked in colour. 15 3 91 68 2 25 31 32 16 4 21 62 1 5 6 7 8 9 10 11 12
Assign i the value 1.
Set pivot to be equal to numbers[i] (next…)

10. Insert sort in action 3 15 91 68 2 25 31 32 16 4 21 62 i pivot1 3 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? Yes!
Move this value into the gap (next…)

11. Insert sort in action 3 15 91 68 2 25 31 32 16 4 21 62 pivot i1 3 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? No.
Move pivot into the gap.
Add 1 to i, then select the ith value as pivot (next…)

12. Insert sort in action 91 3 15 68 2 25 31 32 16 4 21 62 pivot i1 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? No.
Move pivot into the gap.
Add 1 to i, then select the ith value as pivot (next…)

13. Insert sort in action 68 3 15 91 2 25 31 32 16 4 21 62 pivot i1 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? Yes!
Move this value into the gap (next…)

14. Insert sort in action 68 3 15 91 2 25 31 32 16 4 21 62 pivot i1 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? No.
Move pivot into the gap.
Add 1 to i, then select the ith value as pivot (next…)

15. Insert sort in action 2 3 15 68 91 25 31 32 16 4 21 62 pivot i1 2 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? Yes!
Move this value into the gap (next…)

16. Insert sort in action 2 3 15 68 91 25 31 32 16 4 21 62 pivot i1 2 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? Yes!
Move this value into the gap (next…)

17. Insert sort in action 2 3 15 68 91 25 31 32 16 4 21 62 pivot i1 2 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? Yes!
Move this value into the gap (next…)

18. Insert sort in action 2 3 15 68 91 25 31 32 16 4 21 62 pivot i1 2 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? Yes!
Move this value into the gap (next…)

19. Insert sort in action 2 3 15 68 91 25 31 32 16 4 21 62 pivot i1 2 5 6 7 8 9 10 11 12 i
Is the value before the gap greater than pivot? No.
Move pivot into the gap.
Add 1 to i, then select the ith value as pivot (next…)

20. Insert sort in action 25 2 3 15 68 91 31 32 16 4 21 62 pivot i 1 5 6 71 5 6 7 8 9 10 11 12 i

21. And so on…until the list is finally in order:
pivot 2 3 4 15 16 21 25 31 32 62 68 91 1 5 6 7 8 9 10 11 12 i

22. An insertion sort partitions the array into two regions; variable i referred to as pivot.

23. Algorithm InsertionSort(A) Input: An array ‘A’ of n comparable items
Output: The array ‘A’ with elements in non-decreasing order
for i1 to n-1 do
//Insert A[i] at is proper location in A[0]…A[i-1].
pivot  A[i]
j  i-1
While j >= 0 and A[j] > pivot do
A[j+1]  A[j]
j  j – 1
A[j+1]  pivot

24. An insertion sort of an array of five integers

25. Insertion Sort Demo: Another perspective
Sorting problem (recall):
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort (general idea)
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

26. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

27. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

28. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

29. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

30. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

31. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

32. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

33. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

34. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

35. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

36. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

37. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

38. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

39. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

40. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

41. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

42. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

43. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

44. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

45. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

46. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

47. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

48. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

49. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

50. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

51. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

52. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

53. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

54. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

55. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

56. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

57. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

58. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

59. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

60. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

61. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

62. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

63. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

64. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

65. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

66. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

67. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

68. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

69. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

70. Insertion Sort Demo Sorting problem:
Given an array of N integers, rearrange them so that they are in increasing order.
Insertion sort
Brute-force sorting solution.
Move left-to-right through array.
Exchange next element with larger elements to its left, one-by-one.

71. Insertion Sort: Number of Comparisons
# of Sorted
Elements
Best case
Worst case 1 2 …
n-1
n(n-1)/2
Remark: we only count comparisons of elements in the array.

72. Count the steps InsertionSort(A)
Input: An array ‘A’ of n comparable items
Output: The array ‘A’ with elements in non-decreasing order
for i1 to n-1 do //LOOPS OVER WHOLE ARRAY
//Insert A[i] at is proper location in A[0]…A[i-1].
pivot  A[i]
j  i-1
While j >= 0 and A[j] > pivot do //moves the item.
A[j+1]  A[j]
j  j – 1
A[j+1]  pivot
outer loop
outer steps
inner loop
inner steps
outer step

73. Insertion Sort: Cost Function
1 operation to initialize the outer loop
The outer loop is evaluated n-1 times
5 instructions (including outer loop comparison and increment)
Total cost of the outer loop: 5(n-1)
How many times the inner loop is evaluated is affected by the state of the array to be sorted, so may vary
Best case: the array is already completely sorted, so no “shifting” of array elements is required.
We only test the condition of the inner loop once (2 operations = 1 comparison + 1 element comparison), and the body is never executed
Requires 2(n-1) operations, ie. O(n).

74. Insertion Sort: Cost Function
Worst case: the array is sorted in reverse order (so each item has to be moved to the front of the array)
In the i-th iteration of the outer loop, the inner loop will perform 4i+1 operations
Therefore, the total cost of the inner loop will be 2n(n-1)+n-1, ie. O(n2)
Time cost:
Best case: 7(n-1)
Worst case: 5(n-1)+2n(n-1)+n-1
What about the number of moves?
Best case: 2(n-1) moves
Worst case: 2(n-1)+n(n-1)/2
Aside: Where are the dominant terms, above?

75. Insertion Sort: Average Case
Is it closer to the best case (n comparisons)?
Is it closer to the worst case (n * (n-1) / 2) comparisons?
It turns out that when random data is sorted, insertion sort is usually closer to the worst case
Around n * (n-1) / 4 comparisons
Calculating the average number of comparisons more exactly would require us to state assumptions about what the “average” input data set looked like
This would, for example, necessitate discussion of how items were distributed over the array
Exact calculation of the number of operations required to perform even simple algorithms can be challenging!

76. Now let’s compare against something else…

77. Selection Sort 5 1 3 4 6 2
Comparison
Data Movement
Sorted

78. Selection Sort 5 1 3 4 6 2
Comparison
Data Movement
Sorted

79. Selection Sort 5 1 3 4 6 2
Comparison
Data Movement
Sorted

80. Selection Sort 5 1 3 4 6 2
Comparison
Data Movement
Sorted

81. Selection Sort 5 1 3 4 6 2
Comparison
Data Movement
Sorted

82. Selection Sort 5 1 3 4 6 2
Comparison
Data Movement
Sorted

83. Selection Sort 5 1 3 4 6 2
Comparison
Data Movement
Sorted

84. Selection Sort 5 1 3 4 6 2 
Smallest
Comparison
Data Movement
Sorted

85. Selection Sort 1 5 3 4 6 2
Comparison
Data Movement
Sorted

86. Selection Sort 1 5 3 4 6 2
Comparison
Data Movement
Sorted

87. Selection Sort 1 5 3 4 6 2
Comparison
Data Movement
Sorted

88. Selection Sort 1 5 3 4 6 2
Comparison
Data Movement
Sorted

89. Selection Sort 1 5 3 4 6 2
Comparison
Data Movement
Sorted

90. Selection Sort 1 5 3 4 6 2 
Smallest
Comparison
Data Movement
Sorted

91. Selection Sort 1 5 3 4 6 2 
Smallest
Comparison
Data Movement
Sorted

92. Selection Sort 1 2 3 4 6 5
Comparison
Data Movement
Sorted

93. Selection Sort 1 2 3 4 6 5
Comparison
Data Movement
Sorted

94. Selection Sort Algorithm
Input: An array ‘A’ of n comparable items
Output: The array ‘A’ with elements in non-decreasing order
for i0 to n-2 do // note n-2 not n-1
//Insert smallest item in 0th slot, 2nd smallest in 1st, etc.
min  i
for j  i+1 to j <= n-1 do
if (A[j] < A[min])
min  j
swap A[i] and A[min]. // What is really happening here?
What is the complexity of Selection Sort, and does it vary with input?