1. Simple Sorting Methods: Bubble, Selection, Insertion, Shell

2. Outline Importance of sorting Sorting algorithms bubble sort
insertion sort
selection sort
shell sort

3. Searching Want to know if collection contains value
keep looking until find it…
…or determine it isn’t there
Unsorted collections
linear search: O(N)
Sorted collections
binary search: O(log N)

4. O(N) vs. O(log N) Small differences at small numbers
3N vs. 6 log N for N = 30?
30 vs. 42
Large differences at large numbers
3N vs. 6 log N for N = 1,000,000?
3,000,000 vs. 120
Will eventually overcome any amount of overhead (3N vs log N?)

5. Some Simple Sorting Methods
Bubble sort
“swap” items that are out of order
Selection sort
“select” items and place in their final position
Insertion sort
“insert” items into a sorted sub-array
Shell sort
“insert” using longer distances, then shorter

6. Bubble Sort Simplest algorithm:
look at items next to each other
if they’re out of order, swap them
keep going until everything in order
Items percolate toward their final positions
big items move down quickly
small items move up slowly

7. Bubble Sort Start at low end For each pair in the list…
if lower item is bigger, swap the pair
biggest item will move to the end
repeat until no more moves 6
100 3 –2 8 6
100 3 3
100 –2 8
100 –2 8
100
100 3 6 –2 –2 –2 3 3 3 6 –2 6 6 8 8
100

8. Bubble Sort to BubbleSort( List a ) for i  Length(a) down to 1
for j  1 .. i - 1
if (a[j-1] > a[j])
Swap(a[j], a[j-1]);

9. Bubble Sort Complexity
for i  Length(a) down to 1
for j  1 .. i - 1
if (a[j-1] > a[j]) // 1 comparison
Swap(a[j], a[j-1]); // 3 assignments
N i-1
∑ ∑ (1 + (0 or 3))  best 1, worst 4, average 2.5
i=1 j=0
N i N
∑ ∑ 2.5 = 2.5 ∑ i = 2.5(N)(N+1)/2 = 1.25(N2 + N)
i=1 j=1 i=1

10. Bubble Sort Considered Bad
Not generally very good
lots of work moving things around
Can be improved
track where last swap took place
everything above there is sorted
Improved version OK if list almost sorted
(Head-2-Head sorting link uses this version)
just need to know that ahead of time

11. Exercise
Show the evolution of the following array under bubble sort. Show the result after each (top-level) pass thru the array
[15, 3, 21, 45, 7]

12. Selection Sort Based on “copying” a list in order
select the smallest item in the list
add it to end of new list
cross it off the old list
But do it all in one array
front part of list is sorted
won’t need to be changed again – everything left “unsorted” is bigger

13. Selection Sort Start at low end For each position in the list…
find the smallest unsorted element
swap it into position 6
100 3 –2 8 6 –2 6 3
100
100
100 3 6 8 –2
100 8
100 3 8 –2 6

14. Selection Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
SelectionSort( lst );
lst ==
1st pass:
find smallest item in list
swap it with first item in list
i.e. swap -2 with 6

15. Selection Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
SelectionSort( lst ); _
lst ==
lst == [-2]
2nd pass:
find smallest remaining item in rest of list
swap it into the second position
i.e. swap 3 with 100

16. Selection Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
SelectionSort( lst );
lst ==
lst == [-2]
lst == [ ]

17. Selection Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
SelectionSort( lst );
lst ==
lst == [-2]
lst == [ ]
lst == [ ]

18. Selection Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
SelectionSort( lst );
lst ==
lst == [-2]
lst == [ ]
lst == [ ]
lst == [ ] 100

19. Exercise
Show the evolution of the following array under selection sort. Show the result after each (top-level) pass
[15, 3, 21, 45, 7]

20. Finding the Smallest
Each pass requires finding the smallest item remaining unsorted
need its location in the array, so we can swap it
on ith pass search locations i–1 .. N–1
can start assuming first of those is smallest

21. Selection Sort to SelectionSort( List a ) for i  0 .. Length(a) – 2
p  i;
for j  i Length(a) – 1
if (a[j] < a[p])
p  j;
Swap(a[p], a[i]);

22. Exercise
List all the comparisons made between array elements while selection-sorting the following list:
[2, 8, 12, 4, 9]

23. Work in Selection Sort to SelectionSort( List a )
for i  1 .. Length(a) – 1
p  i;
for j  i Length(a)
if (a[j] < a[p]) p  j;
temp  a[p];
a[p]  a[i];
a[i]  temp;
Comparison
Assignment

24. Counting Ops in Selection Sort
Outer loop runs i from 1 to N – 1
inner runs j from i+1 to N
WSel,worst(N) = i=1SN–1 (3 + j=i+1SN 1)
WSel,worst(N) = i=1SN–1 (3 + j=1SN 1 – j=1Si 1)
WSel,worst(N) = i=1SN–1 (3 + N – i)
WSel,worst(N) = i=1SN–1 (3 + N) – i=1SN–1 i
WSel,worst(N) = (N – 1)(3 + N) – (N – 1)(N)/2
WSel,worst(N) = N2 + 2N – 3 – (N2 – N)/2
WSel,worst(N) = (N2 + 5N – 6)/2

25. Magnitude of Selection Sort
Average case same as worst case
second smallest item could be anywhere
always need to look at every remaining element
Worst case:
WSel,worst(N) = (N2 + 5N – 6)/2 = O(N2)
Average case:
WSel,ave(N) = (N2 + 5N – 6)/2 = O(N2)

26. Insertion Sort Based on inserting into an ordered list
For when you’re creating a list
Keep it sorted right from the start
Insert each item into its proper place...
...shifting other items up as required
But all items already in the array
“split” array into sorted list part and input part

27. Insertion Sort Start at low end For each unsorted element…
find its place
make space
insert it 6
100 3 –2 8 –2 6 6
100 3 6
100 3 6
100
100 3
100 8 –2 8
100 6 8 3 –2

28. Insertion Sort, Top Level Loop
Multiple passes thru the list
At start of ith pass, first i items are in order
list of length 1 always “in order”
At end of ith pass, first i+1 items in order
thus need only N – 1 passes in all
On ith pass, “insert” i+1st item into sorted part of list

29. Insertion Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
InsertionSort( lst );
_0 _
lst == [ 6]
1st pass:
list from 0 to 0 is sorted
insert item 1 into the sorted part of the list
i.e. insert 100 into [6]

30. Insertion Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
InsertionSort( lst );
_0 _
lst == [ 6]
lst == [ ]
2nd pass:
list from 0 to 1 is sorted
insert item 2 into the sorted part of the list
i.e. insert 3 into [6, 100]

31. Insertion Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
InsertionSort( lst );
_0 _
lst == [ 6]
lst == [ ]
lst == [ ]

32. Insertion Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
InsertionSort( lst );
_0 _
lst == [ 6]
lst == [ ]
lst == [ ]
lst == [ ] 8

33. Insertion Sort Top-Level Trace
lst  [6, 100, 3, -2, 8];
InsertionSort( lst );
_0 _
lst == [ 6]
lst == [ ]
lst == [ ]
lst == [ ] 8
lst == [ ]

34. Exercise
Show the evolution of the following array under insertion sort. Show the result after each (top-level) pass
[15, 3, 21, 45, 7]
[17, 4, 8, 3]

35. Insertion on Pass i Move i+1st item into temporary storage
Starting at the “end” of the sorted part…
…move items up until you find where “new” item goes
find a smaller item
fall off the front of the list

36. Insertion on Pass 2 Trace
lst == [6, 100], 3, -2, 8
Insert 3 into [6, 100]
temp
lst == [ ] 3
lst == [ ] 3
lst == [ ] 3
lst == [ ] 3

37. Insertion on Pass 2 Trace
lst == [-2, 3, 6, 100], 8
Insert 8 into [-2, 3, 6, 100]
temp
[ ] 8
[ ] 8
[ ] 8

38. Insertion Sort Pseudo-Code
to InsertionSort( List a )
for i  0 .. Length(a) – 2
p  i + 1
temp  a[p];
while (p > 0 && a[p – 1] > temp)
a[p]  a[p – 1];
p  p – 1;
a[p]  temp;

39. Exercise
Show the evolution of the following array under insertion sort. Show the result after each assignment into the array (that is, inner as well as outer loops):
[17, 4, 8, 3]

40. Work in Insertion Sort to InsertionSort( List a )
for i  1 .. Length(a) – 1
p  i + 1
temp  a[p];
while (p > 1 && a[p – 1] > temp)
a[p]  a[p – 1];
p  p – 1;
a[p]  temp;
Comparison
Assignment

41. Counting Ops in Insertion Sort
Let N be the size of the array/vector
number of elements in the list
Outer loop iterates N – 1 times
contains two assignments
Inner loop iterates at most i times
list was in reverse order – need to compare all
contains one comparison & one assignment

42. Counting Ops in Insertion Sort
Outer loop runs i from 1 to N – 1
inner runs p from i+1 down to 2 (“worst case”)
WIns,worst(N) = i=1SN–1 (2 + p=2Si+1 2)
WIns,worst(N) = i=1SN–1 (2 + 2i)
WIns,worst(N) = 2(N – 1) + 2(N – 1)(N)/2
WIns,worst(N) = 2N – 2 + N2 – N
WIns,worst(N) = N2 + N – 2

43. Average Case Analysis
On average, we expect inner loop of insertion to only go ½ way to the front
WIns,ave(N) = i=1SN–1 (2 + ½(2i))
WIns,ave(N) = 2(N–1) + (N–1)(N)/2
WIns,ave(N) = 2N – 2 + (N2 – N)/2
WIns,ave(N) = (N2 + 3N – 4)/2

44. Comparing Selection & Insertion
WSel,worst(N) = (N2 + 5N – 6)/2
WIns,worst(N) = N2 + N – 2
N = 1000 gives:
Selection sort: 502,987 operation
Insertion sort: 1,000,998 operations
Selection looks faster
but that’s worst-case time for Insertion sort

45. Average Selection & Insertion
WSel,ave(N) = (N2 + 5N – 6)/2
WIns,ave(N) = (N2 + 3N – 4)/2
N = 1000 gives:
Selection sort: 502,987 operation
Insertion sort: 501,498 operations
They do about the same amount of work
and both are O(N2)

46. Exercise
Show the evolution of each of the following arrays under both insertion and selection sorts
[99, 14, 21, 12, 5]
[3, 7, 14, 5, 2, 8]
[9, 8, 7, 6, 5, 4, 3, 2, 1]

47. Shell Sort Sort far apart items first
then sort the ones that are closer together
Sort items separated by a given “gap”
every fifth element, for example
Use any sorting method
insertion sort, e.g.
Reduce the gap size & repeat
stop when you’ve sorted with a gap of 1

48. Shell Sort: Gap = 5 81 94 11 32 12 24 17 29 28 18 41 77 75 15 81 24
Start at location 6
Up by 1 each time
Use insertion sort
Swap 24 81 94 17
Swap 17 94 11 OK 29 32 28
Swap 28 32 12 OK 18

49. Shell Sort: Gap = 5 (cont)24 17 11 28 12 81 94 29 32 18 41 77 75 15 81 41
Swap 24 OK 41 81 94 77
Swap 17 OK 77 94 29 OK 75 32 15
Swap 28 15 32
Swap 15 28

50. Shell Sort: Gap = 5 (cont)24 17 11 15 12 41 77 29 28 18 81 94 75 32
List is now “5-sorted” 41 24 81 77 17 94 11 75 29 15 32 28 12 18

51. Shell Sort: Gap = 3 Continue with a smaller gap 24 17 11 15 12 41 7729 28 18 81 94 75 32
Continue with a smaller gap 24 15 77 18 75
Swap OK 17 12 29 81 32
Swap OK 11 41 28 94 OK
Swap (41)

52. Shell Sort: Gap = 3 (cont)24 28 17 18 11 12 15 81 77 94 32 75 41 29 24 17 11 15 12 41 77 29 28 18 81 94 75 32
List is now 3-sorted 15 18 24 75 77 12 17 29 32 81 11 28 41 94

53. Shell Sort: Gap = 1 Now it’s just normal insertion sort15 12 11 18 17 28 24 29 41 75 32 94 77 81
Now it’s just normal insertion sort
but everything’s “pretty close” to where it’s going to end up
Number of assignments:
shell sort: = 41
insertion sort: 57 15 12 11 18 17 28 24 41 29 75 94 32 77 81

54. Gap Sizes Any descending sequence will do Some sequences:
so long as it ends at 1
Some sequences:
N/2, N/4, N/8, …, 1 not especially good
A = 2log N – 1, A/2, A/4, …, 1
N/3, N/9, N/27, …, 1
… other sequences may be even better

55. Gap Sizes N = 13 insertion sort = 52 comps, 65 stores
1st gap sequence: 6, 3, 1 41 comps, 77 stores
2nd gap sequence: 7, 3, 1 46 comps, 81 stores
3rd gap sequence: 4, 1 37 comps, 63 stores
N = 130 (random values) 4487c, 4618s
1st: 65, 32, 16, 8, 4, 2, c, 2236s
2nd: 127, 63, 31, 15, 7, 3, c, 1792s
3rd: 43, 14, 4, c, 1592s 17 24 12 32 11 94 81 15 77 41 18 28 29

56. Shell Sort to ShellSort( List a ) gap  Length(a)  2; /* or … */
while (gap  1)
if (gap % 2 == 0) ++gap; /* much better! */
for i  gap .. Length(a)–1
p  i, temp  a[p];
while (p  gap && a[p–gap] > temp)
a[p]  a[p–gap], p  p–gap;
a[p]  temp;
gap  gap  2; /* or … */

57. Exercise
Show the evolution of the following arrays under shell sort. Use length/3 as the starting gap:
[15, 3, 21, 45, 7, 17, 4]
[13, 11, 20, 15, 16, 6, 5, 8]

58. Complexity of Shell Sort
Depends on the gap sequence
O(N2) for N/2 version
O(N3/2) if we do the only-odd-numbers version
O(N3/2) for 2log N – 1 version
…, 63, 31, 15, 7, 3, 1
some others have this as well
some versions have O(N4/3)
one version has O(N1+sqrt(8ln(5/2)/ln(N)))
others we don’t even know!

59. Comments All those sort methods O(N2) in worst case
doubling size of array quadruples time
Shell sort can be O(N3/2), but still not good enuf
Want something better!
Recursion offers a way up
next time: recursion
after that: recursive sorting; other fast sorting