1. Parallel Sorting Algorithms

2. The simple sorting algorithms (Bubble Sort, Insertion Sort, Selection Sort, …) are
Lower bound on comparison based algorithms (Merge Sort, Quicksort, Heap Sort, …) is
The best we can hope to do with parallelizing a comparison-based algorithm using n processors is

3. Question: What can we expect if we use n2 processors?
Answer: Typically, it is still:

4. In place sorting – sorting the values within the same array
Not in-place (or out-of-place) sorting – sorting the values to a new array
Stable sorting algorithm – Identical values remain in the same order as the original after sorting

5. Compare and Exchange
Most sorting algorithms (although not all) compare two values and possibly exchange them
Two numbers in the array are compared with each other:
if (A[i] > A[j]) {
temp = A[i];
A[i] = A[j];
A[j] = temp; }

6. Message-passing Compare and Exchange (version 1)
In a distributed-memory system, we have to send message to exchange values
P1 send value to P2. P2 compares the two values; keeps the larger and sends the smaller back to P1
A[i]
A[i]
Processor 1
A[j]
Processor 2
A[i] is sent
Smaller Value Returned
Larger Value Kept

7. Message-passing Compare and Exchange (version 2)
Both processors send their values to each other. They both compare values. P1 keeps the smaller value, P2 keeps the larger values.
A[i]
A[i]
Processor 1
A[j]
Processor 2
A[i] is sent
A[j]
A[j] is sent
Smaller Value Kept
Larger Value Kept

8. Message-passing Compare and Exchange (version 1)
When we have the number of values (n) is much larger than the number of processors (p), then each processors is responsible of n/p values
Processor 1
Processor 2
Final Values
Original Values (sorted first) 88 50 28 25
Original Values (sorted first) 98 80 43 42
merge 98 88 80 50 43 42 28 25
Larger Numbers Kept 43 42 28 25 88 50 28 25
Final Values
Smaller Numbers Sent

9. Message-passing Compare and Exchange (version 2)
When we have the number of values (n) is much larger than the number of processors (p), then each processors is responsible of n/p values
Processor 2
Processor 2 98 88 80 50 43 42 28 25 88 50 28 25 98 80 43 42
merge 98 88 80 50 43 42 28 25
merge
Larger Numbers Kept
Original Values sent 98 80 43 42 88 50 28 25
Smaller Numbers Kept

10. Parallelizing Common Sorting Algorithms

11. Bubble Sort N passes are made comparing/exchanging adjacent values
Larger values settle to the bottom (Sediment Sort?)
After pass m, the m largest values are in the m last locations in the array.

12. After pass 1, the largest values is in it’s location

13. Complexity of Bubble Sort
The number of comparison/exchanges with Bubble Sort is:
Very slow but easy to parallelize
We can use a pipeline-sort of technique

14. Parallel Bubble Sort P0 P1 P2 P3 P4 P5 P6 P7 Phase 1 Time Phase 2

15. Even-Odd Transposition Sort
Uses alternating even and odd phases
Even phase
even processors exchange with next higher ranked processor
odd processors exchange with the next lower ranked processor
Odd phase
even processors exchange with next lower ranked processor
odd processors exchange with the next higher ranked processor

16. Even-Odd Sort P0 P1 P2 P3 P4 P5 P6 P7 4 2 7 8 5 1 3 6 2 4 7 8 1 5 3 6
Time 2 4 1 7 3 8 5 6 2 1 4 3 7 5 8 6 1 2 3 4 5 7 6 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8

17. What is the time complexity of Parallel Bubble Sort with n processors?
Answer: 2n=O(n)
What is the time complexity of Even-Odd Sort with n processors?
Answer: O(n)

18. Divide and Conquer Pattern
Characterized by dividing problem into sub- problems of same form as larger problem.
Further divisions into still smaller sub-problems, usually done by recursion. Creates a tree.
Recursive divide and conquer amenable to parallelization because separate processes can be used for divided parts. Also usually data becomes naturally localized.

19. Sorting - Several sorting algorithms can often be partitioned or constructed in a recursive divide and conquer fashion, e.g. Mergesort, Quicksort,
Searching algorithms dividing search space recursively

20. Merge Sort Merge Sort is a “Divide and Conquer” algorithm
The array is divided into smaller and smaller parts
Then the small parts are merged into a new sorted sub array

21. Merge Sort Divide Merge P0 P1 P2 P3 P4 P5 P6 P7 4 2 7 8 5 1 3 6 4 2 7
Time 4 2 7 8 5 1 3 6
Merge 2 4 7 8 1 5 3 6 2 4 7 8 1 3 5 6 1 2 3 4 5 6 7 8

22. Complexity of Parallel Merge Sort
Complexity of the sequential merge sort:
Complexity of parallel merge sort with n processors
There are 2logn steps, but each merge step takes longer and longer
Turns out to be:

23. Quicksort
Pick a pivot
The move all the values less than the pivot to the left side of the array
And all the values greater than the pivot to the right side
Then recursively sort the two sides

25. Quicksort Works well when the pivot is approximately in the middle
If the pivot is chosen poorly, quicksort resorts to

26. Neither Merge Sort nor Quicksort parallelize efficiently
Still
Any tree structure (reduction, merge, quicksort) cannot be parallelized by n processors efficiently

27. Batcher’s Parallel Sorting Algorithms
Odd-even Merge Sort (not the same as even- odd transposition sort)
Bitonic Merge Sort
Complexity with n processors:

28. Odd-Even Merge Sort

29. Summary (using n processors to sort n numbers)
Parallel Bubble sort and Odd-even transposition sort - O(n)
Parallel mergesort - O(n) but unbalanced processor load and communication (because of tree)
Parallel quicksort - O(n) but unbalanced processor load and communication (because of tree) and can degenerate to O(n2)
Odd-even Mergesort - O(log2n)

30. Sorting on Specific Networks
Two network structures have received specific attention:
Mesh
Hypercube
because parallel computers have been built with these networks.
Of less interest nowadays because underlying architecture often hidden from user.
There are MPI features for mapping algorithms onto meshes.
Can always use a mesh or hypercube algorithm even if the underlying architecture is not the same.

31. Mesh - Two-Dimensional Sorting
Final sorted list -- could be row by row or snakelike.

32. Shearsort Alternate row and column sorted until list fully sorted
Result is a snake-like sorting
On a mesh, this requires
steps

33. Shearsort

34. Other Sorts (Coming Up Next)
Rank Sort
Counting Sort
Radix Sort

35. Rank Sort
The idea of rank sort is to count the number of values less than a[i]
That count is the “rank” of that value
The rank is where in the sorted array that value will be placed
So we can put in into that spot b[rank] = a[i].

36. Rank Sort (Sequential Code)
for (i = 0; i < n; i++) { // for each number x = 0; for (j = 0; j < n; j++) // count number less than it if (a[i] > a[j]) x++; b[x] = a[i]; // copy number into correct place }
Doesn’t handle duplicate values (How can this be fixed?)
The complexity is O(n2)
However, this is easy to parallelize

37. Rank Sort (Parallel Code)
forall (i = 0; i < n; i++) { // each processor allocated different value x = 0; for (j = 0; j < n; j++) // count number less than it if (a[i] > a[j]) x++; b[x] = a[i]; // copy number into correct place }
Uses n processors and complexity is O(n)
Very easy to implement in OpenMP
As good as many of the parallel sorts (except bitonic and shear sort)

38. Rank Sort using n2 processors
Instead of one processors comparing a[i] with other values, we use n-1 proces- sors
Incrementing counter is a critical section; must be sequential, so this is still O(n)

39. Rank Sort using n2 processors
Can do this as a reduction
Complexity is O(logn) with n2 processors
Low processor efficiency

40. Sequential Complexity Parallel Complexity with n processors
Summary
Sorting Algorithm
Sequential Complexity
Parallel Complexity with n processors
Bubble Sort
O(n2)
O(n)
Even-Odd Sort
Merge Sort
O(nlogn)
Quicksort
Odd-Even Merge Sort
O(log2n)
Shearsort
O(√n(logn+1))
Rank Sort

41. Counting Sort The best sequential sorting algorithms are O(nlogn).
We can do better if we can make assumption about the data
If the values are all integers within a relatively small range of values, we can do Counting Sort in O(n) instead of O(n2) or O(nlogn).

42. Counting Sort
Counting Sort is a stable sort (identical values remain in same relative position as in orginal array)
However, it is not an “in-place” sort
Further more, we need extra space
Suppose original values in the array a[ ]
Sorted array will be b[ ]
Create an array c[ ] of size m where values of a[ ] are in the range 1..m.

43. Counting Sort – Step 1
c[ ] is the histogram of values in the array. In other words, it is the sum of equal values
Complexity is O(m + n)
for (i = 1; i <= m; i++)
c[i] = 0;
for (i = 1; i <= n; i++)
c[ a[i] ]++;

44. Counting Sort – Step 2
We can find the rank of each value by doing a prefix sum on c[ ]
for (i = 2; i <= m; i++)
c[i] = c[i] + c[i-1];
Complexity is O(m)
c[0]
c[1]
c[2]
c[3]
c[4] … c0
c0+ c1
c1+ c2
c2+ c3
c3+ c4

45. Counting Sort – Step 3
We can now use the prefix sum to place the values where they should go:
for (i = n; i >= 1; i--) {
b[ c[ a[i] ] ] = a[i]
c[ a[i] ]--; // ensures stable sorting }
Sequential complexity for all 3 steps is O(m + n)
If m is linear with n, then complexity is O(n)

46. Counting Sort Step 3 moves backwards thro a[]1 2 3 4 5 6 7 8 5 2 3 7 6 4 1
Original sequence
a[]
Step 3 moves backwards thro a[]
Step 1. Histogram
c[] 1 2
Step 2. Prefix sum
c[] 1 2 3 4 6 7 8
Move 5 to position 6. Then decrement c[5]
Step 3. Sort
b[] 1 2 3 4 5 6 7
Final sorted sequence

47. Parallel Counting Sort (using n processors)
Step 1 – Creating the Histogram: O(logn). We use the same tree as with rank sort. Updating c[a[i]]++ is a critical section when there are duplicate values in a[ ].
Step 2 – Parallel version of prefix sum using n processors is O(logn)
Step 3 – Placing number in place: O(logn) again because of possible duplicate values (c[a[i]]– is a critical section)
All 3 steps: O(logn)

48. Radix Sort
Assumption: Values are either unsigned decimal or unsigned binary
Sort values based on least significant digit
Then sort on next least significant digit
Continue until the most significant digit is used to sort.
This works if we use a stable sorting algorithm
Since each digit will be in the range 0-9 or 0-1, then we can use Counting Sort for each digit

49. Radix Sort on Decimal Values

50. Radix Sort on Binary Values

51. Radix Sort
When sorting binary values, we can improve the counting sort
To sort a column of bits, the prefix sum gives us the number of 1’s
We can use that to place the ones (decremented each time to deal with the duplicates)
The prefix sum of the inverse of the bits (called the diminished prefix sum) gives us the number of 0’s and can be used to place the zeros.
The complexity of sorting this column is O(logn), same as using counting sort on the column

52. Radix Sort
The complexity of Radix Sort is the complexity of sorting each digit times the number of digits
If the range of values is 1..N, then the number of digits is log2N for binary numbers and log10N for decimal.
O(logn*logN) where n is the number of values and N is the range of values

53. Sequential Complexity Parallel Complexity with n processors
Summary
Sorting Algorithm
Sequential Complexity
Parallel Complexity with n processors
Bubble Sort
O(n2)
O(n)
Even-Odd Sort
Merge Sort
O(nlogn)
Quicksort
Odd-Even Merge Sort
O(log2n)
Shearsort
O(√n(logn+1))
Rank Sort
Counting Sort
O(logn)
Radix Sort
O(nlogN)
O(logn*logN)

54. Questions