SORTING NETWORKS

Overview Sorting Network Components Sorting Network Comparison Networks Comparator Zero-one Principle Bitonic Sorting Network Half-Cleaner Bitonic Sorter Merging Network Sorting Network Previously we have seen several sorting algorithms that operate in a serial fashion by executing one operation at a time. What we will be presenting today is a look at a form of sorting algorithm based on the comparison-network model which allows for multiple operations to be performed simultaneously. A comparison network differs from a Random Access Machines implementation of an algorithm in two main areas. First, a comparison network can only perform comparisons. This means that an algorithm with a counting component such as counting sort cannot be implemented using a comparison network. Secondly, operations in a comparison network can be performed at the same time or in a parallel fashion, which differs significantly from the RAM model in which operations occur one after another or serially . Our presentation will cover the building blocks of a sorting network and some helpful principles and conclude with the development of a sorting network. We’ll begin our discussion of sorting networks by first taking a look at what is a comparison network and it’s components, in particular the comparator device. Then will discuss the zero-one principle which will assist in generalizing the operation of a comparison network to work correctly on arbitrary input. We’ll then begin the construction of the a parallel version of merge sort using a bitonic sorter. And we’ll make a modification of the bitonic sorter into a merging network. And conclude with the implementation of a sorting network, using these components, that can sort n values in O(lg2 n) time.

Comparison Networks Sorting networks are Comparison Networks that always sort their inputs Comparison Network composed of: wires comparators Wires: transmits values from place to place Comparators: two inputs  two outputs First up is a look at what is comparison network and it’s relation to sorting networks. Essentially, a sorting network is a comparison network that always sorts it inputs. A comparison network is comprised solely of wires and comparators. The wires transmit the input and subsequent values through the network to the output. Traditionally, we view the input traveling along horizontal wires in a left to right direction and we’ll maintain that format throughout our presentation. Between the input and output wires we have devices called comparators. A comparator operates on two inputs from the wires. The sole purpose of the comparator is to compare it’s inputs and deliver it’s output back to the wires and in our case it orders the outputs in a decending fashion.

Comparison Networks Comparators: inputs: x and y outputs: x’ and y’ x’ = min(x, y) y’ = max(x, y) input output x   x’ = min(x, y) y   y’ = max(x, y) x ------------------------------ x’ = min(x, y) y ------------------------------ y’ = max(x, y) comparator If we consider our inputs to the comparators to be the values contained in x and y, the output from the comparators is held in x’ and y’. The output values in x’ and y’ are calculated by the functions: x’ = min(x, y) and y’ = max(x, y) This results in the output being ordered in decending order in the network. So if we look at the diagram: We have our inputs x and y entering the comparator and the sorted values x’ and y’ exiting the comparator. To make matters a little easier to depict we will illustrate the comparators as vertical lines across the wires with nodes on the wires that the comparator is receiving input from and delivering output to.

Comparison Networks Comparators n input wires: a1, a2, …, an input sequence: < a1, a2, …, an> n output wires: b1, b2, …, bn output sequence: <b1, b2, …, bn> Properties: requirement: graph is acyclic output produced only when input is available comparators process in parallel if input is available Looking at the design of a comparison network we have our comparators and our input wires. Since we have the same number of inputs as outputs across the network we can represent our input wires as: a1, a2, …, an ( a1, a2 through an) And the output wires as: b1, b2, …, bn (b1, b2 through bn) For simplicity sake we represent the values input to the network as the sequence < a1, a2, …, an> And the output from the network as the sequence <b1, b2, …, bn> Some requirements for representing the comparison network graph is that: The graph of the wires and comparators interconnections must be acyclic, so we can never go through the same comparator twice. The output from a comparator can only be produced when then input is made available to it. And the comparators are able to process the input in parallel as the input is made available.

Comparison Networks comparators a1 9 ----------------------------------------- b1 a2 5 ----------------------------------------- b2 a3 2 ----------------------------------------- b3 a4 6 ----------------------------------------- b4 values A C D E B So taking a look at a simple example of a comparison network made up of 4 inputs(a1 – a4) and outputs(b1 - b4) with 5 comparators operating in the network we have the following diagram. Our input sequence is <9, 5, 2, 6> and we state that this sequence arrives at the input wires at time 0. We designate our five comparators as A, B, C, D and E.

Comparison Networks outputs a1 9 ----------------------------------------- 2 b1 a2 5 ----------------------------------------- 5 b2 a3 2 ----------------------------------------- 6 b3 a4 6 ----------------------------------------- 9 b4 depth: 1 1 2 2 3 depth starts at 0 input wires depth: dx and dy output wires depth: max(dx, dy) + 1 5 2 A C 9 6 D E 2 5 B 6 9 This comparison network happens to also be a sorting network and the input should be sorted into the order on the right 2, 5, 6 and 9 Walking through the networks operations: At time 0 the comparators A and B both have their inputs available. If we assume that each comparator takes 1 time unit to compute it’s output values. Then at time 1 both comparators A and B produce their output. Resulting in A computing the min and max of the outputs and producing 5 on the top and 9 on the bottom from the inputs 9 and 5 And B generating 2 and 6 respectively on the output wires unchanged from the input since they were already sorted. At time 1, when A and B have finished, both comparators C and D have their input available. C receives 5 and 2 and outputs 2 and 5 at time 2 since it took 1 time for the comparator to compute And D receives 9 and 6 and outputs 6 and 9 also at time 2 At this point both the top and bottom wires have completed and the values on the middle wires are available to comparator E. So at time 2, E receives 6 and 5 and outputs 5 and 6 at time 3. So under the assumption that each comparator takes unit time, we can define the running time of a comparison network, as the time it takes for all the output wires to receive their values once the input wires receive theirs. Informally, this would be the largest number of comparators that any input element can pass through as is travels across the network. Defining this more formally we use the depth of a wire. Where the depth of the initial input wire to the network has depth 0 and if a comparator as two input wires with depths dx and dy then the comparators output wires have depth max(dx, dy) + 1. So we define the depth of a comparator to be the depth of it’s output wires. Looking at the depths from the example. We can see that since both comparators A and B receive their inputs at time 0 and it takes 1 unit of time to process the output, then since the depth of the comparators is defined by their output wires, both A and B have depth 1. It follows that C and D would receive their inputs at time 1 resulting in their depths both being 2 when they deliver their output. And finally E would have a depth of 3. Therefore, in this example the comparison network would have a depth of 3.

Comparison Networks Sorting Network is a Comparison Network where the output is sorted output sequence is monotonically increasing (b1 <= b2 …<= bn) for every input sequence Not all Comparison Networks are Sorting Networks Comparison Network is like a procedure in that it specifies how comparisons are to occur So to review. A sorting network is a comparison network that sorts it’s output. That is the output sequence is monotonically increasing for every input sequence. Now the example that we saw of a comparison network just happened to also be a sorting network, since it would sort all of it’s inputs, however, not all comparison networks are sorting networks. So essentially a comparison network is like a procedure in that it specifies how comparisons are to occur from the arrangement of the comparators. But it is unlike a procedure in that its size, or the number of comparators it contains, depends on the number of inputs and outputs.

Zero-One Principle If a Sorting Network works correctly when each input is drawn from the set {0, 1}, then it works correctly on arbitrary input numbers (integers, reals, or values from any linearly sorted set)

Zero-One Principle Lemma 27.1 If a Comparison Network transforms the input sequence a = <a1, a2, …, an> into the output sequence b = <b1, b2, …, bn>, then for any monotonically increasing function f, the network transforms the input sequence: f(a) = <f(a1), f(a2), …, f(an)> into the output sequence: f(b) = <f(b1), f(b2), …, f(bn)>

Zero-One Principle Proof: prove claim: if f is a monotonically increasing function, then a single comparator with inputs f(x) and f(y) produces output f(min(x, y)) and f(max(x, y)) f(x) ------------------------------ min(f(x), f(y)) f(y) ------------------------------ max(f(x), f(y)) min(f(x), f(y)) = f(min(x, y)) max(f(x), f(y)) = f(max(x, y))

Zero-One Principle outputs f(x) = ceiling(x/2) a1 9 ----------------------------------------- 1 b1 a2 5 ----------------------------------------- 3 b2 a3 2 ----------------------------------------- 3 b3 a4 6 ----------------------------------------- 5 b4 3 1 A C 5 3 D E 1 3 B 3 5

Zero-One Principle Theorem 27.2 (Zero-One Principle) If a Comparison Network with n inputs sorts all 2n possible sequences of 0’s and 1’s correctly, then it sorts all sequences of arbitrary numbers correctly.

Zero-One Principle Proof: By contradiction: Suppose that the network sorts all zero-one sequences, but there exists a sequence of arbitrary numbers that the network does not sort: <a1, a2, …, an> contains elements ai and aj where ai < aj but the network places aj before ai in the output sequence

Zero-One Principle Define monotonically increasing function f: f(x) = { 0 if x <= ai, 1 if x > ai. aj is placed before ai in the output sequence f(aj) placed before f(ai) when input is <a1, a2, …, an> since f(aj) = 1 and f(ai) = 0, from above we have a contradiction

Bitonic Sorting Network Bitonic Sequence: a sequence that monotonically increases and then decreases, or can be circularly shifted to become monotonically increasing and then monotonically decreasing examples: <1, 4, 6, 8, 3, 2>, < 6, 9, 4, 2, 3, 5>, and <9, 8, 3, 2, 4, 6> zero-one: 0i1j0k or 1i0j1k when i,j,k >= 0

Bitonic Sorting Network Half-Cleaner a bitonic sorter is composed of several stages, each of which is called a half-cleaner half-cleaner is a Comparison Network of depth 1 line i compared with i + n/2 for i = 1, 2,…, n/2 assume that n is even

Bitonic Sorting Network Example: Half-Cleaner[8] all 1’s or 0’s 0 ---------------------------------- 0 0 ---------------------------------- 0 bitonic 1 ---------------------------------- 0 clean bitonic 1 ---------------------------------- 0 ________ 1 ---------------------------------- 1 0 ---------------------------------- 1

Bitonic Sorting Network Lemma 27.3 If the input to a half-cleaner is a bitonic sequence of 0’s and 1’s, then the output satisfies the following properties: both the top half and the bottom half are bitonic every element in the top half is at least as small as every element in the bottom half at least one half is clean (either all 0’s or 1’s)

Bitonic Sorting Network Proof: the Comparison Network Half-Cleaner[n] compares inputs i and i + n/2 for i = 1, 2, … n/2 suppose input is of the form: 00…011…100…0 there are 3 possible cases for the n/2 midpoint to fall and the situation where the midpoint falls in a block of 1’s is separated into 2 cases So 4 cases in total

Bitonic Sorting Network Cases: bitonic n/2 top top bitonic clean 1 1 bitonic 1 1 bottom bottom top top 1 bitonic clean 1 1 bitonic 1 bottom bottom

Bitonic Sorting Network Cases: bitonic n/2 top top 1 bitonic 1 1 1 1 1 bitonic clean 1 bottom bottom top top bitonic clean 1 1 1 1 bitonic 1 1 bottom bottom 1

Bitonic Sorting Network Bitonic Sorter by recursively combining half-cleaners, we can build a bitonic sorter, a network that sorts bitonic sequences first stage: Half-Cleaner[n] subsequent sorts with Bitonic-Sorter[n/2] Depth D(n) of Bitonic-Sorter[n]: D(n) = { 0 if n =1, D(n/2) + 1 if n = 2k and k>= 1

Bitonic Sorting Network Example: Half-Cleaner[n] Bitonic- Sorter[n/2] Bitonic- Sorter[n/2]

Bitonic Sorting Network Example: sorted bitonic 1 1 1 1 1 1 1 1 1 1 1 1

Bitonic Sorting Network Bitonic-Sorter can be used to sort a zero-one sequence By the Zero-One principle it follows that any bitonic sequence of arbitrary numbers can be sorted using this network

A Merging Network Networks that can merge two sorted input sequences into one sorted output sequence The merging network is based on the following Given two sorted sequences, if we reverse the second sequence and then concatenate the two, the resulting sequence is bitonic. ie. given : X = 00001111 and Y = 000011111 YR = 111110000 X● YR = 00001111111110000 Thus to merge the two input sequences X and Y it suffices to perform a bitonic sort on X concatenated with Y reverse

A Merging Network We can construct MERGER[n] by modifying the first half cleaner of BITONIC-SORTER[n]. The key is to perform the reversal of the second half of the inputs implicitly. Given two sorted sequences <a1, a2, …, an/2> and <an/2+1, an/2+2, …, an> Want the effect of bitonically sorting the sequence < a1, a2, …, an/2, an, an-1, an-2, ..., an/2+1> Given two sorted sequences <a1, a2, …, an/2> and <an/2+1, an/2+2, …, an> to be merged we want the effect of bitonically sorting the sequence < a1, a2, …, an/2, an, an-1, an-2, ..., an/2+1>

A Merging Network BITONIC-SORTER[n] compares inputs i and n/2+i for i= 1,2,3,…,n/2. a1 0 ---------------------------------- 0 b1 a2 0 ---------------------------------- 0 b2 bitonic a3 1 ---------------------------------- 0 b3 clean bitonic a 4 1 ---------------------------------- 0 b4 a8 1 ---------------------------------- 1 b8 a7 0 ---------------------------------- 0 b7 bitonic a6 0 ---------------------------------- 1 b6 a5 0 ---------------------------------- 1 b5 Review of the first half cleaner of the Bitonic-Sorter[n], with the reverse of the second sequence of values

A Merging Network First stage of the merging network compares inputs i and n/2 + i for i= 1,2,3,…,n/2. a1 0 ---------------------------------- 0 b1 sorted a2 0 ---------------------------------- 0 b2 bitonic a3 1 ---------------------------------- 0 b3 clean a4 1 ---------------------------------- 0 b4 a5 0 ---------------------------------- 1 b5 sorted a6 0 ---------------------------------- 1 b6 bitonic a7 0 ---------------------------------- 0 b7 a8 1 ---------------------------------- 1 b8 First stage of the merging network compare inputs I and n – I + 1. The only subtlety is that the order of the outputs from the bottom of the first stage of Merger[n] are reversed compared with the order of the outputs from an ordinary half-cleaner.

A Merging Network Since the reversal of a bitonic sequence is bitonic sequence. Both top and bottom outputs from the first stage of the merging network satisfy the properties of Lemma 27.3 both the top half and the bottom half are bitonic every element in the top half is at least as small as every element in the bottom half at least one half is clean (either all 0’s or 1’s)

A Merging Network The top and bottom outputs can be bitonically sorted in parallel to produce the sorted output of the merging network.

A Merging Network Bitonic- Sorter[n/2] Bitonic- Sorter[n/2] The first stage of the meger decomposed followed by two parrallel copies of bitonic sorter[n/2]

A Merging Network sorted 1 1 1 1 1 sorted 1 1 1 1 sorted 1 1 1 1 1 1 1 sorted 1 1 1 1 1 sorted 1 1 1 1 sorted 1 1 1 1 1 1 1 1 1 1 The same network as the previous slide, but recursively unrolled. Sample zero-one values are shown

A Merging Network First stage is different Consequently the depth is the same as BITONIC-SORTER[n] Depth D(n) of MERGER[n]: D(n) = { 0 if n =1, D(n/2) + 1 if n = 2k and k>= 1 D(n) = lg n The only thing different from the Bitonic Sorter[n] is the first stage. Consequently the depth of Merger[n] is lg n the same as the Bitonic Sorter

A Sorting Network Now we have all the necessary tools to construct a network that can sort any input sequence We are going to use the a fore mentioned merging network to implement a parallel version of merge sort, called SORTER[n]

A Sorting Network Sorter[n/2] Merger[n] Sorter[n/2] The sorting network Sorter is constructed by recursievely combining merging networks. This the recursive construction

A Sorting Network Merger[2] Merger[4] Merger[8] Merger[2] Merger[2] If we unroll the recursion the network looks like this for n = 8

A Sorting Network 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 3 4 4 4 4 5 5 6 depth 1 1 1 1 1 1 1 1 1 1 1 This is what the Merger boxes look like with the actual merging networks. Notice that all the mergers keep with the properties that are necessary for them to work. A sorted input, and gives out a sorted output. Show the bitonic clean areas that would occour. The first stage of the Mergers…… 1 2 2 3 4 4 4 4 5 5 6 depth

A Sorting Network

Conclusion Sorting Network Components Sorting Networks Comparison Networks Zero-one Principle Bitonic Sorting Network Half-Cleaner Bitonic Sorter Merging Network Sorting Networks