Download presentation

Presentation is loading. Please wait.

Published byLynne Rose Modified over 2 years ago

1
Analysis of Cilk

2
A Formal Model for Cilk A thread: maximal sequence of instructions in a procedure instance (at runtime!) not containing spawn, sync, return For a given computation, define a dag Threads are vertices Continuation edges within procedures Spawn edges Initial & final threads (in main)

3
Work & Critical Path Threads are sequential: work = running time Define T P = running time on P processors Then T 1 = work in the computation And T ∞ = critical-path length, longest path in the dag

4
Lower Bounds on T P T P ≥ T 1 / P (no miracles in the model, but they do happen occasionally) T P ≥ T ∞ (dependencies limit parallelism) Speedup is T 1 / T P (Asymptotic) linear speedup means Θ(P) Parallelism is T 1 / T ∞ (average work available at every step along critical path)

5
Greedy Schedulers Execute at most P threads in every step Choice of threads is arbitrary (greedy) At most T 1 / P complete steps (everybody busy) At most T ∞ incomplete steps All threads with in-degree = 0 are executed Therefore, critical path length reduced by 1 Theorem: T P ≤ T 1 / P + T ∞ Linear speedup when P = O(T 1 / T ∞ )

6
Cilk Guarantees: T P ≤ T 1 / P + O(T ∞ ) expected running time Randomized greedy scheduler The developers claim: T P ≈ T 1 / P + T ∞ in practice This implies near-perfect speedup when P << T 1 / T ∞

7
But Which Greedy Schedule to Choose? Busy leaves property: some processor executes every leaf in the dag Busy leaves controls space consumption; can show S P = O(P S 1 ) Without busy leaves, worst case is S P = Θ(T 1 ), not hard to reach A processor that spawns a procedure executes it immediately; but another processor may steal the caller & execute it

8
Work Stealing Idle processors search for work on other processors at random When a busy victim is found, the theif steals the top activation frame on victim’s stack (In practice, stack is maintained as a dequeue) Why work stealing? Almost no overhead when everybody busy Most of the overhead incurred by theives Work-first principle: little impact on T 1 / P term

9
Some Overhead Remains To achieve portability, spawn stack is maintained as an explicit data structure Theives steal from this dequeue Could implement stealing directly from stack, more complex and nonportable Main consequence: Spawns are more expensive than function calls Must do significant work at the bottom of recursions to hide this overhead Same is true for normal function calls, but to a lesser extent

10
More Cilk Features

11
Inlets x = spawn fib(n-1) is equivalent to: cilk int fib(int n) { int x = 0; inlet void summer(int result) { x += result; } if (n<2) return n; summer( spawn fib(n-1) ); summer( spawn fib(n-2) ); sync; return x;

12
Inlet Semantics Inlet: an inner function that is called when a child completes its execution An inlet is a thread in a procedure (so spawn, sync are not allowed) All the threads of a procedure instance are executed atomically with respect to one another (i.e., not concurrently) Easy to reason about correctness x += spawn fib(n-1) is an implicit inlet

13
Aborting Work An abort statement in an inlet aborts already-spawned children of a procedure Useful for aborting speculative searches Semantics: Children may not abort instantly Aborted children do not return values, so don’t use these values, as in x = spawn fib(n-1) Does not prevent future spawns; be careful with sequences of spawns

14
The SYNCHED Built-In Variable True only if no children are currently executing False if some children may be executing now Useful for avoiding space and work overheads that reduce the critical path when there is no need to

15
Cilk’s Memory Model Memory operations of two threads are guaranteed to be ordered only if there is a dependence path between them (ancestor- descendant relationship) Unordered threads may see inconsistent views of memory

16
Locks Mutual-exclusion variables Memory operations that a thread performs before releasing a lock are seen by other threads after they acquire the lock Using locks invalidates all the performance guarantees that Cilk provides In short, Cilk supports locks but don’t use them unless you must

17
Useful but Obsolete Cilk as a library Can call Cilk procedures from C, C++, Fortran Necessary for building general-purpose C libraries Cilk on clusters with distributed memory Programmer sees the same shared-memory model Used an interesting memory-consistency protocol to support shared-memory view Was performance ever good enough?

18
Some Open Problems Perhaps good enough for a thesis

19
Open Issues in Cilk Theoretical question about the distributed- memory version: is performance monotone in the size of local caches? Cilk as a library: resurrect Distributed-memory version: resurrect, is it fast enough? Can you make it faster?

20
Parallel Merge Sort in Cilk

21
Parallel Merge Sort merge_sort(A, n) if (n=1) return spawn merge_sort(A, n/2) spawn merge_sort(A+n/2, n-n/2) sync merge(A, n/2, n-n/2)

22
Can’t Merge In Place! merge_sort(A, T, n, AorT) if (n=1) { T 0 =A 0 ; return } spawn merge_sort(A, T, n/2, !TorA) spawn merge_sort(A+n/2, n-n/2, !TorA) sync if (TorA=A) merge(A, T, n/2, n-n/2) if (TorA=T) merge(T, A, n/2, n-n/2)

23
Analysis Merging uses two pointers, move the smaller into sorted array T 1 (n) = 2T 1 (n/2) + Θ(n) T 1 (n) = Θ(n log n) We fill the output element by element T ∞ (n) = T ∞ (n/2) + Θ(n) T ∞ (n) = Θ(n) Not very parallel...

24
Parallel Merging p_merge(A,n,B,m,C) // C is the output swap A,B if A is smaller if (m+n = 1) { C 0 =A 0 ; return } if (n = 1 /* implies m = 1 */) { merge ; return } locate A n/2 between B j and B j+1 (binary search) spawn p_merge(A,n/2,B,j,C) spawn p_merge(A+n/2, n-n/2, B+ j, n-j, C+n/2+j) sync

25
Analysis of Parallel Merging When we merge n elements, both recursive calls merge at most 3n/4 elements T ∞ (n) ≤ T ∞ (3n/4) + Θ(log n) T ∞ (n) = Θ(log 2 n) Critical path is short! But the analysis of work is more complex (extra work due to binary searches) T 1 (n) = T 1 (αn) + T 1 ((1-α)n) + Θ(log n), ¼ ≤ α ≤ ¾ T 1 (n) = Θ(n) using substitution (nontrivial) Critical path for parallel merge sort T ∞ (n) = T ∞ (n/2) + Θ(log 2 n) = Θ(log 3 n)

26
Analysis of Parallel Merge Sort T ∞ (n) = T ∞ (n/2) + Θ(log 2 n) = Θ(log 3 n) T 1 (n) = Θ(n) Impact of extra work in practice? Can find the median of 2 sorted arrays of total size n in Θ(log n) time, leads to parallel merging and merge-sorting with shorter critical paths

27
Analysis of Parallel Merge Sort T ∞ (n) = T ∞ (n/2) + Θ(log 2 n) = Θ(log 3 n) T 1 (n) = Θ(n) Impact of extra work in practice? Can find the median of 2 sorted arrays of total size n in Θ(log n) time, leads to parallel merging and merge-sorting with shorter critical paths Parallelizing an algorithm can be nontrivial!

28
Cache-Efficient Sorting

29
Caches Store recently-used data Not really LRU Usually 1, 2, or 4-way set associative But up to 128-way set associative Data transferred in blocks called cache lines Write through or write back Temporal locality: use same data again soon Spatial locality: use nearby data soon

30
Cache Misses in Merge Sort Assume cache-line size = 1, LRU, write back Assume cache holds M words When n <= M/2, exactly n reads, write backs When n > M, at least n cache misses (cover all cases in the proof!) Therefore, number of cache misses is Θ(n log n/M) = Θ(n log n – log M) We can do much better

31
The Key Idea Merge M/2 sorted runs into one, not 2 into 1 Keep one element from each run in a heap, together with a run label Extract the min, move to sorted run, insert another element from same run into heap Reading from sorted runs & writing to sorted ouput removes elements of the heap, but this cost is O(n) cache misses Θ(n log M n) = Θ(n log n / log M)

32
This is Poly-Merge Sort Optimal in terms of cache misses Can adapt to long cache lines, sorting on disks, etc Originally invented from sorting on tapes on a machine with several tape drives Often, Θ(n log n / log M) is really Θ(n) in practice Example: 32 KB cache, 4+4 bytes elements 4192-way merges Can sort 64 MB of data in 1 merge, 256 GB in 2 merges But more merges with long cache lines

33
From Quick Sort To Sample Sort Same number of cache misses with normal quick sort Key idea Choose a large random sample, Θ(M) elements Sort the samples Classify all the elements using binary searches Determine size of intervals Partition Recursively sort the intervals Cache miss # probably similar to merge sort

34
Distributed-Memory Sorting

35
Issues in Sample Sort Main idea: Partition input into P intervals, classify elements, send elements in ith interval to processor i, sort locally Most of the communication in one global all- to-all phase Load balancing: Intervals must be similar in size How do we sort the sample?

36
Balancing the Load Select a random sample of sP elements OK even if every processor selects s The probability of an interval larger than cn/P grows linearly with n, shrinks exponentially with s; a large s virtually ensures uniform intervals Example: n = 10 9, s = 256 Pr[max interval > 2n/P] < 10 -8

37
Sorting the Sample Can’t do it recursively! Sending the sample to one processor Θ(sP+n/P) communication in that processor Θ(sP log sP+n/P log( n/P )) work Not scalable, OK for small P Using a different algorithm that works well for small n/P, e.g., radix sort

38
Distributed-Memory Radix Sort Sort blocks of r bits from least significant to most significant; use a stable sort Counting sort of one block Sequentially, count occurrences using a 2 r array, compute prefix sums, use as pointers In parallel, every processor counts occurrences, then pipelined parallel prefix sums, send elements to destination processor Θ((b/r) (2 r + n/P) work & comm / proc

39
Odds and Ends

40
CPU Utilization Issues Avoid conditionals In sorting algorithms, the compiler & processors cannot predict outcome, so the pipeline stalls Example: partitioning in qsort without conditionals Avoid stalling the pipeline Even without conditionals, rapid use of computed values stalls the pipeline Same example

41
Dirty Tricks Exploit uniform input distributions approximately (quick sort, radix sort) Fix mistakes by bubbling To avoid conditionals when fixing mistakes Split the array into small blocks Use and’s to check for mistakes in a block Fix a block only if it contains mistakes Some conditionals, but not many Compute probability of mistakes to optimize

42
The Exercise Get sort.cilk, more instuctions inside Convert sequential merge sort into parallel merge sort Make it as fast as possible as long is it is a parallel merge sort (e.g., make the bottom of the recursion fast) Convert the fast sort into the parallel fastest sort you can Submit files in home directory, one page with output on 1, 2 procs + possible explanation (on one side of the page)

Similar presentations

OK

Adaptive Parallel Sorting Algorithms in STAPL Olga Tkachyshyn, Gabriel Tanase, Nancy M. Amato

Adaptive Parallel Sorting Algorithms in STAPL Olga Tkachyshyn, Gabriel Tanase, Nancy M. Amato

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google