1. Computer Architecture II 1 Computer architecture II Introduction

2. Computer Architecture II 2 Recap Parallelization strategies –What to partition? –Embarrassingly Parallel Computations –Divide-and-Conquer –Pipelined Computations Application examples Parallelization steps 3 programming models –Data parallel –Shared memory –Message passing

3. Computer Architecture II 3 4 Steps in Creating a Parallel Program Decomposition of computation in tasks Assignment of tasks to processes Orchestration of data access, comm, synch. Mapping processes to processors

4. Computer Architecture II 4 Plan for today Programming for performance Amdahl’s law Partitioning for performance –Addressing decomposition and assignment Orchestration for performance

5. Computer Architecture II 5 Creating a Parallel Program Assumption: Sequential algorithm is given –Sometimes need very different algorithm, but beyond scope Pieces of the job: –Identify work that can be done in parallel –Partition work and perhaps data among processes –Manage data access, communication and synchronization –Note: work includes computation, data access and I/O Main goal: Speedup (plus low prog. effort and resource needs) Speedup (p) = For a fixed problem: Speedup (p) = Performance(p) Performance(1) Time(1) Time(p)

6. Computer Architecture II 6 Amdahl´s law Suppose a fraction f of your application is not parallelizable 1-f : parallelizable on p processors Speedup(P) = T 1 /T p <= T 1 /(f T 1 + (1-f) T 1 /p) = 1/(f + (1-f)/p) <= 1/f

7. Computer Architecture II 7 Amdahl’s Law (for 1024 processors) See: Gustafson, Montry, Benner, “Development of Parallel Methods for a 1024 Processor Hypercube”, SIAM J. Sci. Stat. Comp. 9, No. 4, 1988, pp.609.

8. Computer Architecture II 8 Amdahl´s law But: –There are many problems can be “embarrassingly” parallelized Ex: image processing, differential equation solver –In some cases the serial fraction does not increase with the problem size –Additional speedup can be achieved from additional resources (super-linear speedup due to more memory) “

9. Computer Architecture II 9 Speedup speedup p linear speedup sub-linear speedup superlinear speedup!!

10. Computer Architecture II 10 Superlinear Speedup? Possible causes Algorithm –e.g., with optimization problems, throwing many processors at it increases the chances that one will “get lucky” and find the optimum fast Hardware –e.g., with many processors, it is possible that the entire application data resides in cache (vs. RAM) or in RAM (vs. Disk)

11. Computer Architecture II 11 Parallel Efficiency Eff p = S p / p Typically 1, unless superlinear speedup Used to measure how well the processors are utilized –If increasing the number of process by a factor 10 increases the speedup by a factor 2, perhaps it’s not worth it: efficiency drops by a factor 5

12. Computer Architecture II 12 Performance Goal => Speedup Architect Goal –observe how program uses machine and improve the design to enhance performance Programmer Goal –observe how the program uses the machine and improve the implementation to enhance performance

13. Computer Architecture II 13 4 Steps in Creating a Parallel Program Decomposition of computation in tasks Assignment of tasks to processes Orchestration of data access, comm, synch. Mapping processes to processors

14. Computer Architecture II 14 Partitioning for Performance First two phases of parallelization process: decomposition & assignment Goal 1.Balancing the workload and reducing wait time at synch points 2.Reducing inherent communication 3.Reducing extra work for determining and managing a good assignment (static versus dynamic) Tensions between the 3 goals –Maximize load balance => smaller tasks => increase communication –No communication (run on 1 processor) => extreme load imbalance (all others idle) –Load balance => extra work to compute or manage the partitioning (ex. dynamic techniques)

15. Computer Architecture II 15 1. Load Balance –Work: data access, computation –Not just equal work, but must be busy at same time –Ex: Speedup ≤ 1000/400 = 2.5 Sequential Work Max Work on any Processor Speedup ≤

16. Computer Architecture II 16 1. Load balance a)Identify enough concurrency Data and functional parallelism (last class) b)Managing concurrency c)Task granularity d)Reduce communication and synchronization

17. Computer Architecture II 17 1 b) Static versus Dynamic assignment Static: before the program starts is clear who does what #pragma omp parallel for schedule(static) for(i=0;I<N;i++) { a[i] = a[i] + b[i];} Dynamic –External scheduler –Self-scheduled Each process picks a chunk of loop iterations and executes them #pragma omp parallel for schedule(dynamic,4) for(i=0;I<N;i++) { a[i] = a[i] + b[i];} Dynamic guided self-scheduling: processes take first larger chunks and then reduce this number progressively #pragma omp parallel for schedule(guided,4) for(i=0;I<N;i++) { a[i] = a[i] + b[i];}

18. Computer Architecture II 18 Dynamic Tasking with Task Queues Centralized queue: simple protocol –Problems: Communication, synchronization, contention Distributed queues: complicated protocol –Initial distribution of jobs May cause load imbalance Solution: task stealing: whom to steal from, how many tasks to steal,... –Termination detection

19. Computer Architecture II 19 1.c Task granularity Task granularity: amount of work associated with a task General rule: –Coarse-grained: often less load balance –Fine-grained: better load balance, but more overhead, often more communication and contention Processor 1 Processor 2 Processor 1 Processor 2

20. Computer Architecture II 20 1.d Reducing Serialization Synchronization for task assignment may cause serialization (for instance the access to a queue) Sequential Work Max (Work + Synch Wait Time) Speedup < Process 1 Process 2 Process 3 Synchronization pointWorkSynchronization wait time

21. Computer Architecture II 21 Reducing Serialization Event synchronization –Reduce use of conservative synchronization point-to-point instead of barriers finer granularity of access may reduce the synchronization time –But fine-grained synchronization more difficult to program, more synchronization operations. Mutual exclusion –Separate locks for separate data lock per task in task queue, not per queue finer grain => less contention/serialization, more space, less reuse –Smaller, less frequent critical sections don’t do reading/testing in critical section, only modification e.g. searching for task to dequeue outside critical section

22. Computer Architecture II 22 2. Reducing Inherent Communication Communication is expensive! Measure: communication to computation ratio Inherent communication –Determined by assignment of tasks to processes –Actual communication may be larger (artifactual) One principle: Assign tasks that access same data to same process Sequential Work Max (Work + Synch Wait Time + Comm Cost) Speedup < Synchronization pointWorkSynchronization wait time Communication Process 1 Process 2 Process 3

23. Computer Architecture II 23 Domain Decomposition Ocean Example: communicate with the neighbors, compute in the assigned domain Perimeter to Area communication-to-computation ratio (area to volume in 3-d): Depends on n,p: decreases with n, increases with p

24. Computer Architecture II 24 Domain Decomposition Communication/computation: for block, for strip –Block better –Application dependent: strip may be better in other cases 4*√p n 2*p n Best domain decomposition depends on information requirements Block versus strip decomposition:

25. Computer Architecture II 25 Finding a Domain Decomposition GOALS: load balance & low communication Static, by inspection –Must be predictable: Ocean Static, but not by inspection –Input-dependent, require analyzing input structure –E.g sparse matrix computations Semi-static (periodic repartitioning) –Characteristics change but slowly; e.g. Barnes-Hut Static or semi-static, with dynamic task stealing –Initial decomposition, but highly unpredictable; e.g ray tracing

26. Computer Architecture II 26 3. Reducing Extra Work Common sources of extra work: –Computing a good partition e.g. partitioning in Barnes-Hut –Using redundant computation to avoid communication –Task, data and process management overhead applications, languages, runtime systems, OS –Imposing structure on communication coalescing small messages Sequential Work Max (Work + Synch Wait Time + Comm Cost + Extra Work) Speedup <

27. Computer Architecture II 27 PART II: memory aware optimizations So far we have seen the parallel computer as a collection of communicating processors –Goals: balance load, reduce inherent communication and extra work –We have assumed an unlimited memory In reality the parallel computer uses a multi-cache, multi- memory system

28. Computer Architecture II 28 Memory-oriented View Multiprocessor as Extended Memory Hierarchy Levels in extended hierarchy: –Registers, caches, local memory, remote memory –Glued together by communication architecture –Levels communicate at a certain granularity of data transfer Proc Cache L2 Cache L3 Cache Memory Proc Cache L2 Cache L3 Cache Memory potential interconnects Granularity increases, access time increases Capacity increases, cost/unit decreases

29. Computer Architecture II 29 Memory-oriented view Performance depends heavily on memory hierarchy Time spent by a program (usually given in cycles) Time prog =Time compute + Time access Data access time can be reduced by: – Optimizing machine larger caches lower latency Larger bandwidth – Optimizing program temporal and spatial locality

30. Computer Architecture II 30 Artifactual Communication in Extended Hierarchy poor allocation of data across distributed memories –Data accessed by a node is allocated in the memory of another even though accessed only by the remote node unnecessary data in a transfer (e.g. sender sends more) unnecessary transfers due to system granularities (e.g. cache coherent at block granularity) redundant communication of data (e.g. update-protocols : data sent at every modification, but needed only once) finite replication capacity (in cache or main memory): replicated data has to be evicted from memory y brought again later

31. Computer Architecture II 31 Replication induced artifactual communication Communication induced by finite capacity is most fundamental artifact –Like cache size and miss rate memory traffic in uniprocessors View as three level hierarchy for simplicity –Local cache, local memory, remote memory (ignore network topology) Classify “misses” in “cache” at any level as for uniprocessors (4 “C”s) compulsory or cold misses (no size effect) capacity misses (yes): has to evict smth because limited space conflict or collision misses (yes): memory that maps on the same cache blocks communication or coherence misses (no)

32. Computer Architecture II 32 Working set –Working set: size of data that fits in a certain level of memory 1. A few points for near-neighbor reuse 2. Three sub-rows 3. Whole matrix 15.while (!done) do 16.diff = 0; 17.for i  1 to n do 18.for j  1 to n do 19.temp = A[i,j]; 20.A[i,j]  0.2 * (A[i,j] + A[i,j-1] + A[i-1,j] + 21.A[i,j+1] + A[i+1,j]); 22.diff +=abs(A[i,j] - temp); 23.endfor 24.endfor 25.if (diff/(n*n) < TOL) then done = 1; 26.end while

33. Computer Architecture II 33 Working Set Perspective –Hierarchy of working sets –At first level cache (fully assoc, one-word block), inherent to algorithm working set curve for program –Traffic from any type of miss can be local or nonlocal (communication)

34. Computer Architecture II 34 Orchestration for Performance Reducing amount of communication: –Inherent: change the partitioning (seen earlier) –Artifactual: exploit spatial, temporal locality in the memory hierarchy Techniques often similar to those on uniprocessors Structuring communication to reduce cost

35. Computer Architecture II 35 Reducing Artifactual Communication Message passing model –Communication and replication are both explicit –Even artifactual communication is in explicit messages Shared address space model –Occurs transparently due to interactions of program and system –used for explanation

36. Computer Architecture II 36 Exploiting Temporal Locality –Def: reusing of data elements already brought into cache –Structure algorithm so working sets fit into the cache often techniques to reduce inherent communication: –assign tasks accessing the same elements to the same processor schedule tasks for data reuse once assigned –Ocean Solver example: blocking Each grid element accessed 5 times First time brought into cache then reused Rewrite the loops

37. Computer Architecture II 37 Exploiting Spatial Locality Def: when a data element is accessed, its neighbors are accessed Major spatial-related causes of artifactual communication : –Conflict misses –Data distribution/layout (allocation granularity) –Fragmentation (communication granularity) –False sharing of data (coherence granularity) AVOIDING ARTIFACTUAL COMMUNICATION: keep contiguous data accessed by one processor –Fix problems by modifying data structures, or layout/alignment

38. Computer Architecture II 38 Spatial Locality Example – Repeated sweeps over 2-d grid, each time adding 1 to elements – 4-d grid to achieve spatial locality ( line processor x column processor x line index x column index)

39. Computer Architecture II 39 Tradeoffs with Inherent Communication Partitioning grid solver: blocks versus rows –Blocks have a spatial locality problem on remote data: when accessing the elements of neighboring processors whole cache blocks are fetched at column boundary –Row-wise can perform better despite worse inherent communication-to-computation ratio

40. Computer Architecture II 40 Example Performance Impact on Origin2000 OceanKernel solver