1. STAPL The C++ Standard Template Adaptive Parallel Library Alin Jula Department of Computer Science, Texas A&M Ping An, Silvius Rus, Steven Saunders, Tim Smith, Gabriel Tanase, Nathan Thomas, Nancy M. Amato and Lawrence Rauchwerger http://www.cs.tamu.edu/research/parasol

2. 2Texas A&M UniversitySTAPL Motivation Building block library – Nested parallelism Inter-operability with existing code – Superset of STL Portability and Performance – Layered architecture – Run-time adaptivity STAPL – C++ Standard Template Adaptive Parallel Library

3. 3Texas A&M UniversitySTAPL Philosophy Interface Layer – STL compatible Concurrency & Communication Layer – Generic parallelism, synchronization Software Implementation Layer – Instantiates concurrency & communication Machine Layer – Architecture dependent code

4. 4Texas A&M UniversitySTAPL Related Work STAPLAVTLCHARM++CHAOS++CILK * NESL*POOMAPSTLSPLIT C * Paradigm SPMD/ MIMD SPMDMIMDSPMDSPMD/ MIMD SPMD Architecture shared/ dist distshared/ dist shared/ dist shared/ dist shared/ dist shared/ dist Nested Parallelism yesno yes no yes Adaptive yesno Generic yes Yes (limits)yesnoyes no Irregular yesnoyes noyes Data Decomp. auto/ user autouserauto/ user auto/ user Data Mapping auto/ user auto auto/ user auto userauto/ user auto Scheduling user, static, dyn, block user, MPI- based prioritized execution data decom. based work stealing work & depth model pthread scheduling Tulip RTS user Comm/Comp Overlap yesnoyesno yes * Parallel programming language

5. 5Texas A&M UniversitySTAPL STL Overview Containers Data is stored in Containers Algorithms STL provides standardized Algorithms Iterators Iterators bind Algorithms to Containers – are generalized pointers Example Container AlgorithmIterator vector vect; … // initialization of ‘vect’ variable sort(vect.begin(),vect.end());

6. 6Texas A&M UniversitySTAPL STAPL Overview pContainers Data is stored in pContainers pAlgorithms STAPL provides standardized pAlgorithms pRanges pRanges bind pAlgorithms to pContainers Similar to STL Iterators, but must also support parallelism

7. 7Texas A&M UniversitySTAPL pRange pRange is the Parallel Counterpart of STL iterator: – – Binds pAlgorithms to pContainers – – Provides an abstract view of a scoped data space – – data space is (recursively) partitioned into subranges More than an iterator since it supports parallelization – – Scheduler/distributor decides how computation and data structures should be mapped to the machine – – Data dependences among subranges can be represented by a data dependence graph (DDG) – – Executor launches parallel computation, manages communication, and enforces dependences

8. 8Texas A&M UniversitySTAPL pRange Provides random access to a partition of the data space –View and access provided by a collection of iterators describing pRange boundary pRanges are partitioned into subranges – Automatically by STAPL based on machine characteristics, number of processors, partition factors, etc. – Manually according to user-specified partitions pRange can represent relationships among subspaces as Data Dependence Graphs (DDG) ( for scheduling )

9. 9Texas A&M UniversitySTAPL pRange Each subspace is disjoint and could be itself a pRange – Nested parallelism Data Space stapl::pRange ::iterator> dataRange(segBegin, segEnd); dataRange.partition(); stapl::pRange ::iterator> dataSubrange = dataRange.get_subrange(3); dataSubrange.partition_like ( * size);

10. 10Texas A&M UniversitySTAPL pRange Each subspace is disjoint and could be itself a pRange – Nested parallelism Prange Data Space stapl::pRange ::iterator> dataRange(segBegin, segEnd); dataRange.partition(); stapl::pRange ::iterator> dataSubrange = dataRange.get_subrange(3); dataSubrange.partition_like ( * size);

11. 11Texas A&M UniversitySTAPL pRange Each subspace is disjoint and could be itself a pRange – Nested parallelism Prange Data Space subspace stapl::pRange ::iterator> dataRange(segBegin, segEnd); dataRange.partition(); stapl::pRange ::iterator> dataSubrange = dataRange.get_subrange(3); dataSubrange.partition_like ( * size);

12. 12Texas A&M UniversitySTAPL pRange Each subspace is disjoint and could be itself a pRange – Nested parallelism subspace Prange Data Space stapl::pRange ::iterator> dataRange(segBegin, segEnd); dataRange.partition(); stapl::pRange ::iterator> dataSubrange = dataRange.get_subrange(3); dataSubrange.partition_like ( * size);

13. 13Texas A&M UniversitySTAPL pRange Each subspace is disjoint and could be itself a pRange – Nested parallelism subspace Prange Data Space stapl::pRange ::iterator> dataRange(segBegin, segEnd); dataRange.partition(); stapl::pRange ::iterator> dataSubrange = dataRange.get_subrange(3); dataSubrange.partition_like ( * size);

14. 14Texas A&M UniversitySTAPL pRange Each subspace is disjoint and could be itself a pRange – Nested parallelism subspace Prange Data Space stapl::pRange ::iterator> dataRange(segBegin, segEnd); dataRange.partition(); stapl::pRange ::iterator> dataSubrange = dataRange.get_subrange(3); dataSubrange.partition_like ( * size);

15. 15Texas A&M UniversitySTAPL pContainer pContainer is the parallel counterpart of STL container – Provides parallel and concurrent methods Maintains internal pRange – Updated during insert/delete operations – Minimizes redistribution Completed: pVector, pList, pTree Example: pVector pRange STL vector

16. 16Texas A&M UniversitySTAPL pAlgorithm pAlgorithm is the parallel counterpart of STL algorithm Parallel Algorithms take as input – pRange – Work functions that operate on subRanges and apply the work function to all subranges template class pAddOne : public stapl::pFunction { public:... void operator()(SubRange& spr) { typename SubRange::iterator i; for (i=spr.begin(); i!=spr.end(); i++) (*i)++ } }... p_transform(pRange, pAddOne);

17. 17Texas A&M UniversitySTAPL Run-Time System Support for different architectures – HP V2200 – SGI Origin 2000, SGI Power Challenge Support for different paradigms – OpenMP, Pthreads – MPI Memory allocation – HOARD Run-Time Cluster 1 Proc 12 Proc 14 Proc 13 Proc 15 Cluster 4 pAlgorithm Cluster 2Cluster 3

18. 18Texas A&M UniversitySTAPL Run-Time System Scheduler – Determine an execution order (DDG) – Policies: Automatic :Static, Block, Dynamic, Partial Self Scheduling, Complete Self Scheduling User defined Distributor – Hierarchical data distribution – Automatic and user defined Executor – Execute DDG Processor assignment Synchronization and Communication

19. 19Texas A&M UniversitySTAPL STL to STAPL Automatic Translation C++ preprocessor converts STL code into STAPL parallel code Iterators used to construct pRanges User is responsible for safe parallelization #include accumulate(x.begin(), x.end(), 0); for_each(x.begin(), x.end(), foo()); #include pi_accumulate(x.begin(), x.end(), 0); pi_for_each(x.begin(), x.end(), foo()); p_accumulate(x_pRange, 0); p_for_each(x_pRange,foo()); Preprocessing phase pRange construction In some cases automatic translation provides similar performance to STAPL written code (5% deterioration)

20. 20Texas A&M UniversitySTAPL Performance: p_find Experimental results on HP V2200

21. 21Texas A&M UniversitySTAPL Performance: p_inner_product Experimental results on HP V2200

22. 22Texas A&M UniversitySTAPL pTree Parallel Tree supports bulk commutative operations in parallel Each processor is assigned a set of subtrees to maintain Operations on the base are atomic Operations on subtrees are parallel Base (atomic) Subtrees (parallel) P1 P2 P3 Example : Parallel Insertion Algorithm Each processor is given a set of elements 1) 1)Each proc creates local buckets corresponding to the subtrees 2) 2)Each processor collects the buckets that correspond to its subtrees 3) 3)Elements in the subtree buckets are inserted into tree in parallel

23. 23Texas A&M UniversitySTAPL pTree Basis for STAPL pSet, pMultiSet, pMap, pMultiMap containers – Covers all remaining STL containers Results are sequentially consistent although internal structure may vary Requires negligible additional memory pTrees can be used either sequentially or in parallel in the same execution – allows switching back and forth between parallel & sequential

24. 24Texas A&M UniversitySTAPL Performance: pTree Experimental results on HP V2200

25. 25Texas A&M UniversitySTAPL Algorithm Adaptivity Problem - Parallel algorithms are highly sensitive – Architecture – number of processors, memory interconnection, cache, available resources, etc – Environment – thread management, memory allocation, operating system policies, etc – Data Characteristics – input type, layout, etc Solution - implement a number of different algorithms and adaptively choose the best one at run-time

26. 26Texas A&M UniversitySTAPL Adaptive Framework

27. 27Texas A&M UniversitySTAPL Case Study - Adaptive Sorting SortStrengthWeakness ColumnTheoretically time optimal Many passes over data MergeLow memory overhead Poor scalability RadixExtremely fastIntegers only SampleTwo passes over data High memory overhead

28. 28Texas A&M UniversitySTAPL Performance: Adaptive Sorting V2200 Power Challenge Origin 2000 Performance on 10 million integers

29. 29Texas A&M UniversitySTAPL Performance: Run-Time Tests if (data_type = INTEGER) radix_sort(); else if (num_procs < 5) merge_sort(); else column_sort(); Origin 2000

30. 30Texas A&M UniversitySTAPL Performance: Molecular Dynamics* Discrete time particle interaction simulation – Written in STL – Time steps calculate system evolution (dependence) – Parallelized within time step STAPL utilization: – pAlgorithms: p_for_each, p_transform, p_accumulate – pContainers: pVector (push_back) – Automatic vs. Manual (5% performance deterioration ) * Code written by Danny Rintoul at Sandia National Labs

31. 31Texas A&M UniversitySTAPL Performance: Molecular Dynamics Number of particles 108K 23k Number of processors 1 4 8 12 16 2815 1102 546 386 309 627 238 132 94 86 Execution Time (sec) Experimental results on HP V2200 40%-49% parallelized Input sensitive Use pTree on rest

32. 32Texas A&M UniversitySTAPL Performance - Particle Transport* Generic particle transport solver – Regular and arbitrary grids – Numerically intensive, 25k line, C++ STAPL code – Sweep function unaware of parallel issues STAPL utilization: – pAlgorithms: p_for_each – pContainers: pVector (for data distribution) – Scheduler: determine grid data dependencies – Executor: satisfy data dependencies * Joint effort between Texas A&M Nuclear Engineering and Computer Science, funded by DOE ASCI

33. 33Texas A&M UniversitySTAPL Performance - Particle Transport Profile and Speedups on SGI Origin 2000 using 16 processors Code Region% Seq.Speedup Create computational grid10.0014.5 Scattering across group-sets0.05N/A Scattering within a group-set0.4015.94 Sweep86.8614.72 Convergence across group sets0.05N/A Convergence within group sets0.05N/A Other2.59N/A Total100.0014.70

34. 34Texas A&M UniversitySTAPL Performance - Particle Transport Experimental results on SGI Origin 2000

35. 35Texas A&M UniversitySTAPL Summary Parallel equivalent to STL – Many codes can immediately utilize STAPL – Automatic translation Building block library – Portability (layered architecture) – Performance (adaptive) – Automatic recursive parallelism STAPL performs well in small pAlgorithm test cases and large codes

36. 36Texas A&M UniversitySTAPL STAPL Status and Current Work pAlgorithms - fully implemented pContainers - pVector, pList, pTree pRange - mostly implemented Run-Time – Executorfully implemented – Scheduler fully implemented – Distributorwork in progress Adaptive mechanism (case study – sorting) OpenMP + MPI (mixed)work in progress – OpenMP versionfully implemented – MPI versionwork in progress

37. 37Texas A&M UniversitySTAPL http://www.cs.tamu.edu/research/parasol Project funded by Project funded by NSF NSF DOE DOE