A Parallel Communication Infrastructure for STAPL Steven Saunders Lawrence Rauchwerger PARASOL Laboratory Department of Computer Science Texas A&M University

Overview STAPL The Parallel Communication Infrastructure the Standard Template Adaptive Parallel Library parallel superset to the C++ Standard Template Library provides transparent communication through parallel containers and algorithms The Parallel Communication Infrastructure foundation for communication in STAPL maintains high performance and portability simplifies parallel programming PARASOL Lab: Texas A&M

Common Communication Models level of abstraction PARASOL Lab: Texas A&M

STL Overview The C++ Standard Template Library set of generic containers and algorithms generically bound by iterators containers: data structures with methods algorithms: operations over a sequence of data iterators: abstract view of data abstracted pointer: dereference, increment, equality Example: std::vector<int> v( 100 ); ...initialize v... std::sort( v.begin(), v.end() ); PARASOL Lab: Texas A&M

STAPL Overview The Standard Template Adaptive Parallel Library set of generic, parallel containers and algorithms generically bound by pRanges pContainers - distributed containers pAlgorithms - parallel algorithms pRange - abstract view of partitioned data view of dist. data, random access, data dependencies Example: stapl::pvector<int> pv( 100 ); ...initialize pv... stapl::p_sort( pv.get_prange() ); PARASOL Lab: Texas A&M

Fundamental Requirements for the Communication Infrastructure statement: tell a process something question: ask a process for something Synchronization mutual exclusion: ensure atomic access event ordering: ensure dependencies are satisfied PARASOL Lab: Texas A&M

Design Goal: abstract underlying communication model to enable efficient support for the requirements focus on parallelism for STL-oriented C++ code Solution: message passing makes C++ STL code difficult shared-memory not yet implemented on large systems remote method invocation can support the communication requirements can support high performance through message passing or shared-memory implementations maps cleanly to object-oriented C++ PARASOL Lab: Texas A&M

Design Communication Synchronization statement: question: template<class Class, class Rtn, class Arg1…> void async_rmi( int destNode, Class* destPtr, Rtn (*method)(Arg1…), Arg1 a1… ) question: Rtn sync_rmi( int destNode, Class* destPtr, groups: void broadcast_rmi(), Rtn collect_rmi() Synchronization mutual exclusion: remote methods are atomic event ordering: void rmi_fence(), void rmi_wait() PARASOL Lab: Texas A&M

Design: Data Transfer Transfer the work to the data only one instance of an object exists at once no replication and merging as in DSM transfer granularity: method arguments pass-by-value: eliminate sharing Argument classes must implement a method that defines its type internal variables are either local or dynamic used to pack/unpack as necessary PARASOL Lab: Texas A&M

Integration with STAPL User Code pAlgorithms pContainers pRange Address Translator Communication Infrastructure Pthreads OpenMP MPI Native PARASOL Lab: Texas A&M

Integration: pContainers Set of distributed sub-containers pContainer methods abstract communication decision between shared-memory/message passing made in communication infrastructure Communication patterns: access: access data in another sub-container handled by sync_rmi update: update data in another sub-container handled async_rmi group update: update all sub-containers handled by broadcast_rmi PARASOL Lab: Texas A&M

Integration: pAlgorithms Set of parallel_task objects input per parallel_task specified by the pRange intermediate results stored in pContainers RMI for communication between parallel_tasks Communication patterns: event ordering: tell workers when to start data parallel: apply operation in parallel followed by a parallel reduction handled by collect_rmi bulk communication: large number of small messages handled async_rmi PARASOL Lab: Texas A&M

Case Study: Sample Sort Common parallel algorithm for sorting based on distributing data into buckets Algorithm: sample from the input a set of splitters send elements to appropriate bucket based on splitters (e.g., elements less than splitter 0 are sent to bucket 0) sort each bucket splitters input buckets PARASOL Lab: Texas A&M

Case Study: Sample Sort //...all processors execute code in parallel... ... stapl::pvector<int> splitters( p-1 ); splitters[id] = //sample stapl::pvector< vector<int> > buckets( p ); for( i=0; i<size; i++ ) { //distribute int dest = //...appropriate bucket based on splitters... stapl::async_rmi( dest, ..., &stapl::pvector::push_back, input[i] ); } stapl::rmi_fence(); sort( bucket[id].begin(), bucket[id].end() ); //sort template<class T> T& pvector<T>::operator[](const int index) { if( /*...index is local...*/ ) return //...element... else return stapl::sync_rmi( /*owning node*/, ..., &stapl::pvector<T>::operator[], index ); PARASOL Lab: Texas A&M

Implementation Issues RMI request scheduling tradeoff local computation with incoming RMI requests current solution: explicit polling async_rmi automatic buffering to reduce network congestion rmi_fence deadlock: native barriers block while waiting must poll while waiting current solution: custom fence implementation completion: RMI’s can invoke other RMI’s... current solution: overlay a distributed termination algorithm PARASOL Lab: Texas A&M

Performance Two implementations: Three benchmark platforms: Pthreads (shared-memory) MPI-1.1 (message passing) Three benchmark platforms: Hewlett Packard V2200 16 processors, shared-memory SMP SGI Origin 3800 48 processors, distributed shared-memory CC-NUMA Linux Cluster 8 processors, 1Gb/s Ethernet switch PARASOL Lab: Texas A&M

Latency Overhead due to high cost of RMI request creation and scheduling versus low cost of communication. Overhead due to native MPI optimizations that are not applicable with RMI. Time (us) to ping-pong a message between two processors using explicit communication or STAPL (async_rmi/sync_rmi) PARASOL Lab: Texas A&M

Effect of Automatic Aggregation PARASOL Lab: Texas A&M

Effect of Automatic Aggregation PARASOL Lab: Texas A&M

Native Barrier vs. STAPL Fence Overhead due to native optimizations within MPI_Barrier that are unavailable to STAPL. Message Passing Majority of overhead due to cost of polling and termination detection. Shared-Memory PARASOL Lab: Texas A&M

Sample Sort for 10M Integers PARASOL Lab: Texas A&M

Sample Sort for 10M Integers PARASOL Lab: Texas A&M

Inner Product for 40M Integers Time (s) to compute the inner product of 40M element vectors using shared-memory PARASOL Lab: Texas A&M

Inner Product for 40M Integers Time (s) to compute the inner product of 40M element vectors using message passing PARASOL Lab: Texas A&M

Conclusion STAPL The Parallel Communication Infrastructure Future Work provides transparent communication through parallel containers and algorithms The Parallel Communication Infrastructure foundation for communication in STAPL maintains high performance and portability simplifies parallel programming Future Work mixed-mode MPI and OpenMP additional implementation issues PARASOL Lab: Texas A&M