OpenMP for Networks of SMPs Y. Charlie Hu, Honghui Lu, Alan L. Cox, Willy Zwaenepoel ECE1747 – Parallel Programming Vicky Tsang

Background Published in the Journal of Parallel and Distributed Computing, vol. 60 (12), pp , December 2000 Work to further improve TreadMarks Presents an alternative solution to MPI

Roadmap Motivation Solution OpenMP API TreadMarks OpenMP Translator Performance Measurement Results Conclusion

Motivation To enable the programmer to reply on a single, standard, shared-memory API for parallelization within and between multiprocessors. To provide another standard other than MPI?

Solution Presents the first system that implements OpenMP on a network of shared-memory multiprocessors Implemented via a translator converting OpenMP directives to calls in modified TreadMarks Modified TreadMarks uses POSIX threads for parallelism within an SMP node

Solution Original version of TreadMarks:  A Unix process was executed on each processor of the multiprocessor node and communication between processes was achieved through message passing  Fails to take advantage of hardware shared memory

Solution Modified version of TreadMarks  POSIX threads used to implement parallelism  OpenMP threads within a multiprocessor share a single address space  Positive: Reduces the number of changes to TreadMarks to support multithreading on a multiprocessor OS maintains the coherence of page mappings automatically  Negative: More difficult to provide uniform sharing of memory between threads on the same node and threads on different nodes

OpenMP API Three kinds of directives:  Parallelism/work sharing  Data environment  Synchronization Based on a fork-join model Sequential code sections executed by master thread Parallel code sections are executed by all threads, including the master thread

OpenMP API Parallel directive – all threads perform the same computation Work sharing directive – computation is divided among the threads Data environment directive – control the sharing of program variables Synchronization directive – control the synchronization between threads

TreadMarks User-level SDSM system Provides a global shared address space on top of physically distributed memories Key functions performed are memory coherence and synchronization

TreadMarks – Memory Coherence Minimize the amount of communication performed to maintain memory consistency by:  a lazy implementation of release consistency  reducing the impact of false sharing by allowing multiple concurrent writers to modify a page Propagation of consistency information is postponed until the time of an acquire

TreadMarks - Synchronization Barrier implemented as acquire and release messages Governed by a centralized manager

TreadMarks – Modifications for OpenMP Inclusion of two primitives:  Tmk_fork  Tmk_join All threads created at the start of a program’s execution to minimize overhead. Slave threads are blocked during sequential execution until the next Tmk_fork is issued by the master thread.

TreadMarks – Modifications for Networks of Multiprocessors POSIX thread enabled sharing of data between processors. Addition of some data structures, such as message buffers, in thread-private memory for data that is to remain private within a thread. A per-page mutex was added to allow greater concurrency in the page fault handler. Synchronization functions in TreadMarks were modified to use POSIX thread-based synchronization between processors within a node and existing TreadMarks synchronization functions between nodes. A second mapping was added for the memory that is shared between nodes so shared-memory pages can be updated while the first mapping remains invalid until the update is complete. This reduces the number of page protection operations performed by TreadMarks.

OpenMP Translator Synchronization directives translate directly to TreadMarks synchronization operations. The complier translates the code sections marks with parallel directives to fork-join code. Data environment directives implemented to work with both TreadMarks and POSIX threads, hiding the interface issues from the programmer.

Performance Measurement Platform  IBM SP2 consisting of four SMP nodes  Per node: Four IBM PowerPC 604 processors 1 GB memory Running AIX 4.2

Performance Measurement Applications  SPLASH-2 Barnes-Hut  NAS 3D-FFT  SPLASH-2 CLU  SPLASH-2 Water  Red-Black SOR  TSP  Modified Gramm-Schmidt (MGS)

Results

Conclusion Enables the programmer to rely on a single, standard, shared-memory API for parallelization within and between multiprocessors. Using shared hardware memory reduced data and messages transmitted. The speedups of multithreaded TreadMarks codes on four four-way SMP SP2 nodes are within 7-30% of the MPI versions.

Critique Solution allows easier implementation of program parallelization across multiprocessors if speedup is not crucial OpenMP is easier on the programmer but speedup still not as good as MPI

Critique Issues:  AIX has inefficient implementation of page protection Paper claims that every other brand of Unix, including Linux, uses data structures that handle mprotect operations more efficiently Why wasn’t the solution implemented on another platform?  Paper failed to present a big motivation for using this solution over MPI.

Thank You