1. Performance Tools Interface for OpenMP A presentation to the OpenMP Futures Committee
Allen D. Malony
Computer & Information Science Department
Computational Science Institute
University of Oregon

2. Outline Goals for OpenMP performance tools interface
Performance state and event model
Fork-join execution states and events
Performance measurement model
Event generation (callback) interface
Proposal based on directive transformation
Sample transformations
Comments and other additions
Describing execution context
General Issues
Experience using TAU with OpenMP/MPI
January 29, 2019

3. Goals for an OMP Performance Tools Interface
Goal 1: Expose OpenMP events and execution states to a performance measurement system
What are the OpenMP events / states of interest?
What is the nature (mechanism) of the interface?
Goal 2: Make the performance measurement interface portable
“Standardize” on interface mechanism
Define interface semantics and information
Goal 3: Support source-level and compiler-level implementation of interface
Source transformation and compiler transformation
January 29, 2019

4. Performance State and Event Model
Based on performance model for (nested) fork-join parallelism, multi-threaded work-sharing, and thread-based synchronization
Define with respect to multi-level state view
Level 1: serial and parallel states (with nesting)
Level 2: work-sharing states (per team thread)
Level 3: synchronization states (per team thread)
Level 4: runtime system (thread) states
Events reflect state transitions
State enter / exit (begin / end)
State graph with event edges
January 29, 2019

5. Fork-Join Execution States and Events
master slave
Parallel region operation
master starts serial execution X S
parallel region begins X
slaves started X
team begins parallel execution X X P
team threads hit barrier X X
slaves end; master exits barrier X X
master resumes serial execution X S
January 29, 2019

6. Performance Measurement Model
Serial performance
Detect serial transition points
Standard events and statistics within serial regions
Time spent in serial execution
Locations of serial execution in program
Parallel performance
Detect parallel transitions points
Time spent in parallel execution
Region perspective and work-sharing perspective
Performance profiles kept per region
More complex parallel states of execution
January 29, 2019

7. Event Generation (Callback) Interface
Generic event callback function (pseudo format)
omperf(eventID, contextID[, data])
Single callback routine
Must define events (not necessarily standardize)
Place burden on callback routine to interpret eventID
omperf_{begin/end}(eventID, contextID[, data])
Directive-specific callback functions (pseudo format)
omperf_{directive}_{begin/end/…}(contextID[, data])
Standardize function names
What about execution context data?
January 29, 2019

8. Instrumentation Alternatives
Source-level instrumentation
Manual instrumentation
TAU performance measurement
Directive transformation
Compiler instrumentation
Could allow more efficient implementation
JOMP (EPCC), Java Instrumentation Suite (Barcelona)
Runtime system instrumentation
Use to see RTL events, in addition to OMP events
GuideView (KAI/Intel)
Dynamic instrumentation
January 29, 2019

9. Proposal Based on Directive Transformation
Consider source-level approach
For each OMP directive, generate an “instrumented” version which calls the performance event API.
What is the event model for each directive?
Issues
OMP RTL execution behavior is not fully exposed
May not be able to generate equivalent form
Possible conflicts with directive optimization
May be less efficient
Hard to access RTL events and information
Sample transformations (B. Mohr, KFA)
January 29, 2019

10. Example: parallel regions, work-sharing (do)
Parallel region (parallel)
#omp parallel omperf_parallel_fork(regionID) #omp parallel omperf_parallel_begin(regionID)
#omp end parallel omperf_parallel_end(regionID) omperf_barrier_begin(regionID) #omp barrier omperf_barrier_end(regionID) #omp end parallel omperf_parallel_join(regionID)
#omp is just pseudo notation
IDs vs. context descriptor (see below)
Work-sharing (do/for)
#omp do omperf_do_begin(loopID) #omp do
#omp end do nowait #omp end do nowait omperf_do_end(loopID)
#omp end do #omp end do nowait omperf_do_end(loopID) omperf_barrier_begin(loopID) #omp barrier omperf_barrier_end(loopID)
January 29, 2019

11. Example: work-sharing (sections)
#omp sections omperf_sections_begin(sectionsID) #omp sections
#omp section (first section only) #omp section omperf_section_begin(sectionID)
#omp section (other sections only) omperf_section_end(prevsectionID) #omp section omperf_section_begin(sectionID)
Work-sharing (sections)
#omp end sections nowait omperf_section_end(lastsectionID) #omp end sections nowait omperf_sections_end(loopID)
#omp end sections omperf_section_end(lastsectionID) #omp end sections nowait omperf_barrier_begin(sectionsID) #omp barrier omperf_barrier_end(sectionsID) omperf_sections_end(sectionsID)
January 29, 2019

12. Example: work-sharing (single, master)
#omp single omperf_single_enter(singleID) #omp single omperf_single_begin(singleID)
#omp end single nowait omperf_single_end(singleID) #omp end single nowait omperf_single_exit(singleID)
#omp end single omperf_single_end(singleID) #omp end single nowait omperf_barrier_begin(singleID) #omp barrier omperf_barrier_end(singleID) omperf_single_exit(singleID)
Work-sharing (master)
#omp master #omp master omperf_master_begin(regionID)
#omp end master omperf_master_end(regionID) #omp end master
January 29, 2019

13. Example: synchronization (critical, atomic, lock)
Mutual exclusion (critical section)
#omp critical omperf_critical_enter(criticalID) #omp critical omperf_critical_begin(criticalID)
#omp end critical omperf_critical_end(criticalID) #omp end critical omperf_critical_exit(criticalID)
Mutual exclusion (automic)
#omp atomic omperf_atomic_begin(atomicID) #omp atomic atomic-expr-stmt omperf_atomic_end(atomicID)
Mutual exclusion (lock routines)
omp_set_lock(lockID) omperf_lock_set(lockID) omp_set_lock(lockID) omperf_lock_acquire(lockID)
omp_unset_lock(lockID) omp_unset_lock(lockID) omperf_lock_unset(lockID)
omp_test_lock(lockID) …
Overhead issues here
January 29, 2019

14. Comments Appropriate transformations for short-cut directives
#omp parallel do  #omp parallel sections
Performance initialization and termination routines
omperf_init()  omperf_finalize()
User-defined naming to use in context description
New attribute? New directive? Runtime function?
RTL events and information
How to get thread information efficiently?
How to get thread-specific context data?
Supports portability and source-based analysis tools
January 29, 2019

15. Other Additions Support for user-defined events Measurement control
!$omp perf event ...
#pragma omp perf event …
Place at arbitrary points in program
Translated by compiler into corresponding omperf()
Measurement control
!$omp perf on/off
#pragma omp perf on/off
Place at “consistent” points in program
Translate by compiler into omperf_on/off()
January 29, 2019

16. Describing Execution Context (B. Mohr)
Describe different contexts through context descriptor
struct region_descr { char name[]; /* region name */ char filename[]; /* source file name */ int begin_lineno; /* begin line # */ int end_lineno; /* end line # */ WORD data[4]; /* unspecified data */ struct region_descr* next; };
Generate context descriptors in global static memory:
struct region_descr rd42675 = { “r1”, “foo.c”, 5, 13 };
Table of context descriptors
January 29, 2019

17. Describing Execution Context (continued)
Pass descriptor address (or ID) to performance callback
Advantages:
Full context information available, including source reference
But minimal runtime overhead
just one argument needs to be passed
implementation doesn’t need to dynamically allocate memory for performance data!!
context data initialization at compile time
Context data is kept together with executable
avoids problems of locating (the right) separate context description file at runtime
January 29, 2019

18. General Issues Portable performance measurement interface
OMP event-oriented (directives and RTL operation)
Generic “standardized” performance event interface
Not specific to any particular measurement library
Cross-language support
Performance measurement library approach
Profiling and tracing
No built-in (non-portable) measurement
Overheads vs. perturbation
Iteration measurement overhead can be serious
Dynamic instrumentation – is it possible?
January 29, 2019

19. TAU Architecture
Dynamic
January 29, 2019

20. Hybrid Parallel Computation (OpenMPI + MPI)
Portable hybrid parallel programming
OpenMP for shared memory parallel programming
Fork-join model
Loop level parallelism
MPI for cross-box message-based parallelism
OpenMP performance measurement
Interface to OpenMP runtime system (RTS events)
Compiler support and integration
2D Stommel model of ocean circulation
Jacobi iteration, 5-point stencil
Timothy Kaiser (San Diego Supercomputing Center)
January 29, 2019

21. OpenMP + MPI Ocean Modeling (Trace)
Thread message pairing
Integrated OpenMP + MPI events
January 29, 2019

22. OpenMP + MPI Ocean Modeling (HW Profile)
% configure -papi=../packages/papi -openmp -c++=pgCC -cc=pgcc
-mpiinc=../packages/mpich/include -mpilib=../packages/mpich/libo
Integrated OpenMP + MPI events FP
instructions
January 29, 2019