1. James “Jeeembo” Kohl, M.C. Oak Ridge National Laboratory
2nd Biennial, Official, 100% Pure, CCA Approved… MxN Working Group Meeting Report!
(This is OUR brain…! )
(This is your brain…)
At The Radisson, Knoxville, TN
January 22, 2004
James “Jeeembo” Kohl, M.C.
Oak Ridge National Laboratory
ORNL/Kohl 2004
Research supported by the Mathematics, Information and Computational Sciences Office, Office of Advanced Scientific
Computing Research, U.S. Department of Energy, under contract No. DE-AC05-00OR22725 with UT-Battelle, LLC.

2. The “Basic” Problem: Parallel Data Exchange
Because the CCA departs from traditional component-based architectures, it is essential to provide specialized
interfaces for handling these challenging new parallel semantics.
The “MxN” thrust area focuses on providing a means for specifying how two parallel components, of potentially
different cardinality and decomposition type, can extract, redistribute and share parallel data.
Each specific data element on the “M” side must be mapped to a corresponding element on the “N” side.
Note that this most basic parallel data exchange capability, while still highly complex, does not yet even begin
to address the more complicated issues of spatial and temporal interpolation, or units conversion, as is necessary
for a full-fledged parallel model coupling environment.
“M”
ORNL/Kohl 2004

3. Cooperating Parallel Components…?!
CCA-specific capability (non-ad-hoc…)
Requirement of emerging scientific apps
Generalized model coupling (multi-parallel)
Efficient daisy-chaining of numerical libraries
Visualization (parallel rendering / serial gathering)
Fundamental issue ~ data decompositions
Different distribution for each parallel component
Need to describe/reconcile disparate data organizations
And, more than just parallel data…
…Parallel method invocations, too!
ORNL/Kohl 2004

4. MxN Internal Structure
High-Level MxN Run-Time Library
MxN Data Exchange and Synchronization Interface
Parallel Data Mapping and Communication
Data Decomposition Specification
Data Distribution System
Local Data
ORNL/Kohl 2004

5. MxN Progress to Date… Generalized MxN specification
Based on CUMULVS (ORNL) and PAWS (LANL)
SCMD peer MxN component solution
Visualization; combustion and climate apps…
Parallel Remote Method Invocation (PRMI)
Preliminary semantics and specifications
Distributed MxN solution
MPI I/O approach
ORNL/Kohl 2004

6. MxN “Explicit” Component Solution
Port-based direct invocation of MxN methods
Most general solution, but…
Most challenging to the end-user scientist.
“Assembly language” level interface…
Preliminary platform for experimentation, higher-level functs
Several demonstrations (SC00, SC01, SC02)
Increasingly elaborate visualization solutions
Incorporated generalized data description since SC01 (DADF)
Two main prototypes (reused several apps):
CumulvsMxN ~ first inter-framework component capabilities!
PawsMxN ~ ping-pong coupling experiments
ORNL/Kohl 2004

7. MxN Meeting Topics Overview of MxN Specification ~ Jeeeembo
Data Reorganization Interface (DRI) ~ David
Distributed Data Broker (DDB) ~ Dan
Distributed CCA Framework (DCA) ~ Felipe
ECho and MxN ~ Hasan
Open Discussion
ORNL/Kohl 2004

8. Data Reorganization Interface (DRI)
DARPA Effort for Distributed Array Support
No academic involvement (as opposed to CCA? )
Related to (absorbed by) HPSI effort…
Simple series of Puts/Gets for redistribution
Through user-managed buffers
Uses MPI as basis for the communication design
Evaluation: Different domain (real-time)
But we can steal some of their ideas…!
ORNL/Kohl 2004

9. Distributed Data Broker (DDB)
Designed for Earth System Model
Coupling 2 Atmosphere and 2 Ocean models
Works with MPI or Multi-MPI w/PVM
One large integrated parallel application (Fortran)
DDB library hierarchy on each processor
Handles generic rectangular data decompositions
Registration phase, then SendData / GetData…
Now Implemented as a CCA/Babel Component!
Nice Interface, Gruesome Internals
Steal ideas, simplicity of interface (for simple data)
ORNL/Kohl 2004

10. Distributed CCA Framework (DCA)
MPI-Based MxN-Capable Distributed FW
Most Parallel Cases Don’t Map to Distributed!
3 Levels of Parallelism:
“Component” ~ Distributed Computing
“Process” ~ SPMD/Parallel Computing
“Thread” ~ Node-localized Thread Computing
MPI Communicator Groups for Collective Calls
Caller dictates data layout at provider…
Point of controversy, how to specify provider decomp…?
ORNL/Kohl 2004

11. DCA Decomposition Issues
Static (SIDL) versus Dynamic (Run-Time) Meta-Data?
ORNL/Kohl 2004
© Copyright Images Felipe Bertrand 2004.

12. ECho and MxN Dynamic Publish/Subscribe (Event) Environment
Source-side filtering for data for redistribution
A la “one-sided” communication, but not quite… 
ECL (C-subset) dynamic code generation
PBIO efficient binary data format
PDS Proactive Directory Service for discovery
Simple application demo of SmartPointers for Viz
Future work ~ use XML for describing data distributions and custom filter generation…
ORNL/Kohl 2004

13. MxN Action Items / Tasks:
Parallel MxN Specification Changes:
Eliminate the requestTransfer() suite of methods…
Seemed like a good idea at the time… but no.
Add simplified interface alternatives
One-stop-shopping method, for all-in-one MxN-ing…
Create Combined Distributed/Parallel FW Demo
Co-located MxN and Application Components…
Allow data wrangling in shared address space (cheating!)
Combine multiple FWs as Distributed Components
Theoretically possible, need practical proof-of-concept.
ORNL/Kohl 2004

14. MxN Action Items / Tasks (cont.)
TSTT Mesh Integration with DADF
Use “General Map” Interface…
Distributed MxN and CCA Issues
Continue Education and Exploration of Issues
Parallel Rendering / Viz using MxN
Not just Mx1, but real MxN*…
ORNL/Kohl 2004

15. MxN Action Items / Tasks (cont.)
Killer Integration Application Demo!
TSTT + TOPS + CCA/Babel + DADF + MxN
Demonstrate full integration capabilities… (if any!)
Choose a simple contrived or synthetic appl…
Else perhaps a real application from Climate/Fusion:
Jay’s “Big Lake” dream code…
Ongoing Fusion integration opportunities…
ORNL/Kohl 2004

16. MxN Summary Stable specification and component solutions
Update and Streamline, Provide Simple Alternatives
Distributed/Parallel Framework MxN experiments
TSTT and DADF Integration
Continue Distributed MxN and CCA Front
Parallel Rendering / Viz Demo
Killer Integration Application Demo!
“Can’t we all get along…?”
ORNL/Kohl 2004