1. CCA Common Component Architecture Distributed Array Component based on Global Arrays Manoj Krishnan, Jarek Nieplocha High Performance Computing Group Pacific Northwest National Laboratory CCA Forum

2. CCA Common Component Architecture Overview  Global Arrays  Distributed Array Component  Core Capabilities  Applications  Future Work

3. CCA Common Component Architecture Global Arrays physically distributed dense array single, shared data structure global indexing  shared memory model in context of distributed dense arrays  complete environment for parallel code development  compatible with MPI  ~140 functions  data locality control similar to distributed memory/message passing model e.g., A(4,3) rather than buf(7) on task 2

4. CCA Common Component Architecture Global Array Model of Computations compute/update local memory Shared Object copy to local memory 1-sided communication get Shared Object copy to shared object local memory 1-sided communication put

5. CCA Common Component Architecture Structure of GA Message Passing process creation, run-time environment ARMCI portable 1-sided communication put,get, locks, etc distributed arrays layer memory management, index translation application interfaces Fortran 77, C, C++, Python, SIDL system specific interfaces LAPI, GM/Myrinet, Elan/Quadrics, threads, VIA,..

6. CCA Common Component Architecture Distributed Array Component  GAComponent  GAComponent : Classic and SIDL Interfaces  36+98 (direct+indirect) global arrays classic methods are available through GAClassicPort  GADADFPort provides methods, proposed by Data Working Group of CCA Forum, for creating array descriptors and templates GA Linear Algebra DADF GA Classic

7. CCA Common Component Architecture GA Classic Port –provides public interfaces for creating and accessing distributed arrays i.e., GlobalArray objects GlobalArray –encapsulate all details of the data distribution, addressing, and data access. –offers a set of operations for one-sided data transfer operations (get, put, scatter, gather, etc) collective array operations supportive operations for data locality control and queries

8. CCA Common Component Architecture class GAClassicPort: public virtual ::classic::gov::cca::Port { /* array creation methods, for example */ virtual GlobalArray* createGA(…) = 0; virtual GlobalArray* createGA_Ghosts(…)=0; /* utility operations like reduce, broadcast, etc.,. */ virtual void brdcst(void *buf, int lenbuf, int root)=0; /* cluster & process information e.g. rank, size*/ nnodes(),clusterNnodes(),clusterNodeid(),clusterNprocs /* Interprocess Synchronization: locks, barrier */ lock(), unlock(), sync(), fence(), createMutexes(), … } /* Total: 36 methods available thru’ this port */ Class GlobalArray { /* one-sided communication operations */ put(), get(), accumulate(), scatter, gather,... /* collective array operations (whole and patch) */ copy(), scale(), add(), gemm(), update_ghosts(),... /* element wise operations, ghost cell methods, matrix operatios etc… */ } /* Total: 98 methods available */

9. CCA Common Component Architecture Core Capabilities  Distributed array dense arrays 1-7 dimensions four data types: integer, real, double precision, double complex global rather than per-task view of data structures user control over data distribution: regular and irregular  Collective and shared-memory style operations  Support for ghost cells  Interfaces to third party parallel numerical libraries PeIGS, Scalapack, SUMMA, TAO

10. CCA Common Component Architecture GA DADF Port  Provides standard interface for defining, creating and querying distributed arrays –Supports creating, cloning and destruction of arrays, array templates and descriptors –DADF-Distributed Array Descriptor Factory by Data Working Group of CCA forum.  DADF Array –creates a distributed array  DADF Template: –Virtual multi-dimensional array to which one or more actual distributed arrays may be aligned  DADF Descriptor –To query an existing distributed array

11. CCA Common Component Architecture class DADFPort: : public virtual ::classic::gov::cca::Port { /* methods to create/clone/destroy dscr,array,templates*/ virtual DistArrayDescriptor * createDescriptor(..) = 0; virtual DistArray * createArray (…) = 0; virtual DistArrayTemplate* createTemplate(…) = 0;... } class DistArray { /** Set data type. */ virtual int setDataType(const enum DataType type) = 0; /** Associate this data object with distribution template. */ virtual int setTemplate(DistArrayTemplate * & templ) = 0; /** Sets this process's location in the process topology. */ virtual int setMyProcCoords(const int procCoords[] ) = 0; /** Align object to template with identity mapping. */ virtual int setIdentityAlignmentMap() = 0; /** Signal that data object is completely defined. */ virtual int commit() = 0;... /* set of query & miscellaneous functions */ }

12. CCA Common Component Architecture Class Hierarchy DistArrayTemplateDistArray DADFDescriptorDADFArrayDADFTemplate GAX Example DAs

13. CCA Common Component Architecture GATAO addProvidesPortregisterUsesPort CCA Services GA DADF CCA Services LA getPort(“ga”) GA/TAO Interoperability  TAO - optimization component (Toolkit for Advanced Optimization – ANL) provides advanced optimization algorithms  GA provides TAO core linear algebra support for manipulating vectors, matrices, and linear solvers thru’ LinearAlgebraPort (LA)

14. CCA Common Component Architecture GA Component in Applications (I) GALJMD addProvides Port registerUses Port CCA Services GA DADF CCA Services GA getPort(“ga”) LA  Lennard Jones Molecular Dynamics  Force decomposition method & dynamic load balancing (improves performance over the traditional message-passing version by S.Plimpton, Sandia)  Component overhead is negligible (<1%)  Good scaling (simulation of 12,000 atoms yields a speedup of 7.86 on 8 processors)

15. CCA Common Component Architecture  Chemistry: Molecular geometry optimization (between GA and TAO) GA Component in Applications (II)

16. CCA Common Component Architecture GA Solver addProvidesPort registerUsesPort CCA Services GA DADF CCA Services GA getPort(“ga”) LA CFD registerUsesPort CCA Services GA Visualization registerUsesPort CCA Services GA getPort(“ga”) getPort(“ga”)

17. CCA Common Component Architecture Applications Areas thermal flow simulation Visualization and image analysis electronic structure glass flow simulation material sciences molecular dynamics Others: financial security forecasting, astrophysics, geosciences biology

18. CCA Common Component Architecture Future Work  Additional capabilities in GA component including operations necessary for supporting more TAO optimization algorithms.  will also involve new nonblocking communication interfaces.  Implementation of component that interfaces secondary storage (parallel I/O).  Verify component usability for large apps  Study performance and overhead associated with CCA  ESI (or any generic solver) interfaces to distributed array component

19. CCA Common Component Architecture Feedback  Provide a generic distributed array component  We would like to know  Applications that need distributed array components  Functionalities expected from apps  Additions/modifications required  Suggestions to make it more generic  Communication interfaces in DADF (put/get)..?  Setting up priorities based on feedback