1. CCA Common Component Architecture Manoj Krishnan Pacific Northwest National Laboratory MCMD Programming and Implementation Issues

2. CCA Common Component Architecture 2 Motivation The challenges in developing large-scale applications are … –Addressing complexity Improve productivity –Scaling to massive number of processors How applications can exploit the massive amount of parallelism available in teraflop and petaflop-scale systems

3. CCA Common Component Architecture 3 Multilevel Parallelism in Computational Chemistry: Our Approach Proposed solution to improve scalability –Increase granularity of computation => improve the overall scalability. –Exploitation of multiple levels of parallelism (MLP) Instead of execution entire application on the full set of processors, assign parts of application to appropriately-sized subsets of processors Many apps qualify –Challenge: Difficult to implement Use advanced tools to address programming complexity Common Component Architecture (CCA) Global Arrays (GA) shared-memory programming model Objective: To demonstrate how CCA and GA can be used together to address requirements of real scientific applications

4. CCA Common Component Architecture 4 Technology Technologies for exploiting multiple level parallelism –Global Arrays (GA) shared-memory programming model High level parallel data management abstractions –Common Component Architecture (CCA) Component technology for HPC applications Hiding complexity Enables composition of software modules written in different languages and programming styles Driver Gradient Energy CCA QM

5. CCA Common Component Architecture 5 Multiple Component Multiple Data Model Introducing Multiple Component Multiple Data –i.e. multiple program multiple data (MPMD) model in context of CCA –instantiating components on subgroups of processors –create a dynamic environment to partition computational resources and manage them to execute the overall application effectively Facilitate dynamic behavior of the application itself for example –Resizing processor groups based on memory requirements or scaling characteristics –swapping components based on numerical or computational performance

6. CCA Common Component Architecture 6 Numerical Hessian Example Numerical Hessian Algorithm –determination of energy second derivatives through numerical differentiation of gradients, which may in turn be obtained from numerical differentiation of energies Multiple gradient calculations –Each gradient has multiple energy calculations limited scalability Not effectively utilizing variable degrees of parallelism Gradient Energy Hessian

7. CCA Common Component Architecture 7 Numerical Hessian Scalability - I Single Energy Calculation Single energy calculation does not scale beyond 4 processes* Two-level Parallelism –Native parallel code – Energy level –group-based energy calculations at gradient level using GA processor groups QM Gradient Energy

8. CCA Common Component Architecture 8 Multilevel Parallelism Combining SPMD and MPMD paradigms MCMD – Multi Component Multiple Data MPMD + Component The MCMD Driver launches multiple instances of NWChem QM components on subsets of processors (CCA) Each NWChem QM (gradient) component does multiple energy computations on subgroups (GA) MCMD Hessian Driver Go cProps ModelFactory NWChem_QM_1 ModelFacto ry cProps Param Port Energy NWChem_QM_0 ModelFacto ry cProps Param Port NWChem_QM_2 ModelFacto ry cProps Param Port NWChem_QM_n ModelFacto ry cProps Param Port Driver Gradient Energy CCA QM Gradient Energy

9. CCA Common Component Architecture 9 Multiple Component Multiple Data (CCA’s MCMD Model) MCMD Driver Go cProps ModelFactory Builder Builder Service Builder cProps QM_0 ModelFact ory cProps Parameter QM_0 ModelFact ory cProps Parameter QM_0 ModelFact ory cProps Parameter QM_0 ModelFact ory cProps Parameter MCMD Driver Create new components Create processor groups Assign processor groups to components Connect components Collect results Energy Collect Results

10. CCA Common Component Architecture 10 Numerical Hessian Scalability - II Application efficiency improved 10x times on 256 CPUs Three-level ParallelismThree-level Parallelism Energy-Level –Native parallel code Gradient-Level –group-based single energy calculations using GA groups Hessian Level –Task-based gradient calculations using CCA

11. CCA Common Component Architecture 11 Potential Applications Relevant To This Approach Molecular Dynamics Monte Carlo –Growth nucleation Numerical Hessians –Vibrational spectra Optimization techniques –Simulated annealing with local optimization Nudged Elastic Band methods –Determine reaction path for kinetic rates Trajectory simulations

12. CCA Common Component Architecture 12 MCMD Programming Multi-level parallelism –Nested parallel decomposition –Possibly multiple levels of parallelism –Multiple parallel simulations are run concurrently in a coupled fashion, exchanging data at boundaries or perhaps even within volumes.

13. CCA Common Component Architecture 13 MCMD Services Develop MCMD services to support MLP –Creating and management of processor groups CCA Represenation for Groups id, membership –Mapping of component to groups and their coordination Coordination of concurrent and nested SCMD/MCMD tasks –Communication between groups –Dynamic reconfiguration –Handling termination of processor groups, components MCMD as a service or a component ?

14. CCA Common Component Architecture 14 Activities Year 1: –Develop a model to express Multi-level parallelism through processor groups –Requirements gathering and design of flexible dynamic multi-level parallelism model –Coordinate & interact with other initiatives (ongoing) Year 2 –Define a CCA Standard way of specifying and translating processor group membership and mapping between components Year 3, 4, 5. –…

15. CCA Common Component Architecture 15 Implications of MCMD for CCA model Model for Applications with Multi-Level Parallelism – Important Process group abstraction – compatible with MPI, PVM, GA, GAS languages, HPCS languages (?) –MPI as default ? Group translators –How to address threaded components? OpenMP? Pthreads? Processor group for a threaded component? Group-awareness to CCA and a CCA way of naming groups –i.e. multi-level parallelism at the CCA level/BuilderService

16. CCA Common Component Architecture 16 Implications of MCMD for CCA Implementations Processor group management Run-time configuration –At run-time, user should be able to blow-up connections, create components and assign groups –Swapping components,.. Mapping communicators Overlapping/Disjoint processor groups

17. CCA Common Component Architecture 17 Summary - Found MCMD Effective Implemented a flexible, multi-level software architecture for computational chemistry applications –Exploits variable levels of parallelism –A order of magnitude of performance improvement Hiding complexity and enabling better s/w composition MCMD model has potential for addressing scalability in future large scale systems More work is needed in CCA infrastructure and s/w to take advantage for larger class of apps –Facilitate dynamic groups Make MCMD easier to adopt for apps