1 1 What does Performance Across the Software Stack mean?  High level view: Providing performance for physics simulations meaningful to applications  To FASTMath it means Providing performance for people solving large systems of equations Providing performance for mesh-based simulations Providing performance for multiphysics simulations which can include a combination of mesh and particle based methods  Will focus on performance for mesh-based simulations Even at this level there is a substantial difference between structured and unstructured mesh methods Performance Across the Software Stack

2 2 (A simplified view by an unstructured mesh person – need proper technical description from structured mesh people) Core components of adaptive structured mesh system  Defining the physics on the grids  Management of the grids and their interaction  Dynamic load balancing of the grids  Assembly of algebraic equations  Solving of the systems of algebraic equations over the grids Implementation approach used  Provide an integrated infrastructure for the grids and the solutions of the systems of algebraic equations over the grids  Provide hooks for adding physics through defining the input to stencil operations Structured Mesh Simulations

3 3 Software implementations of structured mesh methods  Provide a parallel framework to support The interactions of the grids Adaptive control of the grids Assemble of the systems of algebraic systems Solution of the algebraic systems Defining stencil operations to account for physics  Key software stack integration points Integration with solver libraries (in reality influenced by both the grid infrastructure and physics being solved) Integration of new physics (note – expect it is not as simple as it “provide a new stencil sounds”) Structured Mesh Simulations

4 4 Historic focus has been on the  Physics equation discretization alternatives  Construction and solution of the algebraic systems  Result is development of unstructured mesh analysis codes Fact that unstructured mesh generation/adaptation can be automated introduced the need for a mesh infrastructure Two approaches to supporting unstructured mesh workflows  Components that can integrate with existing unstructured mesh simulation codes  Provide an unstructured mesh simulation infrastructure for solving new problems  SciDAC applications require both options Unstructured Mesh Simulations

5 5  Mesh generation and adaptation can be automated using design data input Any combinations of CAD and triangulations Voxel (image) to model to mesh capabilities Extensive control of mesh types, orders and layouts – boundary layer, anisotropic, gradation, etc. Curved element meshes Parallel mesh and distributed geometry Mesh generation in parallel Mesh adaptation in parallel Mesh Generation and Adaptation

6 6 Need uniform interface to solve multi-component problems on unstructured meshes  Create a native MOAB implementation that exposes the underlying array-based mesh data structures through the DM (Discretization Manager) object in PETSc (DMMoab)  Discretize the physics PDE described on MOAB mesh while leveraging the scalability of PETSc solvers.  Provide routines to build simple meshes in-memory or load an unstructured grid from file.  Analyze efficient unstructured mesh traversal, FD/FEM-type operator assembly for relevant problems in multi- dimensions. MOAB Discretization Manager

7 7 Albany – Agile Component Architecture Main PDE Assembly Nonlinear Solvers Field Manager Discretization Albany Glue Code Nonlinear Model Nonlinear Transient Optimization UQ Analysis Tools Iterative Linear Solvers Multi-Level Mesh Tools Mesh Adapt PUMI Problem Discretization ManyCore Node Multi-Core Accelerators Application Linear Solve Input Parser Node Kernels Libraries Interfaces PDE Terms Load Balancing

8 8 Components  Unstructured mesh infrastructure Parallel Mesh infrastructures Mesh Generation/Adaptation (includes linkage to geometry) Fields Solution transfer  Dynamic load balancing  Unstructured mesh solver Physics equation discretization System formation System solution Component Based Approach i M 0 j M 1 1 P 0 P 2 P inter-process part boundary intra-process part boundary Proc j Proc i

9 9 Components in Parallel Adaptive Analysis

10 File transfer a serious bottleneck in parallel simulation workflows  All core parallel data and services accessed through APIs  In-memory integration approach uses APIs Migration from file-based components to in-memory  Modify/extend components by wrapping data structures with APIs for:  read/write  memory management  inter-language coupling; typically FORTRAN and C  In-memory has far superior parallel performance In-Memory Coupling of Simulation Components

11 Electromagnetics Analysis

12  Adaptation based on Tracking particles Discretization errors  Full accelerator models Approaching 100 cavities Substantial internal structure Meshes with several hundred million high- order curved elements High-Order EM Coupled with PIC

13 Structural Analysis for Integrated Circuits

14 Must construct 3-D non-manifold solid from input geometry  Input domain defined in terms of 2-D layouts (gdsII/OASIS)  Third dimension based on process knowledge  A component has been developed to construct the model Adaptive loop constructed for thermally loaded case including thin liner Structural Analysis for Integrated Circuits Model of liner film only