1.  A.C. Bauer, M.S. Shephard, E. Seol and J. Wan,  (acbauer@scorec.rpi.edu)  Scientific Computation Research Center  Rensselaer Polytechnic Institute, Troy, NY 12180  Outline: TSTT simulation infrastructure development Structures to support the construction of adaptive loops Mesh modification tools for adaptive mesh control Adaptive loop for DOE accelerator modeling code Adaptive loop for evolving geometry problem solved using commercial analysis engine The TSTT Interface for Mesh-Based Simulations

2. Terascale Simulation Tools & Technologies (TSTT) Center Goal: To enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales Adaptive methods Composite or hybrid solution strategies High-order discretization strategies  Barrier: The lack of easy-to-use interoperable meshing, discretization and adaptive tools requires too much software expertise by application scientists Organizations: DOE Labs: Argonne, Brookhaven, Lawrence Livermore, Oak Ridge, Pacific Northwest, Sandia Universities: Rensselaer Polytechnic Institute, SUNY Stony Brook, University of British Columbia Project Managers: David Brown: dlb@llnl.gov Lori Freitag Diachin: diachin2@llnl.gov Jim Glimm: glimm@ams.sunysb.edu More information: www.tstt-scidac.org

3. TSTT Interoperability Goal Accomplishing this goal requires: An abstract data model that encompasses a broad spectrum of mesh types and usage scenarios A set of common interfaces ● Implementation and data structure neutral ● Small enough to encourage adoption ● Flexible enough to support a broad range of functionality Accomplishing this goal requires: An abstract data model that encompasses a broad spectrum of mesh types and usage scenarios A set of common interfaces ● Implementation and data structure neutral ● Small enough to encourage adoption ● Flexible enough to support a broad range of functionality Geometry Information (Level A) Full Geometry Meshes (Level B) Mesh Components (Level C) To provide interchangeable and interoperable access to different mesh management and discretization strategies Ease experimentation with different technologies Combine technologies together for hybrid solution techniques A different way of thinking about simulation code development

4. TSTT Simulation Infrastructures  7 Interacting structures Mathematical problem definition 1) Space/time domain *** 2) Physical and mathematical model attributes First level of discretization 3) Mesh *** Second level of discretization 4) Discretization operators acting on mathematical equations and degrees-of- freedom (dof) Application of #4 over mesh entities 5) Algebraic system contributors Solving for dof values 6) Algebraic system Description of the physical parameters 7) Fields *** *** Main TSTT focus to date

5. Space/Time Domain Overall domain definition - geometric model and time domain Want a “true definition” of the domain - not a specific approximation Define other structures in relation to the domain definition Problem attributes defined over portions of the domain Mesh represents discretization of domain Fields are discretizations of physical tensors defined over the domain  At SCOREC, direct geometric model access via topology Shape independent abstraction Need general combination of solids, surfaces and curves Non-manifold topology representations known Effectively supported by commercial solid modelers Topology built from topology in modeling source Geometric queries passed through modeler API - supports auto- meshing

6. Attributes  Problem definition attributes Governing equations, material properties, loads, b.c.’s  Solution strategy attributes Meshing parameters, PDE discretization, basis functions, etc.  Attributes applied to geometric model  Problem definition attributes: Are tensors Have distributions Have relationships  Classification used to retrieve attributes

7. TSTT Mesh Data Model  Data model consists of entities and entity sets Entity definition Unique topology Canonical ordering defines adjacency relationships Entity set definition Arbitrary grouping of TSTT mesh entities There is a single “Root Set” Relationship among entity sets  Contained-in  Hierarchical Mesh Definition Simple meshes are a set of connected, non-overlapping entities related through adjacency relationships Meshes can contain collections of simple meshes

8. TSTT Mesh Interface Functionality  Core functionality The smallest set of functions required to be considered TSTT compliant – extensions for implementation of advanced functionality Provides basic access to vertex coordinates and adjacency information Mesh loading and saving Accessing global information such as the root set, geometric dimension, number of entities of a given type or topology Primitive array access for entity sets Global entity arrays for entity sets  Traversal functionality Advanced traversal functions provide additional flexibility and performance Single Entity Traversal  Iterator functionality (init, next, reset, end)  Query (topology, type, adjacency) Workset Traversal  Block iterators allow access of entities in a user-defined array size  Array based query functions for topology, type, adjacency  Increases performance by reducing the number of calls through the interface  Interface is not as intuitive

9. TSTT Mesh Modification Operators  Simple mesh modification operations are also supported Modifications Changing vertex location Add and delete entities Caveats No validity checks Requires simple classification mechanisms against the geometric model Intended to support higher-level functionality Mesh quality improvement Adaptive schemes with validity checking Front tracking procedures Basic mesh generation algorithms

10. Domain Discretization and Mesh Representation  Geometric model discretized into a mesh Discretized geometric domain defined as “union” of the mesh entities Maintain link to shape information Support the field’s distributions used in equation discretization Maintain the dof multiplying the distributions and support assembly of global system  Mesh topology provides flexible support for these functions Mesh entities, M d, of dimension d Mesh entities are constructed from lower dimensional mesh entities e.g. A face is constructed from edges which are constructed from vertices Entities are adjacent if one is used to construct the other Typical complete mesh topological structures:

11. Mesh/Model Relationship  Relationship is termed “classification” Mesh classification: unique association of a mesh entity, M i d i, to a geometric model entity, G j d j, where d i ≤d j is denoted by  M i d i G j d j indicates the left-hand entity (or set) represents a portion of the right-hand entity in the discretization Multiple M i d i classified on a G j d j Boundary mesh entities are identified in terms of their classifications Classification critical to supporting adaptive simulations and high level problem definitions Reverse classification is the set of equal order mesh entities classified on a specific model entity

12. Simulation Fields  The discretization of the tensors used in the PDE in terms of distribution functions times degrees of freedom written over mesh entities  Field definition Tensor qualification - order, symmetries, spatial dimension, coordinate system, constraints Distribution functions - equations, mesh entities they act over, dof used dof - mesh entities they are on, values after evaluation * Typical but not unique form

13. Fields Relationship With Other Simulation Structures  Relationship with the space/time domain Tensor qualifiers specified over the space/time domain Specifies the spatial domain dimension for the tensor Constraints (e.g. boundary conditions, initial conditions, divergence free/solenoidal field, etc.) can be specified with attributes  Relationship with the mesh Information used to discretize the tensor specified over the mesh Distributions -- defined over mesh entities that are the same dimension as the tensor dimension DOFS -- associated with mesh entities that are the dimension of the distribution mesh entities and lower DOF locations on associated entities 5 point FD stencil FV with 1 distribution per element C 0 FEM quadratic distributions

14. Functionality of Field Library Load, save, create and delete functions Point-wise interrogation functions using combinations of operators Tensor operators  Addition  Multiplication  Transpose Differential operators  Identity  Derivatives (x,y,z,t)  Gradient Integral functions Domain defined through TSTT entity sets and/or model entities Uses point-wise interrogation operators for integration of general quantities Mapping fields between meshes Requires mesh information and above operations  Curl  Divergence  Laplacian Locally adapted mesh Meshes sharing a boundary

15. TSTT Interface Implementations Implementations are software libraries that can be used through the TSTT interface Implementations are required to contain a core functionality Includes extensions allowing certain software requirements to be met Multiple implementations can be used simultaneously Tests to verify each implementations compliance with TSTT interface standard  Implementations of the TSTT Mesh Interface currently available at https://svn.scorec.rpi.edu/wsvn/TSTT FMDB - http://www.scorec.rpi.edu/AOMD FronTier - http://frontier.ams.sunysb.edu GRUMMP - http://tetra.mech.ubc.ca/GRUMMP MOAB - http://cubit.sandia.gov/MOAB NWGrid - http://www.emsl.pnl.gov/nwgrid/index_nwgrid.html

16. SCOREC Involvement With TSTT Development of interface standards TSTT Mesh Interface TSTT Geometry Interface TSTT Fields Interface Software development FMDB (Flexible Mesh DataBase) library which is compliant with the TSTT Mesh Interface FANS (Field Approximation for Numerical Simulations) library which is being developed in conjunction with development of the TSTT Fields Interface Model library which is being developed in conjunction with development of the TSTT Geometry Interface Notable uses of software Development of adaptive loops for mesh-based simulations Multiphysics simulations

17. Constructing Adaptive Solution Procedures  Components of an an adaptive mesh control loop High level problem definition (domain and attributes) Domain discretization (mesh generation) Equation discretization Solvers to deal with the large systems of equations Error estimators/indicators Correction indicators Spatial and/or equation discretization enrichment methods  Implementation Tightly coupled using a single set of structures Advantage: Computationally efficient if done well Disadvantage: Complex algorithm and code development Loosely coupled building on existing components Advantage: Ability to use existing analysis codes and adaptive tools and to mix and match tools Disadvantage: Overhead of multiple structures and data conversion

18.  Information needed by an adaptive loop for proper creation of input for a code Overall Domain Definition - Geometric Model Want a “true definition” of the domain - not a specific approximation Physical Attributes Sets of tensors needed to qualify the loads, material properties, boundary conditions and initial conditions Domain Discretizations (meshes) First piece of the two level discretization that is input to analysis codes Simulation Fields Discretizations The discretized versions of tensor variables defined in terms of distributions and degrees of freedom over the meshes as dictated by the PDE discretization process performed within the analysis code  The goal of the TSTT development is to provide all of the above information in a uniform manner Structures/Services for Adaptive Loops

19. Adaptive Loop Driver Code Input and Output  Input Domain definition Mathematical problem attributes Adaptive loop attributes/a priori mesh controls  Adaptive goal  Geometry approximation controls for mesh  Mesh size limits (memory, computation constraints) a posteriori mesh controls Field information Error estimates Convergence rates Adaptive strategies Model modifications through mesh motion  Output Simulation code input Adaptive loop convergence information Mesh

20. Simulation Code Input and Output  Input in form manageable by simulation code Sufficient mesh representation Constructed from complete mesh representation from interface Usually only nodes and elements Boundary and possibly initial condiitons Element “sides” and nodal boundary conditions from classification/reverse classification information Tensor values from field and attributes Other problem attributes From complete problem definition Ideally code input files do not change  Output Computed field information May include error estimates/indicators Field information from visualization output may be sufficient Ideally code output files do not change

21. Adaptive Mesh Control Need adaptive mesh control when Discretization is inadequate - mesh refinement Discretization computationally inefficient - mesh coarsening Element shape becomes unacceptable – mesh improvement Two approaches for h-type mesh adaptation Re-meshing Define new mesh size field Provide mesh generator with domain definition and mesh size field Map any history dependent solution fields from the old to the new mesh Local mesh modifications Determine local operations needed to perform desired modifications Execute the local modifications Apply incremental history dependent solution field updates

22. Mesh Modification Operators  Single step operators Face swap Edge swap Edge collapse Region collapse Edge split Vertex motion Curving interior mesh entity  Compound operators modify two or more entities to allow the modification of the problem entity  Examples of two step compound operators Swap entity A first, and then swap entity B Collapse entity A first, and then collapse entity B Swap entity A first, and then collapse entity B Swap entity A first, and then curve new entity B Swap entity A first, and then reposition vertex B Etc.

23. Moving refinement vertices to boundary required during mesh modification (see IJNME paper, vol58 pp247-276, 2003 ) Coarse initial mesh and the mesh after multiple refinement/coarsening Operations to move refinement vertices Accounting for Curved Domains During Refinement x y z

24. Field Transfer on Adapted Meshes Two approaches Global Computationally expensive Procedures tend to diffuse information - accuracy loss Local Performed with each local mesh modification Limited number of elements involved - efficient No accuracy loss with some operations, others easier to control due to local nature Edge marked for collapse Zone updated by the operation Mesh before collapse Mesh after collapse

25. Adaptive Loop for Accelerator Design  Collaboration with SLAC (I. Malik, K. Ko and Z. Li) Complex CAD geometry Physics modeling by the Omega3P code from SLAC Solves Maxwell’s equations in frequency domain Determine power loss for specific modes High level modeling accuracy needed E.g., 0.1% error in frequency predictions Adaptive mesh control needed to provide desired accuracy Adaptive loop constructed using unmodified Omega3P

26. Providing the Adaptive Loop Components  Components added to construct adaptive loop High level problem definition Automatic mesh generation directly from CAD a-posteriori error indication and convergence rate to define mesh size field a-posteriori error indicator constructed from knowledge of the tensor and its discretized field General mesh modification to construct requested mesh size field accounting for the CAD geometry  Approach to the coupling of the components Interoperability procedures to provide information needed by components Geometry - integration to multiple CAD systems Mesh - integration to various mesh generation and modification procedures Field - integration with different analysis procedures Interoperability procedures used in this example from TSTT SCOREC Simmetrix

27. Integration of Components

28. Using geometry operators means alternate solid modelers can be inserted Using TSTT mesh operators means alternate mesh generators and mesh adaptation procedures can be inserted Using TSTT field operators allows easy construction of alternative error estimators Operator Interfaces Projection-based error estimator used to construct new mesh size field given to mesh modification Mesh adaptation based on local modification linked directly to CAD Unaltered SLAC code Unaltered SLAC code Error estimators from RPI and SLAC

29. Adaptive Results for Trispal Model Frequency Convergence Q Convergence Distribution of Wall-loss Level 0 Level 2 Level 1

30. Metal Forming Simulation Large plastic deformation Meshes often become invalid Evolving geometry Evolving contact Components of automated simulation Commercial analysis engine (DEFORM TM ) Monitoring of mesh discretization errors and element shapes Mesh model topology update Construct mesh size field based on discretization errors and geometry approximation General mesh modification to obtain the desired mesh size field Adjust mesh size and shape to control geometric approximations Local field transfer

31. Model Topology Update Geometric components in forming simulations n Workpiece n Dies Die motions Model topology needs to be updated Contact conditions change as simulation proceeds Mesh updates require complete model topology Simulation engine tracts only nodal contact Must update model topology based on this information n Model update procedure Maintain non-manifold model representation Simulation contact information and “mesh geometry” Uses initial classification to build topology and then corrects ambiguities Mesh classified against updated model topology  Mesh modifications controlled  Attributes properly associated to mesh

32. Steering Link Problem Adapted based on the error indicators on the effective strain

33. Summary of TSTT Work TSTT Mesh 0.6 Interface complete Working document describing the mesh and geometry data models as well as the overall vision Implementations of TSTT Mesh 0.6 Interface available https://svn.scorec.rpi.edu/wsvn/TSTT FMDB - http://www.scorec.rpi.edu/AOMD FronTier - http://frontier.ams.sunysb.edu GRUMMP - http://tetra.mech.ubc.ca/GRUMMP MOAB - http://cubit.sandia.gov/MOAB NWGrid - http://www.emsl.pnl.gov/nwgrid/index_nwgrid.html Preliminary interfaces for geometry, field data, and mesh/geometry data model manager C, C++, and Fortran language interoperability through SIDL/Babel (CCA) Analyzing performance ramifications of SIDL/Babel language interoperability tools (joint with the CCA) TSTT Information http://www.tstt-scidac.org

34. Adaptive loops built from available components Software for multiple sources combined to create functional adaptive loops that provide superior results Initial SLAC adaptive loop constructed in less than 3 months  By people with little background with components  Included development of an error estimation and correction indication routines Approach taken focused on optimizing time to getting adaptive results (not the computing resources needed for the simulation) Did not alter Omega3P or DEFORM codes  Used meshing components and operators to create input files  Used field operators to extract results from output file Closing Remarks on Adaptive Loops