1. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 1 Barry Smith Argonne National Laboratory DOE 2000 Numerics Capability http://acts.nersc.gov

2. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 2 Targets l Large Scale (100,000,000 dof) simulations l computational fluid dynamics l combustion l structures l materials l usually PDE based l Large scale (1000 cv) optimization l often involving simulation l maybe stochastic

3. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 3 Organization of Presentation l Sample computations (to give an idea of scale) l Background on computational infrastructure l Summary of current toolkits l Efforts on toolkit interoperability

4. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 4 Oil Reservoir Simulation Multiphase Flow l Three-dimensional EOS model with 3 components (8 DOF per cell) l Structured Cartesian grid l Over 4 million cell blocks, 32 million DOF l Fully implicit, time-dependent l Over 10.6 gigaflops sustained performance on 128 nodes of an IBM SP l Entire simulation runs in less then 30 minutes

5. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 5 Ocean Circulation Model l Uses second-order finite differences with an explicit leapfrog time stepping scheme on a logically rectangular grid. A. Wallcraft, H. Hurlburt, R. Rhodes, & J. Shriver, NRL - Stennis l 540% speedup over the previous parallel software (64*145.6 Mflop/s) = 9.3 Gflop/s l 2 hours is now 25 minutes

6. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 6 CFD on an Unstructured Grid l Three-dimensional incompressible Euler l Tetrahedral grid l Up to 11.0 million degrees of freedom l Based on a legacy NASA code l Now runs several hundred times faster than legacy code

7. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 7 Unstructured Computational Grid

8. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 8 11,047,096 Unknowns Floating-Point Performance

9. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 9 DOE Hardware Infrastruture l IBM SP l Currently up to 512 nodes with 1-4 way SMP nodes l MPI machine l Processor peak ~ 640 MFlops l SGI Origin l Currently up to 128 processors in a box l cc-NUMA - cache coherent - non-uniform memory access l Boxes connected via message passing (MPI) l Processor Peak ~ 500 MFlops l Clusters (including ASCI Red, Linux, and DEC alphas) l 7000+ processors l MPI based

10. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 10 DOE Numerics Software Infrastruture l MPI basic model for parallelism l Toolkits (libraries) l in F77/F90, C, C++ l sometimes object oriented or at least object based l hide much of the MPI details from application codes l attempt to support use from multiple languages l Applications l F77/F90, C, C++ l some use of scripting languages (Python)

11. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 11 Toolkit Categories l PDE/Grid management systems l ODE integrators l Nonlinear solvers l Eigenvalue computations l Linear solvers l Nonlinear optimization

12. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 12 PDE/Grid management systems l Overture l overlapping composite grids l complex geometry l POOMA (Parallel Object-oriented Methods and Applications) l Data parallel style support for finite difference particle methods

13. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 13 Sample Overture Grid

14. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 14 Sample Overture Grid

15. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 15 ODE Integrators l PVODE (Parallel variable order ODE integrators) l can use linear solvers from other ACTS toolkits l BDF (backward differentiation formula) multi-step methods for stiff problems l Adams multi-step method for non-stiff l also differential-algebraic systems

16. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 16 Nonlinear Solvers l PETSc (Portable Extensible Toolkit for Scientific computation) l Kinsol (part of PVODE, may also be used independently) l Newton-based solvers l Globalization strategies include line search trust region pseudo-transient continuation (false time-stepping) l Will support several toolkits’ linear solvers

17. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 17 Linear Solvers l Aztec l General purpose iterative solver l Hypre l Family of application-centric (physics based) preconditioners l PETSc l Extensible family of general purpose iterative solvers l QMRPack l Full implementation of QMR l ScaLAPACK l Direct solvers

18. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 18 Aztec l General purpose parallel iterative solver l Supports point and block storage of sparse matrices l Includes 5 Krylov solvers l Includes polynomial and overlapping additive Schwarz preconditioning l Has scaled successfully to 1000s of processors (ASCI Red) on some applications

19. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 19 Hypre l Family of application-centric (physics or grid based) preconditioners Data Layout structuredcompositeblock-struc unstrucCSR Conceptual Interfaces Algorithms GMG,...FAC,...Hybrid,... AMGe,...ILU,...

20. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 20 PETSc l Over 10 Krylov subspace methods including l GMRES, TFQMR, CG, Bi-CG-stab l A suite of preconditioners including l additive overlapping Schwarz l access to parallel ILU(0) and ICC(0) via BlockSolve95 l 5 matrix storage formats provided l Many additional tools, such as vector scatter operations and mapping operations to simplify the coding of parallel PDE applications l Modular and object oriented

21. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 21 QMRPack l Implements several variants of the quasi-minimal residual (QMR) algorithm for the solution of general,non-Hermitian linear systems l Only implementation that includes the full suite of look- ahead techniques l Includes an eigensolver l Now interfacing with other ACTS toolkits

22. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 22 ScaLAPACK l SuperLU l Direct solution of large, sparse nonsymmetric systems of linear equations. l Runs on shared memory, MPI version expected soon l Attained 8.3 Gflops on a 52,000 unknown linear system arising from device simulation. 512 processors on a Cray T3E. 014% to 2.4% matrix density. l MPI and PVM based direct dense solvers l Based on BLACS (basic linear algebra communication subroutines), BLAS and LAPACK l Well known, trusted community standard

23. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 23 Eigenvalue Computations l ScaLAPACK l Dense matrix eigensolvers l Based on BLACS (basic linear algebra communication subroutines), BLAS and LAPACK l Well known, trusted community standard l Sustained 605 Gflops on ASCI Red in DOE materials simulations.

24. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 24 Nonlinear Optimization l Opt++ l Allows for rapid prototyping of new optimization methods l Abstract base classes for nonlinear optimization problem: — NLP0: no analytic derivatives — NLP1: analytic first derivatives (gradient) — NLP2: analytic first and second derivatives (gradient, Hessian) l Gradient methods for nonlinear least-squares problems l available in the NEWTON-LIKE optimization class: finite-difference Newton quasi-Newton Gauss-Newton l Derived nonlinear problem classes: FDNLF1: generate numerical-difference gradients NLF2: first and second derivatives supplied

25. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 25 DOE “Standards” Collaborators l CCA - Common Component Architecture group: arising out of DOE2000 activities to develop “standard” ways of managing numerical components so that ones developed by different groups may be mixed-and-matched l ESI - Equation Solver Interface Forum: group arising out of the DOE labs to develop common interfaces for scalable linear solvers

26. ANS 1998 Winter Meeting DOE 2000 Numerics Capabilities 26 DOE 2000 Numerics Capability We provide a variety of powerful toolkits to assist in the development of scalable simulation codes http://acts.nersc.gov