1. PETSc Portable, Extensible Toolkit for Scientific computing

2. Overview Parallel & Sequential Object-oriented Available for virtually all UNIX platforms, as well as Windows 95/NT Flexible: many options for solver algorithms and parameters

3. Motivation Developing parallel, non-trivial PDE solvers that deliver high performance is still difficult and requires months (or even years) of concentrated effort. PETSc is a toolkit that can ease these difficulties and reduce the development time

4. Introduction A freely available and supported research code Available via http://www.mcs.anl.gov/petsc Free for everyone, including industrial users Hyperlinked documentation and manual pages for all routines. Many tutorial-style examples Support via email: petsc-maint@mcs.anl.gov Usable from Fortran 77/90, C, and C++ Portable to any parallel system supporting MPI, including Tightly coupled systems Cray T3E, SGI Origin, IBM SP, HP 9000, Sun Enterprise Loosely coupled systems, e.g., networks of workstations Compaq, HP, IBM, SGI, Sun, PCs running Linux or Windows

5. History PETSc history Begun in September 1991 Keep on development Now: over 8,500 downloads since 1995 (versions 2.0 and 2.1)

6. PETSc Concepts How to specify the mathematics of the problem Data objects - vectors, matrices How to solve the problem Solvers - linear, nonlinear, and time stepping (ODE) solvers Parallel computing complications Parallel data layout - structured and unstructured meshes

7. Structure of PETSc Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK Profiling Interface PETSc PDE Application Codes Object-Oriented Matrices, Vectors, Indices Grid Management Linear Solvers Preconditioners + Krylov Methods Nonlinear Solvers, Unconstrained Minimization ODE Integrators Visualization Interface

8. PETSc Numerical Components Compressed Sparse Row (AIJ) Blocked Compressed Sparse Row (BAIJ) Block Diagonal (BDIAG) DenseOther Line SearchTrust Region Newton-based Methods Other Nonlinear Solvers Additive Schwartz Block Jacobi ILUICC LU (Sequential only) Others Preconditioners Euler Backward Euler Pseudo Time Stepping Other Time Steppers(For ODE) GMRESCGCGSBi-CG-STABTFQMRRichardsonChebychevOther Krylov Subspace Methods Matrices Matrix-free

9. What is not in PETSc? Discretizations Unstructured mesh generation and refinement tools Load balancing tools Sophisticated visualization capabilities

10. BUT! PETSc has interface to external software that provides some of this functionality Linear solvers ( AMG BlockSolve95 LUSOL SPAI SuperLU) Optimization software (TAO, Veltisto) Mesh and discretization tools (overture, SAMRAI) ODE solvers (PVODE) Others (Matlab, ParMETIS)

11. Flow of Control for PDE Solution PETSc codeUser code Application Initialization Function Evaluation Jacobian Evaluation Post- Processing PCKSP PETSc Main Routine Linear Solvers (SLES) Nonlinear Solvers (SNES) Timestepping Solvers (TS)

12. A simple PETSc program #include “petsc.h” int main( int argc, char *argv[] ) { int rank; PetscInitialize(&argc,&argv,PETSC_NULL,PETSC_NULL); MPI_Comm_rank(PETSC_COMM_WORLD,&rank ); PetscSynchronizedPrintf(PETSC_COMM_WORLD, “Hello World from %d\n”,rank); PetscFinalize(); return 0; }

13. Sparse Matrix Computation in PETSc Variety Data Structures Variety Preconditioners Variety Iterative Solvers Variety interface to External Softwares

14. Data Structure for sparse matrix Compressed Sparse Row Blocked compressed sparse Row Block Diagonal

15. Example(BCRS) Block Compressed Sparse Row NR: Number of blocks per side of the matrix NNZB: Number of Non-Zero Blocks NB: Number of elements per side of the Block N: Number of elements per side of the matrix

16. Preconditioners Incomplete LU factorization Jacobi, Gauss-Seidel,SOR Schwarz; within blocks use LU, ILU, SOR, etc.

17. Solvers Direct method (LU) Krylov method (CG, GMRES, BiCGstab, CGS, QMR, …) Non-Krylov iterative method (Jacobi, G-S, SOR) Pre-conditioned Krylov method

18. Interface to external solver Both Iterative and Director solvers AMG BlockSolve95 LUSOL SPAI SuperLU

19. Functions Define the linear system (Ax=b) MatCreate(), MatSetValue(), VecCreate() Create the Solver SLESCreate(), SLESSetOperators() Solve System of Equations SLESSolve() Clean up SLESDestroy()

20. Example Solve Ax = b www.cs.ucsb.edu/~jh/example.cc

21. ComparisonI: Iterative methods provided for different packages

22. ComparisonII: Preconditioners provided by different packages

23. ComparisonIII: Performance Test Matrix:

24. ComparisonIII: Performance Feed these matrices to Aztec and PETSc Generate a zero vector as initial guess and a vector of 1 as RHS Use GMRES without preconditoners, max iteration = 500 Run on 512-processor Cray T3E900

25. ComparisonIII: Performance

27. Aztect suffers from setup varization PETSc did not optimize partition algorithm( PE256 -> PE512, performance decrease). Communication takes more time than computation

28. Conclusion PETSc - Well designed and widely used - First MPI-based program to public - Good set of iterative methods and preconditioners - Good Support and excellent technical documents - Still under developement

29. Thanks Comments ?