1. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. Eric Phipps and Andy Salinger Applied Computational Methods Department Sandia National Laboratories 9 th Copper Mountain Conference on Iterative Methods April 7, 2006 Solving Bordered Systems of Linear Equations for Large-Scale Continuation and Bifurcation Analysis

2. Why Do We Need Stability Analysis Algorithms for Large-Scale Applications? Nonlinear systems exhibit instabilities, e.g.: Buckling Ignition Onset of Oscillations Phase Transitions These phenomena must be understood in order to perform computational design and optimization. Established stability/bifurcation analysis libraries exist: AUTO (Doedel, et al) CONTENT (Kuznetsov, et al) MATCONT (Govaerts, et al) etc… We need algorithms, software, and experience to impact ASC- and SciDAC-sized applications (millions of unknowns) Stability/bifurcation analysis provides qualitative information about time evolution of nonlinear systems by computing families of steady-state solutions. LOCA: Library of Continuation Algorithms

3. LOCA provides: Parameter Continuation: Tracks a family of steady state solutions with parameter Linear Stability Analysis: Calculates leading eigenvalues via Anasazi (Thornquist, Lehoucq) Bifurcation Tracking: Locates neutral stability point (x,p) and tracks as a function of a second parameter Application code provides: Nonlinear steady-state residual and Jacobian fill: Newton-like linear solves: External force Second parameter External force 1 1 3

4. LOCA Designed for Easy Linking to Existing Newton-based Applications Algorithmic choices for LOCA: Must work with iterative (approximate) linear solvers on distributed memory machines Non-Invasive Implementation (e.g. matrix blind) Should avoid or limit:  Requiring more derivatives  Changing sparsity pattern of matrix  Increasing memory requirements LOCA targets existing codes that are: Steady-State, Nonlinear Newton’s Method Large-Scale, Parallel

5. Bordering Algorithms Meet These Requirements … but 4 solves of J per Newton iteration are used to drive J singular! Turning Point Bifurcation Full Newton Algorithm Bordering Algorithm

6. These Techniques Have Been Applied To Many Interesting Problems Capillary CondensationFlow in CVD ReactorYeast Cell-Cycle Control Buckling of Garden HoseBlock Copolymer Self-AssemblyPropane&Propylene Combustion

7. LOCA has been rewritten as part of the Trilinos framework Better algorithms can be implemented with tighter coupling to linear algebra LOCA can build upon the existing interface between application codes and the NOX nonlinear solver Sandia’s parallel iterative solvers are made available to LOCA users Collection of parallel solver packages, with a common build process, made interoperable where appropriate (Heroux et al) Trilinos 6.0 Released: September 1, 2005 2004 HPC Software Challenge

8. Bordered Systems of Equations Only requires solves of J but Requires m+l linear solves Has difficulty when J is nearly singular Solving bordered systems of equations is a ubiquitous computation: Pseudo-Arclength Continuation Constraint Following Turning Point Identification Bordering Algorithm

9. Solving Bordered Systems via QR 1 H.F Walker, SIAM J. Sci. Comput., 1999 2 R. Schreiber, SIAM J. Stat. Comput., 1989 Extension of Householder pseudo-arclength technique by Homer Walker 1 QR Factorization Compact WY Representation 2 where Rearranged Bordered System Write then P is nxn, nonsingular, rank m update to J

10. Snap-through Buckling of a Shallow Cap (66K tri-shell elements, 200K unknowns, 16 procs) Salinas (Reese et al., SNL): Unstructured finite element, linear elasticity Corotational formulation for beams and shells (C. Felippa, CU-Boulder) Updated Lagrangian for solid elements Analytic, Sparse Jacobian Fully Coupled Newton Method (NOX) GMRES (Aztec) with RILU(k) Preconditioner (Ifpack) Distributed Memory Parallelism

11. Turning Point Identification Moore-Spence Formulation … but 4 solves of J per Newton iteration are used to drive J singular! Turning Point Bifurcation Full Newton Algorithm Bordering Algorithm

12. Snap-through Buckling of a Shallow Cap Turning Point Bordering Method RILU fill factor: 6 RILU overlap: 6 Krylov space: 2000

13. Solve 5 bordered systems of equations using QR approach Then Modified Turning Point Bordering Algorithm

14. Snap-through Buckling of a Shallow Cap Modified Turning Point Bordering Method RILU fill factor: 6 RILU overlap: 6 Krylov space: 2000

15. Given and, let then There are constants such that Standard formulation: Note for Newton’s method: 3 linear solves per Newton iteration (5 for modified bordering)! For symmetric problems reduces to 2 solves. Minimally Augmented Turning Point Formulation

16. With iterative solvers, so define where Also Minimally Augmented Turning Point Formulation for Large-Scale Problems

17. Snap-through Buckling of a Shallow Cap Minimally Augmented Formulation RILU fill factor: 6 RILU overlap: 6 Krylov space: 2000 Modified bordering Minimally augmented

18. Pseudo-Arclength Continuation of Turning Points Pseudo-Arclength Equations Minimally Augmented: 3 total linear solves Moore-Spence w/Bordering: 7 total linear solves Newton Solve

19. Snap-through Buckling of a Symmetric Cap Pseudo-arclength Turning Point Continuation MethodContinuation Steps Failed Steps Nonlinear Iterations Linear Solves Linear Iterations Total Time (hrs) Moore-Spence Mod. Bordering 131544271177974.2 Min. Augmented12049122218881.2

20. Summary QR approach provides a convenient way to solve bordered systems –Nonsingular –Only involves one linear solve –Only requires simple vector operations –Doesn’t change dimension of the linear system –Become a workhorse tool in LOCA Highly encouraged by minimally augmented turning point formulation –No singular matrix solves –Improves robustness, scalability and accuracy –Requires linear algebra-specific implementation Future work –Minimally augmented pitchfork, Hopf bifurcations –Preconditioners for

21. Points of Contact LOCA: Trilinos continuation and bifurcation package –Sub-package of NOX –Andy Salinger (agsalin@sandia.gov)agsalin@sandia.gov –Eric Phipps (etphipp@sandia.gov)etphipp@sandia.gov –www.software.sandia.gov/noxwww.software.sandia.gov/nox NOX: Trilinos nonlinear solver package –Roger Pawlowski (rppawlo@sandia.gov)rppawlo@sandia.gov –Tammy Kolda (tgkolda@sandia.gov)tgkolda@sandia.gov –www.software.sandia.gov/noxwww.software.sandia.gov/nox Trilinos: Collection of large-scale linear/nonlinear solvers –Epetra, Ifpack, AztecOO, ML, NOX, LOCA, … –Mike Heroux (maherou@sandia.gov)maherou@sandia.gov –www.software.sandia.gov/Trilinoswww.software.sandia.gov/Trilinos Trilinos Release 6.0 currently available, 7.0 this fall –7.0 will include all LOCA algorithms presented here Salinas: Massively Parallel Structural Dynamics Finite Element Code –Garth Reese (gmreese@sandia.gov)gmreese@sandia.gov

22. With iterative solvers, solves for u and v are in-exact. Does this impact nonlinear convergence? Instead solve for updates to u and v every nonlinear iteration: Modified Minimally Augmented Turning Point Formulation for Large-Scale Problems

23. Snap-through Buckling of a Shallow Cap Modified Minimally Augmented Formulation RILU fill factor: 6 RILU overlap: 6 Krylov space: 2000

24. RILU fill factor: 6 RILU overlap: 6 Krylov space: 2000 Bordering Minimally augmented Snap-through Buckling of a Shallow Cap Minimally Augmented Formulation