1. August 4, 2007, Deflation Methods in Fermion Inverters, 1 Methods for Fermion Inverters Walter Wilcox Baylor University Joint with Ron Morgan (Mathematics Dept.) and Abdou Abdel-Rehim (Baylor Postdoctoral Fellow)

2. August 4, 2007, Deflation Methods in Fermion Inverters, 2 Outline Deflation basics Morgan/Wilcox algorithm (non-Hermitian) GMRES-DR (Morgan) GMRES-Proj (multiple rhs’s of Ax=b) shifting (multi-mass) new results (rhs’s with D-BiCGStab) M. Lüscher algorithm A. Stathopoulos/K. Orginos algorithm (Hermitian) (arXiv: 0706.2298v4) (arXiv: 0707.0131v1; talk) (math-ph/0405053, arXiv: 0707.0505, 0707.0502)

3. August 4, 2007, Deflation Methods in Fermion Inverters, 3 Deflation or related in lattice QCD problems (not comprehensive!) de Forcrand, Nucl. Phys. B (P.S.) 47, p.228,1996. (experiments with multiple rhs’s.) Edwards, Heller, Narayanan, Nucl. Phys. B 540; Dong, Lee, Liu, Zhang, Phys. Rev. Lett., 85, 5051, 2000 457, 1999. (projecting out low overlap H 2 eigenmodes) Neff, Eicker, Lippert, Negele, Schilling, Phys. Rev. D64:114509, 2001. (deflation for all-to-all propagators on  5 M) Giusti, Hoelbling, Lüscher and Wittig, Comput. Phys. Commun. 153, 31, 2003. (quark propagator low-mode preconditioning) DeGrand and Schaefer, Comput. Phys. Commun. 185- 191, 2004. (low mode averaging)

4. August 4, 2007, Deflation Methods in Fermion Inverters, 4 Related talks at this conference J. Bloch, “An Iterative Method to Compute the Overlap Dirac Operator at Nonzero Chemical Potential”. (Bloch, Frommer, Lang, Wettig, arXiv: 0704.3486) M. Clark, “Adaptive Multi-Grid for QCD” (Brannick et al, arXiv: 0707.4018.) K. Orginos, “A Solver for Multiple Right-Hand Sides” (0707.0131v1)

5. August 4, 2007, Deflation Methods in Fermion Inverters, 5 Krylov subspace: Starting, residual vectors: q is poly of degree m or less that has value 1 at 0. Deflation basics

6. August 4, 2007, Deflation Methods in Fermion Inverters, 6 Matrix: bidiagonal, diagonal is 0.1, 1, 2, 3, …1999, superdiagonal is all 1’s GMRES polynomial of degree 10

7. August 4, 2007, Deflation Methods in Fermion Inverters, 7 GMRES polynomial of degree 100 (close up view)

8. August 4, 2007, Deflation Methods in Fermion Inverters, 8 GMRES polynomial of degree 150 (close up view)

9. August 4, 2007, Deflation Methods in Fermion Inverters, 9 Residual norm curve Matrix vector products

10. August 4, 2007, Deflation Methods in Fermion Inverters, 10 Small eigenvalues and Krylov methods SPD matrix: convergence is approximately related to Example: Eigenvalues: 0.1, 1, 2, 3,... n If one removes 4 eigenvalues, convergence is improved by factor of about 6 (remove 10 for improvement factor of 10). Non-restarted methods like CG and BiCGStab naturally remove some eigenvalues as the iteration proceeds – leads to superlinear convergence. Problem: Restarted GMRES often cannot “remove” eigenvalues.

11. August 4, 2007, Deflation Methods in Fermion Inverters, 11 Solution: Add approximate eigenvectors to the subspace. Morgan, 1995 (GMRES-IR), 2002 (GMRES-DR) Subspace: Eigenvector portion + Krylov portion y i ’s are chosen to be harmonic Ritz vectors.

12. August 4, 2007, Deflation Methods in Fermion Inverters, 12 GMRES-DR(m,k): Solves linear equations and compute eigenvalues simultaneously. Add approximate eigenvectors to the Krylov subspace for the linear equations which essentially removes the corresponding eigenvalues and can thus improve convergence.

13. August 4, 2007, Deflation Methods in Fermion Inverters, 13 GMRES-DR vs. other methods Matrix: bidiagonal

14. August 4, 2007, Deflation Methods in Fermion Inverters, 14 Aspects of GMRES-DR GMRES-DR creates an Arnoldi-like recurrence, where V k is the n by k matrix with columns spanning the approximate eigenvectors, and H k (bar) is small, k+1 by k. Have both approximate eigenvectors and their products with A in compact storage.

15. August 4, 2007, Deflation Methods in Fermion Inverters, 15 For multiple right-hand sides Solve first right-hand side with GMRES- DR Use the computed eigenvectors for the other right-hand sides. Method is GMRES-Proj: Alternate: 1) projection over eigenvectors 2) cycles of regular GMRES

16. August 4, 2007, Deflation Methods in Fermion Inverters, 16 GMRES-Proj for the 2nd rhs (following GMRES-DR for 1st rhs) Matrix: bidiagonal

17. August 4, 2007, Deflation Methods in Fermion Inverters, 17 Wilson 20 3 x 32 at  cr

18. August 4, 2007, Deflation Methods in Fermion Inverters, 18 Twisted Mass 20 3 x 32 at  cr

19. August 4, 2007, Deflation Methods in Fermion Inverters, 19 Krylov subspaces are shift invariant in that A-   I generates the same Krylov subspace no matter what the shift. So the goal is to solve all shifted systems with ONE Krylov subspace. For non-restarted methods this has been done, for example; QMR (Freund) and BiCGStab (Frommer). Multi-masses or multiple shifts

20. August 4, 2007, Deflation Methods in Fermion Inverters, 20 Restarted methods with multiple shifts Restarting makes it more difficult because the shifted residuals are not parallel to one another, generating different Krylov subspaces. Frommer and Glassner (restarted GMRES): Force residuals to all be parallel after a restart. Can continue using one Krylov subspace for all shifted systems Minimal residual property is maintained only for the base shift system.

21. August 4, 2007, Deflation Methods in Fermion Inverters, 21 GMRES-DR for multiple shifts Subspaces generated by GMRES-DR are combination of approximate eigenvectors portion and Krylov subspace portion, but remarkably when put together, they are Krylov themselves (with a different starting vector). So, GMRES-DR can be restarted like GMRES for multiple shifts.

22. August 4, 2007, Deflation Methods in Fermion Inverters, 22 Multiple right-hand sides with multiple shifts and deflation Deflating eigenvalues is difficult for multiple shifts because one can not keep residual vectors parallel unless one has exact eigenvectors. Solution: force error to be in the direction of one vector,namely v k+1 from Then can correct error at the end. Need solution of one extra right-hand side.

23. August 4, 2007, Deflation Methods in Fermion Inverters, 23 Solution of ten right hand sides. Matrix: bidiagonal Blue: base system (sigma=0) Red: shifted system (sigma=-2) Green: base (uncorrected)

24. August 4, 2007, Deflation Methods in Fermion Inverters, 24 => Deflated BiCGStab (D-BiCGStab) What if you don’t like restarting, but still want to solve multiple right-hand sides? Problem: Projection over eigenvectors is not good enough to last for the entire run of BiCGStab. Solution: Use a projection over both right and left eigenvectors. Deflated BiCGStab for the second and subsequent right- hand sides: 1) Project over right and left eigenvectors 2) Run BiCGStab

25. August 4, 2007, Deflation Methods in Fermion Inverters, 25 Deflated BiCGStab for the 2nd rhs with Left-Right Projection Matrix: bidiagonal

26. August 4, 2007, Deflation Methods in Fermion Inverters, 26 20 3 x32 Wilson at  cr

27. August 4, 2007, Deflation Methods in Fermion Inverters, 27 20 3 x32 Wilson at  cr : number of eigenvectors Speedup at  cr on 20 3 x32: BiCG/D-BiCG ~ 5 Speedup at  cr on 16 4 : ~ 2.7

28. August 4, 2007, Deflation Methods in Fermion Inverters, 28

29. August 4, 2007, Deflation Methods in Fermion Inverters, 29 Some questions about Wilson matrix computations How does the optimal number of eigenvalues depend on the size of the problem? Test: all at  cr. Answer: it increases, but not nearly proportional to n (fortunately). Proj/D-BiCG n=24,576 (8 4 )k~10 n=49,152 (8 3 x16)k~15 n=98,304 (8 3 x32)k~20 n=393,216 (16 4 )k~20 n=1,536,000(20 3 x32)k~40

30. August 4, 2007, Deflation Methods in Fermion Inverters, 30 Some questions about Wilson matrix computations After eigenvalues are deflated, how does the number of iterations vary with the size of the problem? (  cr.) Answer: it increases, but not nearly proportional to n (fortunately). ProjD-BiCG n=24,576 (8 4 ) iters ~ 100 n=49,152 (8 3 x16) iters ~ 125 n=98,304 (8 3 x32) iters ~ 140 n=393,216 (16 4 ) iters ~ 300300 n=1,536,000 (20 3 x32) iters ~ 600500 ~ 45% increase in iters with volume change of 3.

31. August 4, 2007, Deflation Methods in Fermion Inverters, 31 Hermitian Systems Lüscher’s domain decomposition deflation algorithm. Stathopoulos/Orginos multiple right-hand side deflation algorithm. Both are tested for dynamical Wilson/clover fermions, using M H M.

32. August 4, 2007, Deflation Methods in Fermion Inverters, 32 Lüscher algorithm Breaks problem up into a deflation subspace, S, defined on 4 4 blocks and the orthogonal complement,. Uses projectors P L and/or P R. Basic deflated system: where After SAP preconditioning:

33. August 4, 2007, Deflation Methods in Fermion Inverters, 33 More algorithm details… Outer part - Uses SAP (Schwartz Alternating Procedure) as a right preconditioner on 4 3 x8 blocks and the GCR (Generalized Conjugate Residual - consistent with SAP) algorithm for the Krylov inverter on the global (orthogonal) space. (See Lüscher’s PoS LAT2005:002, 2006 talk based on earlier work.) Preconditioning reduces the iteration count of GCR and deflation overhead. Inner part (“little Dirac”) - This system is fairly large. It is even-odd preconditioned and “global mode” deflated. Also uses GCR here. See Frank, Vuik, SIAM. J. Sci. Comput. 23, 442, 2001 Nabben, Vuik, SIAM. J. Sci. Comput. 27, 1742, 2006 for mathematical background of domain decomposition and preconditioning.

34. August 4, 2007, Deflation Methods in Fermion Inverters, 34 Even more algorithm details… Initial eigenvalue computation: Prepares deflated space for later use. Applies inverse iteration (SAP) to N s (=20) random global vectors to get low modes, which are then projected onto domains. “Little Dirac” subspace (=N s N d ): 20 x 2592 or 20 x 8192 (small vectors). Total overhead: 150s for 24 3 x48 lattice and 184s for 32 3 x64 lattice (tuned to m cr ). Solves the M H M matrix on the full system knocking out the deflated eigenvectors with P L projector while again using GCR algorithm. Done for one mass at a time. Deflation on domains works because of “local coherence”. Only a small number of global vectors projected onto the blocks are needed to project out low modes.

35. August 4, 2007, Deflation Methods in Fermion Inverters, 35 Analogy Domain decomposition deflation achieved by using low modes which are smooth but far from being approximate eigenmodes => “local coherence”. Tested on 2 flavor Wilson/clover configs (50).

36. August 4, 2007, Deflation Methods in Fermion Inverters, 36 32 3 x64 lattice solver times Peak speedup (BiCG/DFL): 366/32=11.4 Integrated speedup (BiCG/DFL): 966/314=3.1 (5 masses) ~ 13% outer (?% inner) increase in iters with a volume change of 3.

37. August 4, 2007, Deflation Methods in Fermion Inverters, 37 Stathopoulos/Orginos algorithm eig-CG(Nev,m) like GMRES-DR, solves linear equations and does simultaneous improvements of the deflated eigenvectors. The eigenvector part is restarted, which, however, does not affect the solution of the CG linear equations. incremental eig-CG(s) calls eig-CG, and adds Nev new eigenvectors to a separate subspace after each rhs, and does orthogonalization. It is used for the first s s 1 rhs’s. init-CG uses the final information generated by incremental eig-CG. Accuracy is the key!

38. August 4, 2007, Deflation Methods in Fermion Inverters, 38 More technical page eig-CG(Nev,m) has a restarted subspace of maximum dimension m. (Made up of Nev previous eigenvectors, Nev current eigenvectors and (m-2*Nev) Krylov vectors). Uses Rayleigh-Ritz to compute eigenvectors and appends portions of the CG search space (Krylov part) to the eigenvectors. Typically, however, the linear equations converge faster than the eigenvalue part. Incremental eig-CG(s) (s = 2,…) accepts (s-1)*Nev eigenvectors, calls eig-CG for s s 1 rhs’s, and accumulates another Nev approximate Ritz vectors from each new right hand side. Needs significant storage. init-CG does a standard Galerkin projection on the initial solution vector. A single restart is done. Tested on “several” anisotropic, 2 flavor Wilson fermion gauge fields. Uses single precision, except on dot products. m=100, Nev=10 for 48 total rhs’s (s 1 =24).

39. August 4, 2007, Deflation Methods in Fermion Inverters, 39 Convergence of deflated eigenvalues Point: converges as fast as if they weren’t restarting.

40. August 4, 2007, Deflation Methods in Fermion Inverters, 40 Incremental RHS solver history Spike on last 24 caused by a restart necessary because of eigenvector accuracy.

41. August 4, 2007, Deflation Methods in Fermion Inverters, 41 Solver performance vs. quark mass Last 24 right-hand sides only compared to non-deflated CG. Peak speedup ~10 on smaller lattice near m cr. Integrated speedup ~6 (all rhs’s.) Peak speedup ~ 6.9 on larger lattice near m cr. ~ 190% increase in iters with a volume change of 3.

42. August 4, 2007, Deflation Methods in Fermion Inverters, 42 Summaries… For Hermitian systems (M H M), the Stathopoulos/Orginos algorithm is effective for a sufficiently large number of rhs’s. Uses many eigenvectors, but no spectral preconditioners. Uses eigenvectors on starts (and a single init-CG restart). Krylov/RayleighRitz based. Needs accurate eigenvectors which improve over additional rhs’s. Like GMRES-DR solves linear equations at the same time as computing eigenvectors. “Large” V 2 problem.

43. August 4, 2007, Deflation Methods in Fermion Inverters, 43 Lüscher’s algorithm, built within his SAP+GCR inverter, applied to M H M, works well for QCD and also defeats critical slow down. There is a overhead in compute time for subspace generation, but gets amortized over many rhs’s or masses. Uses many inexact eigenvectors and makes extensive use of spectral preconditioners. DD+Krylov+preconditioning. Uses eigenvectors at every iteration, but very small number of iterations. Deflation on domains is a new idea. “Small” V 2 problem.

44. August 4, 2007, Deflation Methods in Fermion Inverters, 44 Deflated GMRES is also Krylov/RayleighRitz based. Useful for multiple rhs’s as well as shifting. D-BiCGStab can be used for multiple rhs’s also. We would do Wilson/clover without the M H M step plus shifting for various masses. We don’t need spectral preconditioners for GMRES-Proj or D- BiCGStab; we use a modest number of fairly accurate eigenvectors, which are used at restarts or a single time for D-BiCGStab (better eigenvector accuracy needed for D- BiCGStab than Proj.). “Mild” V 2 problem. (Caveat: Our lattices are 8X smaller than Lüscher’s.)

45. August 4, 2007, Deflation Methods in Fermion Inverters, 45 is a breakthrough method for lattice QCD!

46. August 4, 2007, Deflation Methods in Fermion Inverters, 46 Serial Multi-Mass Because of the twisted mass, , it is not possible to apply multi-mass solvers to twisted mass problems simultaneously with even-odd preconditioning. We can accelerate the convergence of twisted-mass problems with multiple masses and even-odd preconditioning. The method is based on solving the systems serially but using an improved initial guess by making a minimal residual projection over available solutions of the previous systems. Improves as the number of solved systems increases.

47. August 4, 2007, Deflation Methods in Fermion Inverters, 47 Mass number kappa  X 0 =0With projection High -> low With projection Low -> high 10.1572900.00512708201270 20.1572500.00712106401030 30.1572100.0091150580940 40.1571700.0111090520760 50.1571300.0131030460610 60.1570900.015940400490 70.1570500.017880370 80.1570100.019850430310 90.1569700.021790520280 100.1569300.023760610340 110.1568900.025730 190 Total MVP10,7006,0806,590 Using Serial multi-mass with Twisted Mass Fermions