1. Simulation with 2+1 flavors of dynamical overlap fermions
Hideo Matsufuru for JLQCD and TWQCD Collaboration
Thanks.
Good morning.
I would like to thank the organizers for giving me this opportunity, to report the status of our project.
This is the plan of my talk.
We are now performing large-scale simulations with dynamical overlap fermions.
In this talk, I would like to explain our strategy, simulation algorithms, and show you some of recent results.
High Energy Accerelator Research Organization (KEK)

2. Collaboration JLQCD Collaboration
S.Aoki (Tsukuba), S.Hashimoto (KEK), T.Kaneko (KEK), J.Noaki (KEK), T.Onogi (YITP, Kyoto), E.Shintani (KEK), N.Yamada (KEK), H.M. (KEK)
TWQCD Collaboration
T-W.Chiu (Taiwan), X, T-H.Hsieh (Academia Sinica), K. Ogawa (Taiwan)
Other presentations:
Hashimoto (plenary, Wed)
Noaki (spectrum, Tue)
Chiu (topological susceptibility, Tue)
Kaneko (Pion form factors, Thu)
Ohki (Nucleon sigma term, Thu)
Shintani (strong coupling const, Fri)
Yamda (S-parameter, ps-NG boson mass, Fri)
This is current list of members.
We are a collaboration of collaborations.

3. Project Dynamical simulations with exact chiral symmetry
2-flavor runs on a 163x32 lattice (a=0.12fm)
Fixed topological charge (mainly Q=0)
6 sea quark masses (mq~(1/6−1) ms)
Finished to accumulate 10,000 trj
many physical calculations
2+1-flavor runs on 163x48 lattice (a=0.11fm)
5 mud (~(1/6−1) msphys), 2 ms (~msphys)
2,500 trj accumulated
epsilon regime: Nf=2 finished, 2+1 in progress
2+1-flavor runs on 243x48 lattice are being started
Our physics goal is to approach, or to explore the chiral regime with exact chiral symmetry on the lattice.
For example, confirming the chiral symmetry breaking scenario, and testing the effective chiral Lagrangian predictions.
We also want to calculate various matrix elements, with
controlled chiral extrapolation.
The following quantities are just examples, which we are now calculating.

4. Action Neuberger's overlap Dirac operator M0 = 1.6 Iwasaki gauge
With standard Wilson kernel with negative mass −M0
M0 = 1.6
Iwasaki gauge
Extra (two flavors of) Wilson fermions
To suppress near-zero modes of HW
With associated twisted mass ghost
Our physics goal is to approach, or to explore the chiral regime with exact chiral symmetry on the lattice.
For example, confirming the chiral symmetry breaking scenario, and testing the effective chiral Lagrangian predictions.
We also want to calculate various matrix elements, with
controlled chiral extrapolation.
The following quantities are just examples, which we are now calculating.

5. Machines IBM Blue Gene/L at KEK 57.6 Tflops peak (10 racks)
0.5TB memory/rack
8x8x8(16) torus network
~30% performance for Wilson kernel
Overlap HMC: 10~15% on one rack
Hitachi SR11000 (KEK)
2.15TFlops/0.5TB memory
These are machies used in our project.
The main machine is IBM Blue Gene at KEK, which has 57.6 Tflops peak performance with all the 10 racks.
Usually job is executed on each rack or on half rack.
Each rack has 8x8x16 fixed torus network.
The sustained performance for the Wilson Dirac operator is around 30%, which was tuned by IBM Japan.
For HMC, the performance becomes 10 to 15 %.
We are also using Hitachi SR-11,000 at KEK, and NEC SX-8 at Yukawa Institute.

6. Overlap Operator Multi-shift solver can invert (HW+ql) at once
Zolotarev's Rational approximation
Multi-shift solver can invert (HW+ql) at once
Near-zero modes are projected out, treated exactly
Typically N =10 to achieve an accuracy of 10-(7-8)
Overlap solvers
Nested CG with relaxation (Cundy et al., 2005)
5D solver with projection
The overlap fermion was invented by Neuberger.
It is written as this form, where H_W is the hermitian Wilson-Dirac operator with large negative mass.
R is a constant parameter.
The overlap fermion is theoretically elegant.
It satisfies the Ginsparg-Wilson relation, and thus holds exact chiral symmetry on the lattice.
It corresponds to the infinite Ns limit of the domain-wall fermion, and thus free from the residual mass.
However, numerical cost is high because of the implementation of the sign function.
In HMC, one also needs to care the discontinuity of the overlap operator at zero eigenvalue of H_W, during molecular dynamical steps.
So dynamical simulation with overlap fermion has become feasible, only with recent development of improved algorithms and large computrational resources.

7. 5D solver (Borici, 2004, Edwards et al., 2006) Schur decomposition
One can solve by solving (example: N=2 case) ,
: overlap operator (rational approx.)
Even-odd preconditioning
Low-mode projection of HW in lower-right corner
The 5-dimensional solver is implemented by making use of the Schur decomposition.
One can solve the linear equation for the overlap operator by solving 5-dimensional equation.
For example, for N=2 case, M_5 is given by this form.
With this form, the Schur complement S=CA-1B coincides with the overlap operator with Zolotarev's rational approximation.
One can accelerate the solver by the even-odd preconditioning.
The projection of low-lying eigenmodes of H_W is also applicable.
The figure shows the required time to solve the linear system with 4D and 5D Cgs against quark masses.
In the whole region of quark mass, the 5D algorithm is factor 3-4 faster.
So we mainly use 5D solver in HMC.

8. Even-odd preconditioning
1− 𝑀 𝑒𝑒 −1 𝑀 𝑒𝑜 𝑀 𝑜𝑜 −1 𝑀 𝑜𝑒  𝑥 𝑒 = 𝑏 𝑒 ′
Acceleration by solving
Need fast inversion of the “ee” and “oo” block; easy if there is no projection operator
Mee(oo)-1 mixes in the 5th direction, while Meo(oe) is confined in the 4D blocks
Low-mode projection
Lower-right corner must be replaced by
Inversion of Mee(oo) becomes non-trivial, but can be calculated cheaply because the rank of the operator is only 2(Nev+1).
𝑅(1− 𝑃 𝐻 ) 𝛾 5 (1− 𝑃 𝐻 + 𝑝 0 𝐻 𝑊 +  𝑚 0 + 𝑚 2  𝑗=1 𝑁 𝑒𝑣 sgn( 𝜆 𝑗 ) 𝑣 𝑗 ⊗ 𝑣 𝑗 + , 𝑃 𝐻 =1− 𝑗=1 𝑁 𝑒𝑣 𝑣 𝑗 ⊗ 𝑣 𝑗 +
The 5-dimensional solver is implemented by making use of the Schur decomposition.
One can solve the linear equation for the overlap operator by solving 5-dimensional equation.
For example, for N=2 case, M_5 is given by this form.
With this form, the Schur complement S=CA-1B coincides with the overlap operator with Zolotarev's rational approximation.
One can accelerate the solver by the even-odd preconditioning.
The projection of low-lying eigenmodes of H_W is also applicable.
The figure shows the required time to solve the linear system with 4D and 5D Cgs against quark masses.
In the whole region of quark mass, the 5D algorithm is factor 3-4 faster.
So we mainly use 5D solver in HMC.
𝑥 𝑒 , 𝛾 5 𝑥 𝑒 , 𝑣 𝑗𝑒 , 𝛾 5 𝑣 𝑗𝑒

9. Solver performance Comparison on 163x48 lattice
On BG/L 1024-node
(Nsbt=8)
The 5-dimensional solver is implemented by making use of the Schur decomposition.
One can solve the linear equation for the overlap operator by solving 5-dimensional equation.
For example, for N=2 case, M_5 is given by this form.
With this form, the Schur complement S=CA-1B coincides with the overlap operator with Zolotarev's rational approximation.
One can accelerate the solver by the even-odd preconditioning.
The projection of low-lying eigenmodes of H_W is also applicable.
The figure shows the required time to solve the linear system with 4D and 5D Cgs against quark masses.
In the whole region of quark mass, the 5D algorithm is factor 3-4 faster.
So we mainly use 5D solver in HMC.
5D solver is 3-4 times faster than 4D solver

10. Odd number of flavors (Bode et al., 1999, DeGrand and Schaefer, 2006)
Decomposition to chiral sectors is possible
P+H2P+ and P−H2P− share eigenvalues except for zero-modes
1-flavor: one chirality sector
Zero-mode contribution is constant throughout MC, thus neglected
Pseudo-fermion:
σ is either + or -
Refreshing φ from Gaussian distributed ξ as
sqrt is performed using a rational approximation
Other parts are straightforward
The configurations are generated by HMC algorithm.
We apply two popular improvements, the mass preconditioning and the multi-time step.
In addition, for Nf=2, we applied the noisy Metropolis method together with 5D solver without projection of low-modes.
5D solver algorithm will be explained later.
This combination is factor 2 faster than the 4D solver without noisy Metropolis.
Recently we have started Nf=2+1 simulation.
Details of the algorithm will be presented in the poster session.
Nf=1 part is expressed by one chirality sector, by making use of that H2 is decomposed to those of each sector.
We apply the 5D solver with projection of low-modes, which no longer needs noisy Metropolis.

11. Algorithm HMC Hasenbusch preconditioning with heavier mass m'
Multi-time step for PF2(m), PF1(m'), Gauge/ExWilson
Number of two 5D CG iterations
in the calculation of Hamiltonian
The configurations are generated by HMC algorithm.
We apply two popular improvements, the mass preconditioning and the multi-time step.
In addition, for Nf=2, we applied the noisy Metropolis method together with 5D solver without projection of low-modes.
5D solver algorithm will be explained later.
This combination is factor 2 faster than the 4D solver without noisy Metropolis.
Recently we have started Nf=2+1 simulation.
Details of the algorithm will be presented in the poster session.
Nf=1 part is expressed by one chirality sector, by making use of that H2 is decomposed to those of each sector.
We apply the 5D solver with projection of low-modes, which no longer needs noisy Metropolis.
PF2 for ud
PF2 for s

12. 2+1-flavor run status 163x48 lattice, a~0.11 fm
β=2.3, topological charge Q=0
2 strange quark masses around physical ms
ms = 0.10, 0.08
5 ud quark masses covering (1/6~1)ms
mud = 0.015, 0,025, 0,035, 0.050, ms
2,500 trajectories of length 1 for each (mud, ms)
About 2 hours/traj on BG/L 1024 nodes
The simulation with 2 flavors was done on 16^3 x 32 lattices at a=0.12fm.
The production of configuration was finished.
We used 6 quark masses down to one-sixth of the strange quark mass.
The configuration is generated in Q=0 sector, as well as Q=-2 and -4 sectors at half the strange quark mass.
We also performed simulation in the epsilon regime with fairly small quark mass.
2+1 simulation is in progress, with almost the same parameters as Nf=2, but the temporal extent is changed to 48.
We adopt two strange quark masses around the physical strange quark mass, and 5 ud quark masses covering the same region as Nf=2.
At present, about 2 hours are needed on 1 rack of Blue Gene, to generate 1 trajectory of unit length.

13. Lattice scale Scale: set by r0 = 0.49fm
Nf=2 result includes
systematic error.
The lattice scale is set by the hadronic radius r_0, by setting to 0.49fm.
The right figure shows the quark mass dependence of the lattice spacing.
For Nf=2+1, the result is very preliminary.
The left figure shows the beta-shift for the Nf=2 case together with the clover fermion case by CP-PACS.
Compared to the Wilson-type fermions, overlap fermion has milder beta-shift.
Strange quark effect is invisible
Slightly smaller lattice spacing than Nf=2
Milder b-shift than Wilson-type fermions

14. Spectrum Pion mass and decay const Pion mass squared
Comparison with ChPT is in progress
Fit parameter: (fp is mass dependent)
Talk by J.Noaki (Tue)
Pion mass squared
Pion decay constant
These are the results of pion mass and decay constant.
The finite volume corrections, including the fixed Q effect, was already done.
Then the chiral extrapolation is done, with the fit parameter x=(mp/4pfp)2, where fp is quark mass dependent.
We apply NLO and NNLO forms of the chiral perturbation theory.
We found that NLO fit tends to fail, namely does not give good values of the low-energy constants, but NNLO fit seems successful giving these values.
Details will be given in Noaki's talk in this afternoon.

15. Further improvement Chronological estimator (Brower et al., 1997)
Approx. solution for CG solver from previous solutions
From 4D estimate ψ, estimate of 5D solver is constructed:
φ is given by solving (by multi-shift solver)
Keeping 4D solution vectors: less memory consuming
These are the results of pion mass and decay constant.
The finite volume corrections, including the fixed Q effect, was already done.
Then the chiral extrapolation is done, with the fit parameter x=(mp/4pfp)2, where fp is quark mass dependent.
We apply NLO and NNLO forms of the chiral perturbation theory.
We found that NLO fit tends to fail, namely does not give good values of the low-energy constants, but NNLO fit seems successful giving these values.
Details will be given in Noaki's talk in this afternoon.
Also applicable to ''adaptive 5D solver''
Change N (number of poles) during iteration

16. Test of chronological estimator
Convergence of 5D solver
163x48, 2+1 flavor
m=0.015, m'=0.2
m=0.080, m'=0.4
Estimate with 5 previous solution vectors
Polynomial extrap.
MRE (minimum residual extrap.)
These are the results of pion mass and decay constant.
The finite volume corrections, including the fixed Q effect, was already done.
Then the chiral extrapolation is done, with the fit parameter x=(mp/4pfp)2, where fp is quark mass dependent.
We apply NLO and NNLO forms of the chiral perturbation theory.
We found that NLO fit tends to fail, namely does not give good values of the low-energy constants, but NNLO fit seems successful giving these values.
Details will be given in Noaki's talk in this afternoon.
For preconditioner, polynomial extrap. works well
To keep reversibility, higher precision is required
not efficient without other improvement

17. Summary/Outlook
We are performing dynamical overlap project at fixed topological charge
Nf=2 on 163x32, a~0.12fm: producing rich physics results
Nf=2+1 on 163x48, a~0.11fm: generation finished, physics measurements in progress
Nf=2+1 on 243x48 being started
Further improvements of algorithm are essential
Solver with deflation
Improved stimator (Omelyan, etc.)
Supply of configs to ILDG is in preparation
Nf=2 will be soon
Cf. T.Yoshie's talk
Let me summarize my talk.
We are performing dynamical overlap project at fixed topological charge.
Nf=2 simulation is producing rich physics results, and Nf=2+1 simulation has also been stated.
Our goals are understanding the chiral dynamics with exact chiral symmetry, and computation of various matrix elements with good accuracy in the chiral limit.
These configurations will be supplied to International Lattice DataGrid after first publication of spectrum paper.
Outlook.
We of course want to go to larger lattices.
Because of restriction of Blue Gene hardware, we have to go to 24^3 x 48 lattice.
Present algorithm is still not enough for that size, so further improvement is necessary.
If you have good idea, please tell us.
That' all. Thank you.