UT-BATTELLE Scaling of First Principles Electronic Structure Methods on Future Architectures W.A. Shelton Oak Ridge National Laboratory

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Collaborators Locally Self-consistent Multiple Scattering Method (Real Space) Oak Ridge National Laboratory N.Y. Moghadam, D.M.C. Nicholson, G.M. Stocks, X.-G. Zhang, and B. Ujfalussy Pittsburgh Supercomputer Center Y. Wang National Energy Research Supercomputer Center A. Canning Screened Methods (Tight-binding like methods) Oak Ridge National Laboratory A. Smirnov University of Illinois (Urbana-Champaign) D.D. Johnson

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Acknowledgement of Sponsors Department of Energy/Office of Science Office of Advanced Scientific Computing Research Mathematics, Information and Computer Science Applied Mathematical Sciences Program Oak Ridge National Laboratory Laboratory Directors Research and Development Program Computing Resources at the Center of Computational Sciences located at Oak Ridge National Laboratory Pittsburgh Supercomputing Center National Energy Research Supercomputing Center located at Lawrence Berkeley National Laboratory

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Motivation Rethink the mathematical model Design new algorithms that renders a numerical solution Open new possibilities –Improved scaling –For solving problems that previously were untenable –Software technologies being used by researchers with access to less advanced hardware technologies The introduction of new architectures

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Motivation N anoscale S cience and E ngineering T echnology Initiative SOFT MATERIALS –Synthetic Polymers and Bio-Inspired Materials –Systems Dominated by Organic-Inorganic Interconnections –Interfacing Nanostructures to Biological Systems HYBRID SOFT-HARD MATERIALS –Carbon-Based Nanostructures –Characterization of Active Sites in Catalytic Materials –Nanoporous Membranes and Nanomaterials for Ultra-Selective Catalysis COMPLEX HARD MATERIALS –Magnetism in Nanostructured Materials –Nanoscale Manipulation of Collective Behavior –Nanoscale Interface Science (Nanoparticles and Nanograins) –Electromagnetic Fields in Confined Structures THEORY / MODELING / SIMULATION –Virtual Synthesis and Nanomaterials Design –Theoretical Nano-Interface Science

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Multiscale Simulations Connect Microscopic-level Processes to Macroscopic Response of Material

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Scalable and Accurate First Principles Method Atomistic Methods

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Quantum Simulation Goals: Accuracy and Predictive Capabilities

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Advances In Hardware Alone Are Not Sufficient

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Linear Scaling Algorithms Will Enable Solutions to New Problems The combination of new advanced computing platforms and new scaling algorithms will open new areas in quantum-level materials simulations

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Algorithm Design for future generation architectures More accurate Spectral or pseudo-spectral accuracy Wider range of applicability Sparse representation Memory requirements grow linearly Each processor can treat thousands of atoms Make use of large number of processors Message-Passing Each atom/node local message-passing is independent of the size of the system Time consuming step of model Sparse linear solver Direct or preconditioned iterative approach

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Density Functional Theory (DFT) DFT in principle is an exact method for treating the many body quantum mechanical effects of electron exchange and correlation At the heart of this formulation is the ascertain that the ground-state total energy of an electron system in the field of the atomic nuclei is a unique functional of the electronic charge density The total energy functional attains it minimum value when evaluated with the true ground-state electronic charge density. The quantum mechanical effects of electron exchange and correlation are contained in the non-local exchange-correlation potential V ex-corr (r,r ’). –Hence, the electronic interactions are explicitly accounted for by the fundamental quantity, the electronic charge density –Unfortunately, there are no analytical forms for calculating V ex-corr (r,r ’ )

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY LSDA &Multiple Scattering Theory (MST ) No Yes Mix in & out Initial guess n in (r), m in (r) Calculate V eff [n,m] in Solve Schrodinger Equation Recalculate n out (r), m out (r) n in (r) = n out (r) ? m in (r) =m out (r) ? Calculate Total Energy  Multiple Scattering Theory (MST) J. Korringa, Physica 13, 392, (1947) W. Kohn, N. Rostoker, PR, 94, 1111,(1954)  MST Green function methods B. Gyorffy, and M. J. Stott, “Band Structure Spectroscopy of Metals and Alloys”, Ed. D.J. Fabian and L. M. Watson (Academic 1972) S.J. Faulkner and G.M. Stocks, PR B 21, 3222, (1980)

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Complex Energy Plane The scattering properties at complex energy can be used to develop highly efficient real-space and k-space methods Real  Im  ff Semi-conductors and insulators could work well since they have no states at  f Scattering is local since there are no states near the bottom of the energy contour Scattering is local since a large Im  is equivalent to rising temperature which smears out the states Near  f scattering is non-local (metal) Real  Im  ff  f is the highest occupied electronic state in energy

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY  Multiple scattering theory Green function Scattering path matrix Multiple Scattering Theory decay slowly with increasing distance contain free-electron singularities ][ M  Generalization of t-matrix. Converts incoming wave at siteinto outgoing wave at site in the presence of all the other sites

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Real Space Algorithm Design Linear scaling –Each node performs a fixed size local calculation Thus each node performs the same number of flops Message-Passing –Each atom/node local message-passing is independent of the size of the system Time consuming step of model –Reduce to Linear Algebra step BLAS level 3

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Real Space Parallel Implementation Green’s function Scattering path matrix: real space t : scattering from single site G: structure constant matrix M=[t -1 (  )-G(R ij,  ]  =M -1 Once M is fixed increasing N does not affect the local calculation of M -1 The LSMS naturally scales linearly with increasing N

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Matrix Inversion A B CD Take A and continue to partition until the desired matrix size (l max + 1) 2 of the central site is reached Partition the m(l max + 1) 2 matrix,  =MxM into M=M 1 + M 2 into four blocks two of size M 1 and two of size M 2 A -1 =A-B D -1 C Note that the LxL diagonal block of A -1 is the same LxL block that is desired.

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY (N) Scaling of Real Space Method 1998 Gordon Bell Prize 1.02 TFLOPS on a 1500 node Cray T3E

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY (N) Scaling of Real Space Method

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Real Space Accuracy fcc Cu bcc Cu bcc Mo hcp Co

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Tight-Binding MST Representation Tight Binding Multiple Scattering Theory –Embed a constant repulsive potential Shifts the energy zero allowing for calculations at negative energy –Rapidly decaying interactions –Free electron singularities are not a problem –Sparse representation 0 VsVs Constant inside a sphere > } 2 Ryd. eiReiR R

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Screened Structure Constants Screened Structure Constants G s on the left unscreened on the right –Screened structure constants rapidly go to zero, whereas the free space structure constants have hardly changed Linear solve using m atom cluster that is less than the n atom system Easy to perform Fourier transform –K-space method

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Screened MST Methods Formulation produces a sparse matrix representation –2-D case has tridiagonal structure with a few distant elements due to periodicity –3-D case has scattered elements Mainly due to mapping 3-D structure to a matrix (2-D) A few elements due to periodic boundary conditions Require block diagonals of the inverse of  matrix –Block diagonals represent the site  matrix and are needed to calculate the Green’s function for each atomic site Sparse direct and preconditioned iterative methods are used to calculate  ii (  ) –SuperLU –Transpose free Quasi-Minimal Residual Method (TFQMR)

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Screened KKR Accuracy fcc Cu bcc Cu bcc Mo hcp Co

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Timing and Scaling of Scr-K KKR-CPA

Computer Science and Mathematics UT-BATTELLE U.S. D EPARTMENT OF E NERGY O AK R IDGE N ATIONAL L ABORATORY Conclusion Initial benchmarking of the Screened KKR method –SuperLU N 1.8 for finding the inverse of the upper left block of  –TFQMR with block Jacobi preconditioner N 1.06 for finding the inverse of the upper left block of  Extremely high sparsity (97%-99% zeros increases with increasing system size) Large number of atoms on a single processor Real-space/Scr-KKR hybrid may provide the most efficient parallel approach for new generation architectres Single code contains –LSMS, KKR-CPA, Scr-LSMS and Scr-KKR-CPA