1. National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Porting, Benchmarking, and Optimizing Computational Material Science Packages on TeraGrid Resources Dodi Heryadi Advanced Application Support Group

2. Imaginations unbound Top 15 Applications at NCSA based on the # of users Gaussian NAMD CHARMM VASP ABAQUS AMBER ENZO FLUENT MRBAYES GROMACS MOLPRO CACTUS GAMESS FLASH ANSYS [data collected from Oct 1 st 2006 – Dec 31 st 2006]

3. Imaginations unbound Software Packages commonly used in Bio/Molecular Science and Engineering Gaussian Gamess NWChem Molpro ADF Amber Gromacs CHARMM NAMD DLPOLY LAMMPS CPMD VASP Wien2k SIESTA Abinit CASTEP DMol3

4. Imaginations unbound Some of the Packages Used in Material Science and Engineering Community VASP CPMD Wien2k SIESTA Abinit CASTEP DMol3

5. Imaginations unbound Porting, Benchmarking, and Optimizing Computational Material Science packages on the TeraGrid Resources To assist users in selecting the best resources when applying for Allocations To assist users in increasing their productivity using TeraGrid resources

6. Imaginations unbound First package: VASP (Vienna ab-initio simulation package) a package to perform ab-initio quantum-mechanical molecular dynamics (MD) using pseudopotentials and a plane wave basis set large user base (over 20 research groups/PIs at NCSA alone) can scale to 1,024 processors (http://www.hpcx.ac.uk/research/materials/vasp.html) restriction: licensed to individual research groups

7. Imaginations unbound (old) VASP Benchmarks on NCSA platforms http://www.ncsa.uiuc.edu/UserInfo/Resources/Software/CHEM/vaspbench.html cpus IBM p690 (copper) Xeon Cluster (tungsten) IA-64 Linux Cluster (mercury) 17,8406,1683,939 24,0263,9362,496 41,9512,0771,588 81,0391,177 868 16 602 808 525 32 323 613 290 Pure Ni, 3x3x3 supercell, 2x2x2 kpoints, 10 electrons GGA pseudopotential Wall Clock Time (in seconds)

8. Imaginations unbound Porting and Benchmarking VASP on abe and ranger Compilers: Intel 10.1 BLAS and LAPACK Libraries: Intel MKL 10.0.1.014 MPI: MVAPICH Internal FFTW libraries

9. Imaginations unbound Preliminary Results Mn12-acetate Wall clock time (in seconds) # of coresAbe (ppn=8)Ranger (ppn=16) 1635,83619,740 3219,54711,029 6412,6537,159 1287,6887,272 256(job is still in the queue)7,997

10. Imaginations unbound Work to do Port and benchmark VASP (and other widely used Computational Material Science packages) on other TG resources Next: kraken Optimize VASP on abe, ranger, kraken, and other TG resources Performance analysis/tools to identify performance bottlenecks Selecting appropriate compiler options for optimal performance Using optimized Math libraries (e.g. Intel FFTW, SCALAPACK, AMD Math Libraries, etc.) Lonnie Crosby (NICS) and Yang Wang (PSC) will be involved in this effort

11. Imaginations unbound Preliminary Source Level Profiling of VASP on abe with perfsuite (http://perfsuite.ncsa.uiuc.edu/)http://perfsuite.ncsa.uiuc.edu/ Module Summary -------------------------------------------------------------------------------- Samples Self % Total % Module 1590 79.42% 79.42% /cfs/scratch/users/dodi/vaspbench/perf/vasp 397 19.83% 99.25% /usr/local/mvapich2-0.9.8p2patched-intel-ofed-1.2/lib/libmpich.so 12 0.60% 99.85% /usr/local/lib64/tls/libpthread-2.3.4.so 3 0.15% 100.00% /usr/local/lib64/tls/libc-2.3.4.so

12. Imaginations unbound Preliminary Source Level Profiling of VASP on abe with perfsuite (http://perfsuite.ncsa.uiuc.edu/)http://perfsuite.ncsa.uiuc.edu/ File Summary -------------------------------------------------------------------------------- Samples Self % Total % File 1175 58.69% 58.69% ?? 344 17.18% 75.87% /u/ncsa/dodi/vaspnew/vasp.4.6/fft3dlib.f90 105 5.24% 81.12% /u/ncsa/dodi/vaspnew/vasp.4.6/rmm-diis.f90 86 4.30% 85.41% /u/ncsa/dodi/vaspnew/vasp.4.6/nonlr.f90 62 3.10% 88.51% /u/ncsa/dodi/vaspnew/vasp.4.lib/dlexlib.f 61 3.05% 91.56% /u/ncsa/dodi/vaspnew/vasp.4.6/hamil.f90 46 2.30% 93.86% /u/ncsa/dodi/vaspnew/vasp.4.6/fftmpi.f90 41 2.05% 95.90% /u/ncsa/dodi/vaspnew/vasp.4.6/fftmpi_map.f90 15 0.75% 96.65% /u/ncsa/dodi/vaspnew/vasp.4.6/dfast.f90 13 0.65% 97.30% /u/ncsa/dodi/vaspnew/vasp.4.6/wave.f90 12 0.60% 97.90% /u/ncsa/dodi/vaspnew/vasp.4.6/mpi.f90 12 0.60% 98.50% /u/ncsa/dodi/vaspnew/vasp.4.6/subrot.f90 7 0.35% 98.85% /u/ncsa/dodi/vaspnew/vasp.4.6/us.f90 6

13. Imaginations unbound Function Summary -------------------------------------------------------------------------------- Samples Self % Total % Function 208 10.39% 10.39% fpassm 194 9.69% 20.08% M_LOOP 134 6.69% 26.77% ipassm 129 6.44% 33.22% __intel_new_memcpy 105 5.24% 38.46% eddrmm 90 4.50% 42.96% MPIDI_CH3I_MRAILI_Get_next_vbuf 87 4.35% 47.30% MPIDI_CH3I_SMP_pull_header 84 4.20% 51.50% MPIDI_CH3I_SMP_read_progress 70 3.50% 55.00% mkl_lapack_dlaebz 62 3.10% 58.09% length 41 2.05% 60.14% raccmu 41 2.05% 62.19% fftwav 37 1.85% 64.04% hamiltmu 37 1.85% 65.88% mkl_blas_mc_zhemv_nb 36 1.80% 67.68% mkl_blas_mc_zgemm_copyac 35 1.75% 69.43% MPIDI_CH3I_SMP_write_progress 31 1.55% 70.98% rpromu 30 1.50% 72.48% map_forward 28 1.40% 73.88% AY16_Loop_M16 24 1.20% 75.07% A16X8_N4_Loop_M16

14. Imaginations unbound Next: More Detailed Performance Analysis mpiP (http://mpip.sourceforge.net/) -- Lightweight, Scalable MPI Profilinghttp://mpip.sourceforge.net/ 20% of the time spent on MPI TAU (Tuning and Analysis Utilities) (http://www.cs.uoregon.edu/research/tau/home.php)

15. Imaginations unbound Acknowledgement Rick Kufrin and Rui Liu, NCSA Dave McWilliams, NICS