The Distributed Data Interface in GAMESS Brett M. Bode, Michael W. Schmidt, Graham D. Fletcher, and Mark S. Gordon Ames Laboratory-USDOE, Iowa State University 10/7/99

2 b General Atomic and Molecular Electronic Structure System First principles - fully quantum mechanicalFirst principles - fully quantum mechanical Created from other programs in ~1980Created from other programs in ~1980 Developed by Dr. Mark Gordon’s research group since 1982 with Dr. Michael Schmidt as the principle developer.Developed by Dr. Mark Gordon’s research group since 1982 with Dr. Michael Schmidt as the principle developer. Parallelization begin in 1991Parallelization begin in 1991 Emphasis on Distributed memory systemsEmphasis on Distributed memory systems Currently includes methods for treating 1-atom to several hundred atomsCurrently includes methods for treating 1-atom to several hundred atoms What is GAMESS?

3 Partial list of capabilities C = Uses disk storage D = Minimal disk usage P = Parallel execution

4 First Generation Parallel Code b Parallel communications were performed using either: TCGMSGTCGMSG Vendor supplied MPIVendor supplied MPI b Parallel version was usually a slightly modified version of the sequential code

5 IBM-SUR cluster b 22 IBM RS/ P-260: –Dual 200MHz Power3 CPUs –4 Mb of Level 2 cache –1 GByte of RAM –18 GBytes fast local disks –Jumbo Frames Gig Ethernet –Integrated Fast-Ethernet b Fast Ethernet Switch to all b 3x9 port Gigabit Switches

6 Gigabit Performance on the IBM 43P-260 Cluster

7 Test Molecule b b Ti(C 5 H 5 ) 2 C 2 H 4 SiHCl 3 b b Basis Set 6-31G(d,p) on C and H. SBKJC ECP on Si, Ti, and Cl extended with 1 d-type polarization function on Si and Cl. 345 total basis functions

8 Parallel SCF b Very good scaling dependant on the size of the molecule. b Large systems show nearly linear scaling through 256 nodes

9 Successes and Limitations b SCF methods scale very well b Most methods run in parallel b Good use is made of aggregate CPU and disk resources. b MP2 and MCSCF methods scale to only a few (8-32) nodes b The aggregate memory is not utilized so jobs are still limited by the memory size of one node.

10 Second Generation Methods b New methods should take advantage of the aggregate memory of a parallel system Implies a higher communication demandsImplies a higher communication demands Many to many messaging profileMany to many messaging profile b Methods should scale to hundreds of nodes (at least) b Demanding local storage needs

11 The Distributed Data Interface (DDI) DDI provides the core functions needed to treat a portion of the memory on each node as part of a global shared array.

12 DDI b Runs on top of: MPI (MPI-2 preferred)MPI (MPI-2 preferred) TCP/IP socketsTCP/IP sockets b Lightweight - Provides only the functionality needed by GAMESS b Is not intended as a general purpose library. b Does optimize for mixed SMP and distributed memory systems

13 New MP2 implementation b Uses DDI to utilize the aggregate memory of the parallel machine at the expense of communications b Trades some symmetry in the MP2 equations for better parallel scalability Requires more memory than the sequential versionRequires more memory than the sequential version Is slower than the sequential version on 1 CPUIs slower than the sequential version on 1 CPU

14 MP2 Scalability

15 Conclusions b DDI provides a scalable way of taking advantage of the global memory of a parallel system b The new MP2 code demonstrates code written specifically for parallel execution without replacing the sequential version.

16 Future Work b DDI needs further work to enhance the features and increase robustness, or possibly needs to be replaced with a more general library such as the GA tools from PNNL. b The global shared memory approach is being applied to many other parts of GAMESS to increase scalability.

17 Thanks! b David Halstead b Guy Helmer For $: b IBM Corp. for an SUR grant (of 15 Workstations) b DOE MICS program (interconnects and 7 workstations) b Air Force OSR (long term dev. Funding) b DOD CHSSI program (improved parallelization)