1. Computational issues in Carbon nanotube simulation Ashok Srinivasan Department of Computer Science Florida State University

2. Outline Background –Sequential computation –Performance analysis Parallelization –Shared memory –Message passing –Load balancing –Communication reduction Research issues

3. Background Uses of Carbon nanotubes –Materials –NEMS –Transistors –Displays –Etc www.ipt.arc.nasa.gov

4. Sequential computation Molecular dynamics, using Brenner’s potential –Short-range interactions –Neighbors can change dynamically during the course of the simulation –Computational scheme Find force on each particle due to interactions with “close” neighbors Update position and velocity of each atom

5. Force computations Bond angles Pair interactions Dihedral Multibody

6. Performance analysis

7. Profile of execution time 1: Force 2: Neighbor list 3: Predictor/corrector 4: Thermostating 5: Miscellaneous

8. Profile for force computations

9. Parallelization Shared memory Message passing Load balancing Communication reduction

10. Shared memory parallelization Do each of the following loops in parallel –For each atom Update forces due to atom i If neighboring atoms are owned by other threads, update an auxiliary array –For each thread Collect force terms for atoms it owns –Srivastava, et al, SC-97 and CSE 2001 Simulated 10 5 to 10 7 atoms Speedup around 16 on 32 processors Include long-range forces too Lexical decomposition

11. Message passing parallelization Decompose domain into cells –Each cell contains its atoms Assign a set of adjacent cells to each processor Each processor computes values for its cells, communicating with neighbors when their data is needed Caglar and Griebel, World scientific, 1999 –Simulated 10 8 atoms on up to 512 processors –Linear speedup for 160,000 atoms on 64 processors

12. Load balancing Atom based decomposition –For each atom, compute forces due to each bond, angle, and dihedral –Load not balanced

13. Load balancing... 2 Bond based decomposition –For each bond, compute forces due to that bond, angles, and dihedrals –Finer grained –Load still not balanced!

14. Load balancing... 3 Load imbalance was not caused by granularity –Symmetry is used to reduce calculations through –If i > j, don’t compute for bond (i,j) So threads get unequal load –Change condition to If i+j is even, don’t compute bond (i,j) if i > j If i+j is odd, don’t compute bond (i,j) if i < j Does not work, due to regular structure of nanotube –Use a different condition to balance load

15. Load balancing... 4 Load is much better balanced now –... at least for this simple configuration

16. Locality Locality important to reduce cache misses Current scheme based on lexical ordering Alternate: Decompose based on a breadth first search traversal of the atom-interaction graph

17. Locality... 2

18. Research issues Neighbor search Parallelization –Dynamic load balancing –Communication reduction Multi-scale simulation of nano-composites

19. Neighbor search Neighbor lists –Crude algorithm Compare each pair, and determine if they are close enough O(N 2 ) for N atoms –Cell based algorithm Divide space into cells Place atoms in their respective cells Compare atoms only in neighboring cells Problem –Many empty cells –Inefficient use of memory

20. Computational geometry techniques Orthogonal search data structures –K-d tree Tree construction time: O(N log N) Worst case search overhead: O(N 2/3 ) Memory: O(N) –Range tree Tree construction time: O(N log 2 N) Worst case search overhead: O(log 2 N) Memory: O(N log 2 N)

21. Desired properties of search techniques Update should be efficient –But the number of atoms does not change –Position changes only slightly –The queries are known too –May be able to use knowledge of the structure of the nanotube –Parallelization

22. Parallelization Load balancing and locality –Better graph based techniques –Geometric partitioning –Dynamic schemes Use structure of the tube Stochastic versions of –Spectral partitioning –Diffusive schemes

23. Multi-scale simulation of nano- composites Molecular dynamics at nano-scale Finite element method at larger scale –New models to links the scales –New algorithms to dynamically balance the loads, while minimizing communication –Latency tolerance and scalability to large numbers of processors