Presentation is loading. Please wait.

Presentation is loading. Please wait.

1Managed by UT-Battelle for the U.S. Department of Energy Optimal Biomolecular simulations in heterogeneous computing architectures Scott Hampton, Pratul.

Published by Modified over 8 years ago

Similar presentations

Presentation on theme: "1Managed by UT-Battelle for the U.S. Department of Energy Optimal Biomolecular simulations in heterogeneous computing architectures Scott Hampton, Pratul."— Presentation transcript:

1 1Managed by UT-Battelle for the U.S. Department of Energy Optimal Biomolecular simulations in heterogeneous computing architectures Scott Hampton, Pratul Agarwal Funding by:

2 2Managed by UT-Battelle for the U.S. Department of Energy Motivation: Better time-scales Protein dynamical events 10 -15 s10 -12 10 -9 10 -6 10 -3 10 0 s k B T/h Side chain and loop motions 0.1–10 Å Elastic vibration of globular region <0.1 Å Protein breathing motions Folding (3° structure), 10–100 Å Bond vibration H/D exchange 10 -15 s10 -12 10 -9 10 -6 10 -3 10 0 s Experimental techniques NMR: R 1, R 2 and NOE Neutron scattering NMR: residual dipolar coupling Neutron spin echo 2° structure Molecular dynamics (MD) 10 -15 s10 -12 10 -9 10 -6 10 -3 10 0 s 2005 2010 Agarwal: Biochemistry (2004), 43, 10605-10618; Microbial Cell Factories (2006) 5:2; J. Phys. Chem. B 113, 16669-80.

3 3Managed by UT-Battelle for the U.S. Department of Energy A few words about how we see biomolecules protein vibration bulk solvent hydration shell Enzyme substrate Agarwal, P. K., Enzymes: An integrated view of structure, dynamics and function, Microbial Cell Factories. (2006) 5:2 Solvent is important: Explicit solvent needed Fast motions in active-site Slow dynamics of entire molecules

4 4Managed by UT-Battelle for the U.S. Department of Energy Introduction: MD on future architectures Molecular biophysics simulations – Molecular dynamics: time-evolution behavior – Molecular docking: protein-ligand interactions Two alternate needs – High throughput computing (desktop and clusters) – Speed of light (supercomputers) Our focus is on High Performance Architectures – Strong scaling: Simulating fixed system on 100-1000s of nodes – Longer time-scales: microsecond or better – Larger simulations size (more realistic systems)

5 5Managed by UT-Battelle for the U.S. Department of Energy A look at history 1 Gflop/s 1 Tflop/s 100 Gflop/s 100 Tflop/s 10 Gflop/s 10 Tflop/s 1 Pflop/s 100 Pflop/s 10 Pflop/s N=1 N=500 Top 500 Computers in the World Historical trends Courtesy: Al Geist (ORNL), Jack Dongarra (UTK) 2012 2015 2018

6 6Managed by UT-Battelle for the U.S. Department of Energy Concurrency Fundamental assumptions of system software architecture and application design did not anticipate exponential growth in parallelism Courtesy: Al Geist (ORNL)

7 7Managed by UT-Battelle for the U.S. Department of Energy Future architectures Fundamentally different architecture – Different from the traditional MPP (homogeneous) machines Very high concurrency: Billion way in 2020 – Increased concurrency on a single node Increased Floating Point (FP) capacity from accelerators – Accelerators (GPUs/FPGAs/Cell/? etc.) will add heterogeneity Significant Challenges: Concurrency, Power and Resiliency Significant Challenges: Concurrency, Power and Resiliency 2012  2015  2018 …

8 8Managed by UT-Battelle for the U.S. Department of Energy MD in heterogeneous environments “Using FPGA Devices to Accelerate Biomolecular Simulations”, S. R. Alam, P. K. Agarwal, M. C. Smith, J. S. Vetter and D. Caliga, IEEE Computer, 40 (3), 2007. “Energy Efficient Biomolecular Simulations with FPGA-based Reconfigurable Computing”, A. Nallamuthu, S. S.Hampton, M. C. Smith, S. R. Alam, P. K. Agarwal, In proceedings of the ACM Computing Frontiers 2010, May 2010. “Towards Microsecond Biological Molecular Dynamics Simulations on Hybrid Processors”, S. S. Hampton, S. R. Alam, P. S. Crozier, P. K. Agarwal, In proceedings of The 2010 International Conference on High Performance Computing & Simulation (HPCS 2010), June 2010. “Optimal utilization of heterogeneous resources for biomolecular simulations” S. S. Hampton, S. R. Alam, P. S. Crozier, P. K. Agarwal, Supercomputing Conference 2010. Accepted. Multi-core CPUs & GPUs FPGAs = low power solution?

9 9Managed by UT-Battelle for the U.S. Department of Energy Our target code: LAMMPS NVIDIA GPUs + CUDA *http://lammps.sandia.gov/ **http://code.google.com/p/gpulammps/ LAMMPS*: Highly scalable MD code Scales >64 K cores Open source code Supports popular force-fields – CHARMM & AMBER Supports a variety of potentials – Atomic and meso-scale – Chemistry and Materials Very active GPU-LAMMPS community** Single workstations And GPU-enabled clusters

10 10Managed by UT-Battelle for the U.S. Department of Energy Off-loading: improving performance in strong scaling Other Alternative: Entire (or most) MD run on GPU Computations are free, data localization is not – Host-GPU data exchange is expensive – In a multiple GPU and multi-node: much worse for entire MD on GPU We propose/believe: Best time-to-solution will come from not using a single resource but most (or all) heterogeneous resources Keep the parallelism that already LAMMPS has – Spatial decomposition: Multi-core processors/MPI Our Approach: Gain from multi-level parallelism

11 11Managed by UT-Battelle for the U.S. Department of Energy CPU Compute bonded terms If needed update the neighbor list Compute the electrostatic and Lennard-Jones interaction terms for energy/forces Collect forces, time integration (update positions), adjust temperature/pressure, print/write output Loop: i = 1 to N CPU {Bond}{Neigh}{Pair}{Other + Outpt} {Comm} = communication δtδt Host only Loop: i = 1 to N GPU v1 Compute bonded terms Compute the electrostatic and Lennard-Jones interaction terms for energy/forces Collect forces, time integration (update positions), adjust temperature/pressure, print/write output GPU CPU If needed update the neighbor list atomic positions & neighbor’s list forces GPU as a Co-processor GPU v2/v3 If needed update the neighbor list Compute the electrostatic and Lennard-Jones interaction terms for energy/forces Collect forces, time integration (update positions), adjust temperature/pressure, print/write output Loop: i = 1 to N GPU CPU atomic positions forces Compute bonded terms GPU as an Accelerator With concurrent computations on CPU & GPU Data locality!

12 12Managed by UT-Battelle for the U.S. Department of Energy Results: JAC benchmark Single workstation SerialIntel Xeon E5540 with Tesla C1060 JACCPUGPU v1GPU v2GPU v3 Pair (%) 310.8 (93.9)12.91(27.8)13.6 (40.4)8.4 (54.0) Bond (%) 5.2 (1.6)5.1 (10.9)5.2 (15.3)5.0 (32.4) Neigh (%) 12.9 (3.9)26.5 (57.0)12.9 (38.1)0.1 (0.4) Comm (%) 0.2 (0.1)0.2 (0.5)0.2 (0.7)0.2 (1.5) Other (%) 1.9 (0.6)1.8 (3.9)1.9 (5.5)1.8 (11.6) Total (s) 331.046.533.815.5 Non-bonded 2 GPUsIntel Xeon E5540 with Tesla C1060 JACCPUGPU v1GPU v2GPU v3 Pair (%) 162.6 (78.1)6.5 (23.4)7.0 (34.9)4.6 (45.0) Bond (%) 2.9 (1.4)2.6 (9.2)2.6 (12.7)2.6 (25.2) Neigh (%) 6.7 (3.2)12.9 (46.4)6.4 (31.5)0.0 (0.3) Comm (%) 34.8 (16.7)4.7 (16.9)3.1 (15.2)1.9 (18.3) Other (%) 1.2 (0.6)1.1 (4.1)1.1 (5.7)1.1 (11.1) Total (s) 208.227.920.210.2 4 GPUsIntel Xeon E5540 with Tesla C1060 Pair (%) 76.4 (58.1)3.3 (19.1)3.7 (29.5)2.7 (43.3) Bond (%) 1.3 (1.0)1.3 (7.3)1.3 (10.2)1.3 (20.0) Neigh (%) 3.1 (2.4)6.2 (35.8)3.1 (24.6)0.0 (0.3) Comm (%) 50.1 (38.0)5.8 (33.6)3.8 (30.1)1.6 (25.1) Other (%) 0.7 (0.6)0.7 (4.0)0.7 (5.6)0.7 (11.1) Total (s) 131.617.312.66.3 Joint AMBER-CHARMM 23,558 atoms (Cut-off based) Intel Xeon E5540 2.53 GHz

13 13Managed by UT-Battelle for the U.S. Department of Energy Performance: Single-node (multi-GPUs) Single workstation with 4 Tesla C1060 cards 10-50X speed-ups, larger systems – data locality Super-linear speed-ups for larger systems Beats 100-200 cores of ORNL Cray XT5 (#1 in Top500) Performance metric (ns/day)

14 14Managed by UT-Battelle for the U.S. Department of Energy Challenge: How do all cores use a single (or limited) GPUs? Solution: Use pipelining strategy* Pipelining: scaling on multi-core/ multi-node with GPUs * = Hampton, S. S.; Alam, S. R.; Crozier, P. S.; Agarwal, P. K. (2010), Optimal utilization of heterogeneous resources for biomolecular simulations. Supercomputing 2010. v4

15 15Managed by UT-Battelle for the U.S. Department of Energy Results: Communications overlap Need to overlap off-node/on-node communications – Very important for strong scaling mode less off-node communication more off-node communication Important lesson: Off/on node communication overlap 2 Quad-core Intel Xeon E5540 2.53 GHz

16 16Managed by UT-Battelle for the U.S. Department of Energy Results: NVIDIA’s Fermi Early access to Fermi card: single vs double precision Fermi : ~6X more double precision capability than Tesla series Better and more stable MD trajectories Double precision Intel Xeon E5520 (2.27 GHz)

17 17Managed by UT-Battelle for the U.S. Department of Energy Results: GPU cluster 24-nodes Linux cluster: 4 quad CPUs + 1 Tesla card per node – AMD Opteron 8356 (2.3 GHz), Infiniband DDR Pipelining allows all up to 16 cores to off-load to 1 card Improvement in time-to-solution LAMMPS (CPU-only) Nodes1 c/n2 c/n4 c/n8 c/n16 c/n 115060.07586.93915.92007.81024.1 27532.63927.61990.61052.9580.5 43920.31948.11028.9559.2302.4 81956.01002.8528.1279.5192.6 16992.0521.0262.8168.5139.9* 24673.8335.0188.7145.1214.5 GPU-enabled LAMMPS (1 C1060/node) Nodes1 c/n2 c/n4 c/n8 c/n16 c/n 13005.51749.91191.4825.0890.6 21304.5817.9544.8515.1480.6 4598.6382.2333.3297.0368.6 8297.9213.2180.1202.0311.7 16167.3126.7118.8176.5311.1 24111.189.2*108.3196.3371.1 Protein in water 320,000 atoms (Long range electrostatics)

18 18Managed by UT-Battelle for the U.S. Department of Energy Results: GPU cluster Optimal use: matching algorithm with hardware The best time-to-solution comes from multi-level parallelism – Using CPUs AND GPUs Data locality makes a significant impact on the performance Cut-off = short range forces only PME = Particle mesh Ewald method for long range 96,000 atoms (cut-off) 320,000 atoms (PPPM/PME)

19 19Managed by UT-Battelle for the U.S. Department of Energy Kernel speedup = time to execute the doubly nested for loop of the compute() function (without cost of data transfer) Procedure speedup = time for the compute() function to execute, including data transfer costs in the GPU setup Overall speedup = run-time (CPU only) / run-time (CPU + GPU) baileybaby222: Theoretical limit (computations only) Off-loading non-bonded (data transfer included) Entire simulation Performance modeling

20 20Managed by UT-Battelle for the U.S. Department of Energy Long range electrostatics Gain from multi-level hierarchy Matching the hardware features with software requirements On GPUs – Lennard-Jones and short range terms (direct space) – Less communication and more computationally dense Long range electrostatics – Particle mesh Ewald (PME) – Requires fast Fourier transforms (FFT) – Significant communication – Keep it on the multi-core

21 21Managed by UT-Battelle for the U.S. Department of Energy Summary GPU-LAMMPS (open source) MD software –Highly scalable on HPC machines –Supports popular biomolecular force-fields Accelerated MD simulations –10-20X: single workstations & GPU clusters –Beats #1 Cray XT 5 supercomputer (small systems) –Production quality runs Wide-impact possible –High throughput: Large number of proteins –Longer time-scales Ongoing work: Heterogeneous architectures –Gain from the multi-level hierarchy –Best time-to-solution: use most (or all) resources

22 22Managed by UT-Battelle for the U.S. Department of Energy Acknowledgements Scott Hampton, Sadaf Alam (Swiss Supercomputing Center) Melissa Smith, Ananth Nallamuthu (Clemson) Duncan Poole/Peng Wang, NVIDIA Paul Crozier, Steve Plimpton, SNL Mike Brown, SNL/ORNL $$$ NIH: R21GM083946 (NIGMS) Questions?

Download ppt "1Managed by UT-Battelle for the U.S. Department of Energy Optimal Biomolecular simulations in heterogeneous computing architectures Scott Hampton, Pratul."

Similar presentations

Accelerators for HPC: Programming Models Accelerators for HPC: StreamIt on GPU High Performance Applications on Heterogeneous Windows Clusters

Accelerators for HPC: Programming Models Accelerators for HPC: StreamIt on GPU High Performance Applications on Heterogeneous Windows Clusters
The Development of Mellanox - NVIDIA GPUDirect over InfiniBand A New Model for GPU to GPU Communications Gilad Shainer.

The Development of Mellanox - NVIDIA GPUDirect over InfiniBand A New Model for GPU to GPU Communications Gilad Shainer.
Scalable Multi-Cache Simulation Using GPUs Michael Moeng Sangyeun Cho Rami Melhem University of Pittsburgh.

Scalable Multi-Cache Simulation Using GPUs Michael Moeng Sangyeun Cho Rami Melhem University of Pittsburgh.
Monte-Carlo method and Parallel computing  An introduction to GPU programming Mr. Fang-An Kuo, Dr. Matthew R. Smith NCHC Applied Scientific Computing.

Monte-Carlo method and Parallel computing  An introduction to GPU programming Mr. Fang-An Kuo, Dr. Matthew R. Smith NCHC Applied Scientific Computing.
1 A GPU Accelerated Storage System NetSysLab The University of British Columbia Abdullah Gharaibeh with: Samer Al-Kiswany Sathish Gopalakrishnan Matei.

1 A GPU Accelerated Storage System NetSysLab The University of British Columbia Abdullah Gharaibeh with: Samer Al-Kiswany Sathish Gopalakrishnan Matei.
ACCELERATING MATRIX LANGUAGES WITH THE CELL BROADBAND ENGINE Raymes Khoury The University of Sydney.

ACCELERATING MATRIX LANGUAGES WITH THE CELL BROADBAND ENGINE Raymes Khoury The University of Sydney.
GPU System Architecture Alan Gray EPCC The University of Edinburgh.

GPU System Architecture Alan Gray EPCC The University of Edinburgh.
Early Linpack Performance Benchmarking on IPE Mole-8.5 Fermi GPU Cluster Xianyi Zhang 1),2) and Yunquan Zhang 1),3) 1) Laboratory of Parallel Software.

Early Linpack Performance Benchmarking on IPE Mole-8.5 Fermi GPU Cluster Xianyi Zhang 1),2) and Yunquan Zhang 1),3) 1) Laboratory of Parallel Software.
HPCC Mid-Morning Break High Performance Computing on a GPU cluster Dirk Colbry, Ph.D. Research Specialist Institute for Cyber Enabled Discovery.

HPCC Mid-Morning Break High Performance Computing on a GPU cluster Dirk Colbry, Ph.D. Research Specialist Institute for Cyber Enabled Discovery.
Prof. Srinidhi Varadarajan Director Center for High-End Computing Systems.

Prof. Srinidhi Varadarajan Director Center for High-End Computing Systems.
May 8th, 2012 Higher Ed & Research. Molecular Dynamics Applications Overview AMBER NAMD GROMACS LAMMPS Sections Included * * In fullscreen mode, click.

May 8th, 2012 Higher Ed & Research. Molecular Dynamics Applications Overview AMBER NAMD GROMACS LAMMPS Sections Included * * In fullscreen mode, click.
Implementation methodology for Emerging Reconfigurable Systems With minimum optimization an appreciable speedup of 3x is achievable for this program with.

Implementation methodology for Emerging Reconfigurable Systems With minimum optimization an appreciable speedup of 3x is achievable for this program with.
Cyberinfrastructure for Scalable and High Performance Geospatial Computation Xuan Shi Graduate assistants supported by the CyberGIS grant Fei Ye (2011)

Cyberinfrastructure for Scalable and High Performance Geospatial Computation Xuan Shi Graduate assistants supported by the CyberGIS grant Fei Ye (2011)
OpenFOAM on a GPU-based Heterogeneous Cluster

OpenFOAM on a GPU-based Heterogeneous Cluster
Presented by Next Generation Simulations in Biology: Investigating biomolecular structure, dynamics and function through multiscale modeling Pratul K.

Presented by Next Generation Simulations in Biology: Investigating biomolecular structure, dynamics and function through multiscale modeling Pratul K.
Presented by Performance and Productivity of Emerging Architectures Jeremy Meredith Sadaf Alam Jeffrey Vetter Future Technologies.

Presented by Performance and Productivity of Emerging Architectures Jeremy Meredith Sadaf Alam Jeffrey Vetter Future Technologies.
A many-core GPU architecture.. Price, performance, and evolution.

A many-core GPU architecture.. Price, performance, and evolution.
Scalable Data Clustering with GPUs Student: Andrew D. Pangborn 1 Advisors: Dr. Muhammad Shaaban 1, Dr. Gregor von Laszewski 2, Dr. James Cavenaugh 3, Dr.

Scalable Data Clustering with GPUs Student: Andrew D. Pangborn 1 Advisors: Dr. Muhammad Shaaban 1, Dr. Gregor von Laszewski 2, Dr. James Cavenaugh 3, Dr.
Evaluation of Fast Electrostatics Algorithms Alice N. Ko and Jesús A. Izaguirre with Thierry Matthey Department of Computer Science and Engineering University.

Evaluation of Fast Electrostatics Algorithms Alice N. Ko and Jesús A. Izaguirre with Thierry Matthey Department of Computer Science and Engineering University.
A Performance and Energy Comparison of FPGAs, GPUs, and Multicores for Sliding-Window Applications From J. Fowers, G. Brown, P. Cooke, and G. Stitt, University.

A Performance and Energy Comparison of FPGAs, GPUs, and Multicores for Sliding-Window Applications From J. Fowers, G. Brown, P. Cooke, and G. Stitt, University.

Similar presentations

Ads by Google