1. HPCC Mid-Morning Break Dirk Colbry, Ph.D. Research Specialist Institute for Cyber Enabled Discovery Introduction to the new GPU (GFX) cluster

2. Naming dev - developer node. gfx - graphics node. GPU - Graphics Processing Unit.  I use this term interchangeably with gfx nvx - nVidia Graphics node.  Internal prefix used on the new cluster.

3. HPCC System Diagram

4. System Diagram of GFX dev-gfx08 - (formerly gfx-000) dev-gfx10 – New developer node gfx10 – New 32 node GPU cluster

5. dev-gfx08 (formerly gfx-000) Single Quad core 2.4 Ghz Intel Processor. 8GB of CPU RAM Three nVidia GTX 280 Video cards:  1GB of ram per card  240 CUDA processing Cores per card  1.3 GHz Processor Clock Speed Total of 724 cores on a single machine

6. New gfx10 hardware 32 Node cluster + 1 developer node  CPU: 2x Intel Xeon E5530 Quad-Core 2.40GHZ  Memory: 18GB of Ram (~2GB per core)  Hard drive: 200GB disk for Local Scratch  Network: Ethernet only, (no Infiniband)  GPU: Two Nvidia Tesla M1060 GPUs Note: 21 of these nodes are buy-in nodes and set aside for primary use by the owners.

7. Each nVidia Tesla M1060 Number of Streaming Processor Cores 240 Frequency of processor cores 1.3 GHz Single Precision peak floating point performance 933 gigaflops Double Precision peak floating point performance 78 gigaflops Dedicated Memory 4 GB GDDR3 Memory Speed 800 MHz Memory Interface 512-bit Memory Bandwidth 102 GB/sec System Interface PCIe

8. Installed Software on HPCC Cuda toolkit 2.2 and 2.3  For programming in c/c++ and fortran PGI compilers with graphics accelerator support. cublas – Cuda version of blas libraries cufft – Cuda version of fft libraries pycuda – Python Cuda Interface Openmm/gromacs – Molecular Dynamics Program optimized for GPUs

9. Other Available Software OpenCL  c/c++ interface Jacket  Matlab GPU wrapper Lattice Boltzmann  pde solver OpenVIDIA  Machine Vision Many Many others Cuda Zone  ~90 thousand cuda developers.  Lots of software examples  Developer Forms  Tutorials http://www.nvidia.com/o bject/cuda_home.htmlhttp://www.nvidia.com/o bject/cuda_home.html

10. Running on the new cluster We are still researching the best way to run on the GPU cluster. Looking for Beta Tests:  Probably add the gfx10 attribute to your submission script.  Please sign up and we will add you to a mailing list and give you access to the gfx10 queue.

11. Example script #!/bin/bash –login #PBS –l nodes=1:ppn=1:gfx10,walltime=01:00:00 module load cuda myprogram myarguments

12. What is a GPU? Graphics Processing Unit Originally designed to make Video Games Uses many processing cores to parallelize the math required for real time game play. Early researchers made general programs that looked like graphics so they could run in the GPU. In 2006 nVidia released the CUDA programming interface to allow users to easily make scalable general purpose programs that run on the GPU (GPGPU).

13. GPU vs CPU

14. CPU and GPU working together

15. Running on the GPU Program Starts on the CPU  Copy data to GPU (slow-ish)  Run kernel threads on GPU (very fast)  Copy results back to CPU (slow-ish) There are a lot of clever ways to fully utilize both the GPU and CPU.

16. Pros and Cons Benefits  Lots of processing cores.  Works with the CPU as a co-processor  Very fast local memory bandwidth  Large online community of developers Drawbacks  Can be difficult to program.  Memory Transfers between GPU and CPU are costly (time).  Cores typically run the same code.