1. Parallel Programming Workshop HPC 470 August, 2015

2. Credits Contributors: Dr Charles Antonelli (LSA IT) Mark Champe (LSA IT) Bennet Fauber (ARC) Dr Alexander Gaenko (ARC) Nancy Herlocher (LSA IT) Seth Meyer (LSA IT) Todd Raeker (ARC) Brought to you under the auspices of Advanced Research Computing, U-M Office of Research Advanced Research ComputingU-M Office of Research Advanced Research ComputingU-M Office of Research 10 Aug 2015LSA IT ARS / cja © 20152

3. Roadmap http://arc-ts.umich.edu/hpc470/

4. Roadmap

5. Copying data 10 Aug 2015LSA IT ARS / cja © 20155

6. Copying data F rom Linux or Mac OS X, use scp or sftp Non-interactive (scp) scp localfile uniqname@flux-xfer.engin.umich.edu:remotefile scp -r localdir uniqname@flux-xfer.engin.umich.edu:remotedir scp uniqname@flux-login.engin.umich.edu:remotefile localfile Use "." as destination to copy to your Flux home directory: scp localfile login@flux-xfer.engin.umich.edu:.... or to your Flux scratch directory: scp localfile login@flux-xfer.engin.umich.edu:/scratch/allocname/uniqname Interactive (sftp) sftp uniqname@flux-xfer.engin.umich.edu From Windows, use WinSCP U-M Blue Disc: http://www.itcs.umich.edu/bluedisc/ http://www.itcs.umich.edu/bluedisc/ 05/156cja 2015

7. Globus Online Features High-speed data transfer, much faster than SCP or SFTP Reliable & persistent Minimal client software: Mac OS X, Linux, Windows GridFTP Endpoints Gateways through which data flow Exist for XSEDE, OSG, … UMich: umich#flux, umich#nyx Add your own client endpoint! Add your own server endpoint: contact flux-support@umich.edu flux-support@umich.eduflux-support@umich.edu More information http://arc.umich.edu/flux-and-other-hpc-resources/flux/using- flux/transferring-files-with-globus-gridftp/ http://arc.umich.edu/flux-and-other-hpc-resources/flux/using- flux/transferring-files-with-globus-gridftp/ 05/15cja 20157

8. Parallelism Review 10 Aug 2015LSA IT ARS / cja © 20158

9. Compute Node Processor RAM Local disk 5/15ES15 P Process 9

10. Fine-grained parallelism Cores RAM Local disk 5/15ES15 P 10

11. Fine-grained parallelism Cores RAM Local disk 5/15ES15 P 11

12. Programming Models (1) Fine-grained parallelism The parallel application consists of a single process containing several parallel threads that communicate with each other using synchronization primitives Used when the data can fit into a single process, and the communications overhead of the message-passing model is intolerable “Symmetric multiprocessing (SMP)” or “Shared-memory parallelism” or “multi-threaded parallelism” or … Implemented using compilers and software libraries OpenMP (Open Multi-Processing) 5/15ES1512

13. Coarse-grained parallelism 5/15ES1513

14. Programming Models (2) Coarse-grained parallelism The parallel application consists of several processes running on different nodes and communicating with each other over the network Used when the data are too large to fit on a single node, and simple synchronization is adequate “Message-passing” Implemented using software libraries MPI (Message Passing Interface) 5/15ES1514

15. Good parallel Embarrassingly parallel Folding@home, RSA Challenges, Bitcoin mining, password cracking, … http://en.wikipedia.org/wiki/List_of_distributed_co mputing_projects http://en.wikipedia.org/wiki/List_of_distributed_co mputing_projects 5/15ES1515

16. Amdahl’s Law If you enhance a fraction f of a computation by a speedup S, the overall speedup is: 5/15ES1516

17. Amdahl’s Law 5/15ES1517

18. MPI 10 Aug 2015LSA IT ARS / cja © 201518

19. Crib sheet Login to Stampede ssh trainNNN@stampede.tacc.utexas.edu Get and compile the code cp -r ~cja/hpc470/jacobi. cd hpc470/jacobi make Get on a compute node idev –m 30 Run the vanilla version cd hpc470/jacobi mpirun -bootstrap fork –np N./oned Excellent user guide https://portal.tacc.utexas.edu/user-guides/stampede 19 10 Aug 2015LSA IT ARS / cja © 201519

20. Debugging & profiling 10 Aug 2015LSA IT ARS / cja © 201520

21. Debugging with GDB Command-line debugger Start programs or attach to running programs Display source program lines Display and change variables or memory Plant breakpoints, watchpoints Examine stack frames Excellent tutorial documentation http://www.gnu.org/s/gdb/documentation/ 21 10 Aug 2015LSA IT ARS / cja © 201521

22. Compiling for GDB Debugging is easier if you ask the compiler to generate extra source-level debugging information Add -g flag to your compilation icc -g serialprogram.c -o serialprogram or mpicc -g mpiprogram.c -o mpiprogram GDB will work without symbols Need to be fluent in machine instructions and hexadecimal Be careful using -O with -g Some compilers won't optimize code when debugging Most will, but you sometimes won't recognize the resulting source code at optimization level -O2 and higher Use -O0 -g to suppress optimization 22 10 Aug 2015LSA IT ARS / cja © 201522

23. Running GDB Two ways to invoke GDB: Debugging a serial program: gdb./serialprogram Debugging an MPI program: mpirun -np N xterm -e gdb./mpiprogram This gives you N separate GDB sessions, each debugging one rank of the program Remember to use the -X or -Y option to ssh when connecting to Flux, or you can't start xterms there 10 Aug 2015LSA IT ARS / cja © 201523

24. Useful GDB commands gdb execstart gdb on executable exec gdb exec corestart gdb on executable exec with core file core l [m,n]list source disasdisassemble function enclosing current instruction disas funcdisassemble function func b funcset breakpoint at entry to func b line#set breakpoint at source line# b *0xaddrset breakpoint at address addr i bshow breakpoints d bp#delete beakpoint bp# r [args]run program with optional args btshow stack backtrace ccontinue execution from breakpoint stepsingle-step one source line nextsingle-step, don't step into function stepisingle-step one instruction p vardisplay contents of variable var p *vardisplay value pointed to by var p &vardisplay address of var p arr[idx]display element idx of array arr x 0xaddrdisplay hex word at addr x *0xaddrdisplay hex word pointed to by addr x/20x 0xaddrdisplay 20 words in hex starting at addr i rdisplay registers i r ebpdisplay register ebp set var = expressionset variable var to expression qquit gdb 10 Aug 2015LSA IT ARS / cja © 201524

25. Debugging with DDT Allinea's Distributed Debugging Tool is a comprehensive graphical debugger designed for the complex task of debugging parallel code Advantages include Provides GUI interface to debugging Similar capabilities as, e.g., Eclipse or Visual Studio Supports parallel debugging of MPI programs Scales much better than GDB 10 Aug 2015LSA IT ARS / cja © 201525

26. Running DDT Compile with -g: mpicc -g mpiprogram.c -o mpiprogram Load the DDT module: module load ddt Start DDT: ddt mpiprogram This starts a DDT session, debugging all ranks concurrently Remember to use the -X or -Y option to ssh when connecting to Flux, or you can't start ddt there http://arc-ts.umich.edu/software/ http://content.allinea.com/downloads/userguide.pdf 10 Aug 2015LSA IT ARS / cja © 201526

27. Application Profiling with MAP Allinea's MAP Tool is a statistical application profiler designed for the complex task of profiling parallel code Advantages include Provides GUI interface to profiling Observe cumulative results, drill down for details Supports parallel profiling of MPI programs Handles most of the details under the covers 10 Aug 2015LSA IT ARS / cja © 201527

28. Running MAP Compile with -g: mpicc -g mpiprogram.c -o mpiprogram Load the MAP module: module load ddt Start MAP: map mpiprogram This starts a MAP session Runs your program, gathers profile data, displays summary statistics Remember to use the -X or -Y option to ssh when connecting to Flux, or you can't start ddt there http://content.allinea.com/downloads/userguide.pdf 10 Aug 2015LSA IT ARS / cja © 201528

29. Resources http://arc-ts.umich.edu/flux/ U-M Advanced Research Computing Flux pages http://arc.research.umich.edu/software/ Flux Software Catalog http://arc-ts.umich.edu/flux/flux-faqs/ Flux FAQs http://www.youtube.com/user/UMCoECAC ARC-TS YouTube channel For assistance: flux-support@umich.edu flux-support@umich.edu Read by a team of people including unit support staff Cannot help with programming questions, but can help with operational Flux and basic usage questions 10 Aug 2015LSA IT ARS / cja © 201529

30. References 1.Supported Flux software, http://arc-ts.umich.edu/software/, (accessed May 2015) http://arc-ts.umich.edu/software/ 2.Free Software Foundation, Inc., "GDB User Manual," http://www.gnu.org/s/gdb/documentation/ (accessed May 2015). http://www.gnu.org/s/gdb/documentation/ 3.Intel C and C++ Compiler 14 User and Reference Guide, https://software.intel.com/en- us/compiler_15.0_ug_c (accessed May 2015). https://software.intel.com/en- us/compiler_15.0_ug_chttps://software.intel.com/en- us/compiler_15.0_ug_c 4.Intel Fortran Compiler 14 User and Reference Guide,https://software.intel.com/en- us/compiler_15.0_ug_f(accessed May 2015). https://software.intel.com/en- us/compiler_15.0_ug_fhttps://software.intel.com/en- us/compiler_15.0_ug_f 5.Torque Administrator's Guide, http://www.adaptivecomputing.com/resources/docs/torque/5-1- 0/torqueAdminGuide-5.1.0.pdf (accessed May 2015). http://www.adaptivecomputing.com/resources/docs/torque/5-1- 0/torqueAdminGuide-5.1.0.pdfhttp://www.adaptivecomputing.com/resources/docs/torque/5-1- 0/torqueAdminGuide-5.1.0.pdf 6.Submitting GPGPU Jobs, https://sites.google.com/a/umich.edu/engin- cac/resources/systems/flux/gpgpus (accessed May 2015). https://sites.google.com/a/umich.edu/engin- cac/resources/systems/flux/gpgpushttps://sites.google.com/a/umich.edu/engin- cac/resources/systems/flux/gpgpus 7.http://content.allinea.com/downloads/userguide.pdf (accessed May 2015) http://content.allinea.com/downloads/userguide.pdf 10 Aug 2015LSA IT ARS / cja © 201530