1. NREL is a national laboratory of the U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy, operated by the Alliance for Sustainable Energy, LLC. Peregrine Applications Q2 Peregrine New User Training User Software Chris Chang 6/3/2015

2. 2 Outline User software on Peregrine Configuring runtime environments Software stack (user perspective) Development tools and process Xeon Phi and the Future of Performance Computing Interactive Computing: Jupyter

3. 3 What Do We Mean by “User Software”? Not system software o NOT core OS functions o Toolchains—compilers + MPI o Libraries, etc.—mathematical, scientific, language- specific (e.g., Guile) o Applications—source and binary installations

4. 4 Environment Modules Wiring up dependencies manually = pain System to automate o Adjusting paths o Auto-loading dependencies, or detecting incompatibilities o Choosing versions module [ avail | list | purge | load Y | unload X | swap X Y ] (X loaded; Y not loaded) module show X; module help X Mind the priority of loading modules 1. Explicit naming: impi-intel/4.1.1-13.1.3 2. $MODULEPATH 3. (default)

5. 5 Maintained Modules /nopt/nrel/apps/modules/ candidate  default  deprecated candidate: testing & verification, MAY NOT WORK default: production, WILL WORK deprecated: phasing out, MAY NOT WORK Watch peregrine-users list for announcements

6. 6 Modules Can Go Anywhere /projects Every allocation gets one. /nopt/nrel/ecom/modules Of, by, for the Energy Community $HOME Make your own!

7. 7 Building a Software Stack Extra Packages for Enterprise Linux (EPEL) o “module load epel” o Package listing via rpm—see “module help epel” Ask for additions via HPC-Help core OS EPEL toolchain libraries, frameworks applications

8. 8 Toolchains Dependencies between binary application and binary library code Communication (MPI) particularly problematic “toolchain” = compiler + MPI Three primary stacks supported o “Performance”: impi-intel—when parallel performance is a (not the) concern. Intel Parallel Studio XE Cluster Edition o “Open”: openmpi-gcc—good ‘nuff. Open source, large community o “Scalable”: mvapich-intel—intrepid only

9. 9 Libraries, Frameworks, Suites, Toolkits, Environments,... Modeling Frameworks OpenFOAM, FEniCS, GAMS Solvers/Optimizers PETSc, Dakota, Gurobi Math MKL, BLAS, LAPACK, FFTW, GSL I/O HDF5, netCDF, pnetCDF Language Frameworks Python, R, Boost, MATLAB Between toolchains and applications

10. 10 Applications InputOutput Black box transformer of inputs+configuration to outputs, or Self-contained environment

11. 11 Application Areas Development —debuggers, profilers Visualization & Analysis —Avizo, VisIt, VirtualGL, IDL, Paraview, POV- Ray A wide variety of domain-specific apps, including Quantum chemistry—Gaussian, GAMESS, NWChem, VASP, Q-Chem Molecular Dynamics—CHARMM, AMBER, GROMACS, LAMMPS, NAMD Continuum physics—ANSYS-Fluent, Star-CCM, COMSOL, MEEP, WRF Some applications are more supported than others /nopt/nrel/appsbroad anticipated or actual interest /nopt/nrel/ecomnarrower interest within the “Energy community” (i.e., Peregrine users) /projectsallocation-specific interest $HOMEindividual interest

12. 12 Development—Building Applications autotools, CMake, git, SCons Runs are done on the compute nodes! o What you pick up on the head nodes stays on the head nodes o Build software on the compute nodes Environment is set by modules o Remember to load/unload/purge toolchain & library modules before builds and runs http://hpc.nrel.gov/users/softwar e/developer-notes http://hpc.nrel.gov/users/softwar e/developer-notes Head -dep1.3.3 -dep2 Compute -dep1.3.2 (Build) (Run) static link shared object

13. 13 Development—Performance MPI o Must, if huge memory requirements, multiple nodes needed o Best, if clear problem divisions, low communication o MPI typically accessed through wrappers—mpicc, etc. Threading (OpenMP) o Best if separate processes almost fit into memory; single problem of lightweight tasks; high communication o Threading via compiler options— -qopenmp, -fopenmp

14. 14 Development—Profiling & Debugging When things go wrong (quickly or slowly) GDBDebugger Intel VTuneProfiler Intel Trace Analyzer/CollectorProfiler DDT (Allinea)Debugger MAP (Allinea)Profiler perf-report (Allinea)Profiler Totalview (Rogue Wave)Debugger TauProfiler PAPIProfiling Primarily single-process, multiple threads Primarily multiple-process, single thread Both multiple processes and multiple threads

15. 15 Development—Scripting Probably most of what you’ll do Easily tuned to domain Most support available for Python o Batteries included o Performance where needed

16. 16 Scripting Tools on Peregrine Shells: bash, csh, dash, ksh, ksh93, sh, tcsh, zsh, tclsh Perl, Octave, Lua, Lisp(Emacs, Guile), GNUplot, Tclx, Java, Ruby, Postgres, SQLite IDL, Python, R, Ruby,.NET (Mono) SQL (Postgres, Sqlite, MySQL) MATLAB, GAMS