Presentation is loading. Please wait.

Presentation is loading. Please wait.

Universität Karlsruhe (TH) Rechenzentrum How to use the System SSCK Workshop – Introduction to HP XC6000 Cluster Karlsruhe, March 9 – 11, 2005 Hartmut.

Similar presentations


Presentation on theme: "Universität Karlsruhe (TH) Rechenzentrum How to use the System SSCK Workshop – Introduction to HP XC6000 Cluster Karlsruhe, March 9 – 11, 2005 Hartmut."— Presentation transcript:

1 Universität Karlsruhe (TH) Rechenzentrum How to use the System SSCK Workshop – Introduction to HP XC6000 Cluster Karlsruhe, March 9 – 11, 2005 Hartmut Häfner SSCK Universität Karlsruhe (TH)

2 SSCK Workshop, Karlsruhe, March 9, 2005 page 2 Universität Karlsruhe (TH) Rechenzentrum Interactive Login

3 SSCK Workshop, Karlsruhe, March 9, 2005 page 3 Universität Karlsruhe (TH) Rechenzentrum Available Services (1/2) HWW-Firewall XC1 ssh (scp) passive ftp » No print manager » No exported file system

4 SSCK Workshop, Karlsruhe, March 9, 2005 page 4 Universität Karlsruhe (TH) Rechenzentrum Available Services (2/2) » Login to HP XC6000 Cluster » or within University Karlsruhe » SSH2 from RZ administrated workstations ssh2 –p

5 SSCK Workshop, Karlsruhe, March 9, 2005 page 5 Universität Karlsruhe (TH) Rechenzentrum File Systems (1/2) 10 TB Quadrics QsNet II (single rail) 2x 16x 2x... 16x 2x FC Network 2x... $TMP $HOME$WORK

6 SSCK Workshop, Karlsruhe, March 9, 2005 page 6 Universität Karlsruhe (TH) Rechenzentrum File Systems (2/2) environment variable global/localpermanent/ temporary quotasbackup $HOMEglobalpermanentno, but monitoredyes $WORKglobalone weekno $TMPlocaltemporaryno » global - all nodes access the parallel file system HP SFS, based on Lustre » local – each node has ist own file system » permanent – files are stored permanently » temporary – files are removed at end of job or session

7 SSCK Workshop, Karlsruhe, March 9, 2005 page 7 Universität Karlsruhe (TH) Rechenzentrum Moving Files (HP XC Workstations) » Either by the command scp or by passive ftp $HOME ftp ws.institute.uni-karlsruhe.de

8 SSCK Workshop, Karlsruhe, March 9, 2005 page 8 Universität Karlsruhe (TH) Rechenzentrum Module Concept » module is a user interface to the Modules package. » Typically modulefiles instruct the module command to set or alter environment variables like PATH, MANPATH, …. » Syntax is: module [switches] [sub-command] [modulefile…|path…|directory…] » Important switches are: –--force, -f Force active dependency resolution. This will result in modules found on a prereq command inside a modulefile being loaded automatically. –--verbose, -v Enable verbose messages during module comand execution. Further switches control the amount of output of the module command.

9 SSCK Workshop, Karlsruhe, March 9, 2005 page 9 Universität Karlsruhe (TH) Rechenzentrum Modules (1/2) » module help [modulefile...] Print the useage of each subcommand. If an argument is given, print the Module specific help information for the modulefile. » module add|load modulefile [modulefile...] Load modulefile into the shell environment. » module unload|rm modulefile [modulefile...] Remove modulefile from the shell environment. » module switch|swap modulefile1 modulefile2 Switch loaded modulefile1 with modulefile2. » module display|switch modulefile [modulefile...] Display information about the modulefile. » module list List loaded modules. » module avail [path...] List all available modulefiles in the current MODULEPATH. » module purge Unload all loaded modulefiles. Further commands to add directories to MODULEPATH and to add|remove modulefiles to|from the shell dependent startup files.

10 SSCK Workshop, Karlsruhe, March 9, 2005 page 10 Universität Karlsruhe (TH) Rechenzentrum Modules (2/2) ModulefileDescription dotadds the current directory to your env. Variable PATH intel-compilers/7.1loads Intel Fortran and C/C++ compiler in version 7.1 intel-compilersloads Intel Fortran and C/C++ compiler in version 8.1 nag-compilersloads NAG Fortran90/95 compiler in version 4.2 hp-mpiloads HP MPI in version 2.0 intel-debuggersloads Intel debugger in version 8.0 ddtloads graphical Streamline debugger in version 1.8 mklloads Intel MKL library in version 7.2 mlib/7.1loads HP MLIB library in version 7.1 mlibloads HP MLIB library in version 8.0 naglibloads NAG Fortran library in version 8.0

11 SSCK Workshop, Karlsruhe, March 9, 2005 page 11 Universität Karlsruhe (TH) Rechenzentrum Modulefiles – containing modifications to the environment » modulefile is a file containing Tcl code + extensions for the Modules package. » modulefile contains the changes to a users environment needed to access an application. » modulefiles can also be used to implement site policies regarding the access and use of applications. » modulefiles also hide the notion of different types of shells. From the modulefile writers perspective, this means one set of information will take care of every type of shell. » Change default module environment by inserting module add in the setup file.bash_profile. » Add your own Modulefiles by extending the $MODULEPATH environment variable.

12 SSCK Workshop, Karlsruhe, March 9, 2005 page 12 Universität Karlsruhe (TH) Rechenzentrum Compilers (1/4) » Fortran: 2 Intel Compilers (ifort in V8.1 and efc in V7.1), NAG Compiler (f95), GNU Compiler (g77 - only Fortran77) » C/C++: 2 Intel Compilers (icc in V8.1 and ecc in V7.1), GNU Compiler (gcc) -- General options: -c, -I, -g, -0{0,1,2,3}, -L, -l, -o » NAG Fortran Compiler - best choice to check the Fortran90/95 conformity of your program » Important specific options of the NAG Fortran Compiler –-Ounsafe performs possibly unsafe optimizations –-dusty allows the compilation of legacy software (errors warning) –-ieee=full|nonstd|stop enables|disables all IEEE and deallocation facilities –-C compiles code with all possible runtime checks –-mtrace traces memory allocation and deallocation –-gline compiles code to generate a traceback in case of runtime errors –-gc enables automatic garbage collection of the executable –-tread_safe compiles code for safe execution in a multi-threaded environment –-static prevents linking with shared libraries

13 SSCK Workshop, Karlsruhe, March 9, 2005 page 13 Universität Karlsruhe (TH) Rechenzentrum Compilers (2/4) » Intel Fortran suffix names » NAG Fortran suffix names CommandFile name suffixSource formatLanguage level ifort.f.ftn.for.i-fixed -72-nostand ifort.F.FTN.FOR.fpp.FPP-fixed -72 -fpp-nostand ifort.f90.i90-free-nostand ifort.F90-free -fpp-nostand CommandFile name suffixSource formatLanguage level f95.f.ftn.for-fixed Fortran95 Standard; -strict95 for strict Fortran95 code; -dusty for legacy code. f95.F-fixed -fpp f95.f90.f95-free f95.F90.F95-free -fpp

14 SSCK Workshop, Karlsruhe, March 9, 2005 page 14 Universität Karlsruhe (TH) Rechenzentrum Compilers (3/4) » Change compiler by a simple module command (by default the Intel compiler in version 8.1 is used) : module add|load intel-compilers/7.1 » Using different compilers –don´t use explicit compiler names –use the $FC environment variable for the Fortran compiler –use the $CC environment variable for the C/C++ compiler name

15 SSCK Workshop, Karlsruhe, March 9, 2005 page 15 Universität Karlsruhe (TH) Rechenzentrum Compilers (4/4) » Compiling Fortran90/95 source code with Intel compiler ifort –c –O3 my_prog.f90 » Compiling Fortran90/95 source code with an arbitrary Fortran compiler $FC –c –O3 my_prog.f90 » Compiling C source code with Intel compiler icc –c –O3 my_prog.c » Compiling C++ source code with Intel compiler $CC –c –O3 my_prog.C

16 SSCK Workshop, Karlsruhe, March 9, 2005 page 16 Universität Karlsruhe (TH) Rechenzentrum Linking » Special compiler scripts to (compile and) link MPI programs (the scripts don´t work together with the GNU compilers) mpicc – (compile and) link C programs mpicc.mpich – (compile and) link C programs in MPICH compatibility mode mpiCC – (compile and) link C++ programs mpiCC.mpich – (compile and) link C++ programs in MPICH compatibility mode mpif77 or mpif90 – (compile and) link Fortran programs If MPICH compatibility mode is required, call mpif77.mpich or mpif90.mpich » Example for Fortran90/95 object code with Intel compiler mpif90 –o my_prog my_prog.o sub1.o sub2.o

17 SSCK Workshop, Karlsruhe, March 9, 2005 page 17 Universität Karlsruhe (TH) Rechenzentrum Benchmarks Measurements of Itanium2 (1.5 GHz) on HP XC6000 Cluster Vector- length Min/MaxAdditionMult.DivisionLinked Triad Vector Triad Dot Product 1Min Max Min Max Min Max Min Max Min Max Min Max Min Max Min Max Peak Max. L2-c Max. L3-c Max. mem η a,L η a,L η a,mem What is remarkable? The dot product runs very slow! The scattering of the performance rates, if the data are stored in the L2- cache is very high (up to 40 percent!!!).

18 SSCK Workshop, Karlsruhe, March 9, 2005 page 18 Universität Karlsruhe (TH) Rechenzentrum Benchmarks – Ping Pong within a node Neighbor send/receive speed test Multiple simple Ping/Pong Clock overhead is E-07 secs per snd/rcv. bytes ms MB/s Neighbor send/receive speed test Multiple double Ping/Pong Clock overhead is E-08 secs per snd/rcv. bytes ms MB/s

19 SSCK Workshop, Karlsruhe, March 9, 2005 page 19 Universität Karlsruhe (TH) Rechenzentrum Benchmarks – Ping Pong between nodes Neighbor send/receive speed test Multiple simple Ping/Pong Clock overhead is E-07 secs per snd/rcv. bytes ms MB/s Neighbor send/receive speed test Multiple double Ping/Pong Clock overhead is E-08 secs per snd/rcv. bytes ms MB/s

20 SSCK Workshop, Karlsruhe, March 9, 2005 page 20 Universität Karlsruhe (TH) Rechenzentrum Benchmarks – Overlap for short messages between nodes Neighbor send/receive overlap test Short messages The used message length during computation is the used vectorlength during computation is all times in seconds, >>ol_fac in percent!!! Bal_fac Rep_fac_comm Rep_fac_comp T_comm T_comp T_all T_ol ol_fac The used message length during computation is the used vectorlength during computation is all times in seconds, >>ol_fac in percent!!! Bal_fac Rep_fac_comm Rep_fac_comp T_comm T_comp T_all T_ol ol_fac The used message length during computation is the used vectorlength during computation is all times in seconds, >>ol_fac in percent!!! Bal_fac Rep_fac_comm Rep_fac_comp T_comm T_comp T_all T_ol ol_fac

21 SSCK Workshop, Karlsruhe, March 9, 2005 page 21 Universität Karlsruhe (TH) Rechenzentrum Benchmarks – Overlap for long messages between nodes Neighbor send/receive overlap test Long messages The used message length during computation is the used vectorlength during computation is all times in seconds, >>ol_fac in percent!!! Bal_fac Rep_fac_comm Rep_fac_comp T_comm T_comp T_all T_ol ol_fac The used message length during computation is the used vectorlength during computation is all times in seconds, >>ol_fac in percent!!! Bal_fac Rep_fac_comm Rep_fac_comp T_comm T_comp T_all T_ol ol_fac The used message length during computation is the used vectorlength during computation is all times in seconds, >>ol_fac in percent!!! Bal_fac Rep_fac_comm Rep_fac_comp T_comm T_comp T_all T_ol ol_fac

22 SSCK Workshop, Karlsruhe, March 9, 2005 page 22 Universität Karlsruhe (TH) Rechenzentrum Debugging with DDT » Commands module add ddt ddt hello

23 SSCK Workshop, Karlsruhe, March 9, 2005 page 23 Universität Karlsruhe (TH) Rechenzentrum HP MPI – Execution of Parallel Programs » The syntax to start a parallel application interactively is mpirun [mpirun_options] or mpirun [mpirun_options] –f mpirun OptionsBrief Explanation -n # or -np # MPI job is run on # processors (option is ignored in batch mode) -m block or –m cycle MPI processes will be mapped blockwise or cyclically to the processors -T prints user and system time for each MPI rank -1sided enables one-sided communication -i enables runtime instrumentation profiling for all processes -stdio= specifies standard IO options (refer to HP MPI User Guide) -mpich runs the application in MPICH compatibility mode

24 SSCK Workshop, Karlsruhe, March 9, 2005 page 24 Universität Karlsruhe (TH) Rechenzentrum HP MPI – Environment Variables » Many environment variables HP MPI Env. VariablesBrief Explanation MPI_FLAGSmodifies the general behaviour of MPI – l reports memory leaks caused by not freeing memory – f forces MPI errors to be fatal, ignoring the programmer´s choice of error handlers – z enables zero-buffering mode (MPI_SEND and MPI_RSEND MPI_SSEND) MPI_INSTRenables counter instrumentation for profiling HP MPI applications MPIRUN_OPTIONS sets mpirun options......

25 SSCK Workshop, Karlsruhe, March 9, 2005 page 25 Universität Karlsruhe (TH) Rechenzentrum Numerical Libraries » HP XC Mathematical LIBrary (MLIB) » Intel Mathematical Kernel Library (MKL) » NAG Libraries (non-commercial users) » LINear SOLver package (LINSOL)

26 SSCK Workshop, Karlsruhe, March 9, 2005 page 26 Universität Karlsruhe (TH) Rechenzentrum Well Established Open Source Libraries » BLAS –BLAS{1,2,3} included in HP XC MLIB and Intel MKL » LAPACK –included in HP XC MLIB and Intel MKL contains many functions for the solution of linear systems and eihenvalue problems for dense and banded matrices » ScaLAPACK –included in HP XC MLIB contains above mentioned functions for parallel computers » Metis –included in HP XC MLIB contains a special implementation of the graph partitioning and matrix reordering library

27 SSCK Workshop, Karlsruhe, March 9, 2005 page 27 Universität Karlsruhe (TH) Rechenzentrum HP XC MLIB (1/2) » Functions from several areas: linear equations, least squares, eigenvalue problems, singular value decomposition, vector and matrix computations, convolutions and Fourier Transforms » Four components: VECLIB, LAPACK, ScaLAPACK and SuperLU_DIST » VECLIB includes all BLAS{1,2,3} and sparse BLAS subroutines, sparse linear equation solvers, sparse eigenvalue and eigenvector solvers, FFTs, correlation and convolution subprograms, random number generators and METIS V4.0.1 » Load bevor use module add hp-mlib/7.1 for Intel compiler V7.1 and module add hp-mlib for Intel compiler V8.1

28 SSCK Workshop, Karlsruhe, March 9, 2005 page 28 Universität Karlsruhe (TH) Rechenzentrum HP XC MLIB (2/2) » Appropriate options at link time: –VECLIB $FC –L$MLIBPATH –lveclib –openmp –o myprog myprog.f90 –LAPACK $FC –L$MLIBPATH –llapack –openmp –o myprog myprog.f90 –ScaLAPCK mpif90 –L$MLIBPATH –lscalapack –openmp –o myprog myprog.f90 –SuperLU_DIST mpif90 –L$MLIBPATH –lsuperlu_dist –openmp –o myprog myprog.f90 » More details:

29 SSCK Workshop, Karlsruhe, March 9, 2005 page 29 Universität Karlsruhe (TH) Rechenzentrum Intel MKL (1/2) » Many components: –BLAS, –Sparse BLAS, –LAPACK, –direct sparse solver PARDISO, –Vector Mathematical Library (VML) for core mathematical functions on vector arguments, –Vector Statistical Library (VSL) for generating vectors of pseudorandom numbers, –general Discrete Fourier Transform functions (DFT) and –a subset of FFTs » Load bevor use module add mkl

30 SSCK Workshop, Karlsruhe, March 9, 2005 page 30 Universität Karlsruhe (TH) Rechenzentrum Intel MKL (2/2) » Appropriate options at link time: –BLAS, FFT, VML, VSL etc. $FC –L$MKLPATH –lmkl_ipf –lguide –lpthread –o myprog myprog.f90 –LAPACK $FC –L$MKLPATH –lmkl_lapack –lmkl_ipf –lguide –lpthread –o myprog myprog.f90 –PARDISO mpif90 –L$MKLPATH –lmkl_solver –lmkl_ipf –lguide –lpthread –o myprog myprog.f90 » More details:

31 SSCK Workshop, Karlsruhe, March 9, 2005 page 31 Universität Karlsruhe (TH) Rechenzentrum NAG Libraries » NAG Fortran, NAG Fortran90 and NAG C libraries only for non-commercial customers » Load bevor use module add naglib/7.1 module add mkl/7.1 for Intel compiler V7.1 and module add naglib module add mkl for Intel compiler V8.1 » Appropriate options at compile and link time: –NAG Fortran Library $FC myprog.f90 –I$NAGLIBPATH/interface_blocks –LNAGLIBPATH \ –lnag-mkl –L$MKLPATH –lmkl_lapack –lmkl_ipf –lguide -lpthread –NAG Fortran90 Library $FC myprog.f90 –I$NAGLIBPATH/nag_mod_dir –LNAGLIBPATH \ –lnagfl90-noblas –L$MKLPATH –lmkl_lapack –lmkl_ipf –lguide -lpthread –NAG C Library $CC myprog.c –I$NAGLIBPATH/include –L$NAGLIBPATH/nagc » More details:

32 SSCK Workshop, Karlsruhe, March 9, 2005 page 32 Universität Karlsruhe (TH) Rechenzentrum LINSOL » LINSOL is a program package to solve large sparse linear systems –many iterative solvers –several polyalgorithms –(I)LU direct solvers as preconditioners –optimized for workstations (cache reuse), vectorcomputers and parallel computers (MPI) –supporting 7 different storage patterns for sparse matrices (automatic optimization to the architecture of the computer) » Load bevor use module add linsol » Appropriate options at compile and link time: mpif90 –L$LINSOLPATH –llinsol –lMPI myprog.o running a MPI job $ FC –L$LINSOLPATH –llinsol –lnocomm myprog.o running a serial job » More details:


Download ppt "Universität Karlsruhe (TH) Rechenzentrum How to use the System SSCK Workshop – Introduction to HP XC6000 Cluster Karlsruhe, March 9 – 11, 2005 Hartmut."

Similar presentations


Ads by Google