2. High-Performance Computing @ SERC Supercomputer Education & Research Centre Indian Institute of Science Bangalore Contact : helpdesk@serc.iisc.ernet.in

3. 23/08/2001 SERC Orientation Programme : HPC 2 Introduction to Scientific Computing

4. 23/08/2001 SERC Orientation Programme : HPC 3 Different Faces of Computing o Document Processing o Graphics & Imaging o Commercial & Database Applications o Process Engineering o Design o Network Computing o Scientific Computing (Number Crunching)

5. 23/08/2001 SERC Orientation Programme : HPC 4 Scientific Computing Features o Few weeks to few months of coding for a single developer (most cases) o Developer runs the code herself o Codes are typically less fault-tolerant, less cosmetic o Aims to extract maximum throughput from the system (hardware & software)

6. 23/08/2001 SERC Orientation Programme : HPC 5 What does it involve ? o Deals with numbers - integer, floating-point and complex as well o Involves complicated operations on a lot of variables and arrays o Often requires large resources (cpu, memory, disk) - to handle the complexity of nature

7. 23/08/2001 SERC Orientation Programme : HPC 6 Process Cycle o Developing the Algorithm o Developing the Code o Compiling and Debugging the code to make it error-free o Optimizing & Tuning the Code for efficiency o Production Runs & Result Analysis

8. 23/08/2001 SERC Orientation Programme : HPC 7 Things to Know About o The Science and the Algorithm o Basic Numerical Analysis. o Any High-level Language (Fortran, C, C++) o Packages and Libraries o Programming Environment : Compilers, Debuggers, Profilers, Development Tools.. o Optimization Techniques

9. 23/08/2001 SERC Orientation Programme : HPC 8 Pre-requisite : UNIX o Most systems at SERC run Unix. o Need an working knowledge on :  File and Directory manipulation commands ( ls, cp, mv, rm, cd, grep,..) m Common Desktop Environment (CDE)  Text editor (preferably vi or emacs ) to write your codes. o Advanced Unix concepts (eventually)

10. 23/08/2001 SERC Orientation Programme : HPC 9 Computing Environment @ SERC

11. 23/08/2001 SERC Orientation Programme : HPC 10 Compute Systems @ SERC o PC’s o Unix Access Stations o Servers o Supercomputers o Network

12. 23/08/2001 SERC Orientation Programme : HPC 11 Where to Start ? o Use the PCs only if you need packages that are available in Windows only o Otherwise, use an Unix Access Station. Build and test-run your codes here. o For Production Runs go to a Server.

13. 23/08/2001 SERC Orientation Programme : HPC 12 Unix Access Stations @ SERC o IBM RS/6000 340, PowerPC o SUN Ultra 5, Ultra 10, Ultra 60 o SGI Indy, Indigo2, O2, Octane o HP C200 o Compaq XP 1000 o Intel Pentium II & III o Apple PowerMac & iMac

14. 23/08/2001 SERC Orientation Programme : HPC 13 Using the Access Stations o For code development o Running small jobs (< 30 mins) in background. o Do not lock display o Migrate to a Server for production runs.

15. 23/08/2001 SERC Orientation Programme : HPC 14 Development Environment o Numerical Packages - MATLAB, MAPLE, MATHEMATICA o Fortran/C Development Environment : m Compilers, Debuggers & Profilers or m Integrated Environment (SoftBench, VisualAge, DECFuse) o Numerical Libraries.

16. 23/08/2001 SERC Orientation Programme : HPC 15 Numerical Packages o Interactive packages can do a variety of numerical and symbolic manipulation jobs. o Easy to use with GUI, on-line help and tutorials.

17. 23/08/2001 SERC Orientation Programme : HPC 16 MATLAB http://www.mathworks.com/ o Available in almost all Unix systems at SERC. o Useful for matrix manipulations, solving equations, differential eqns, 2D and 3D data visualization and...

18. 23/08/2001 SERC Orientation Programme : HPC 17 MATHEMATICA http://www.wolfram.com/ o Strong in Symbolic Manipulation o Can also do Numeric Computation and Visualization o Available at most systems on SERC o You can also install in your lab machine Mathematica Seminar @ SERC 12 September 2001

19. 23/08/2001 SERC Orientation Programme : HPC 18 Numerical Libraries Libraries @ SERC o ESSL (on IBM) o DXML (on Compaq) o IMSL (on IBM 59x) o MLIB (on HP) o PESSL (on IBM SP) o Free stuff such as Scalapack, Lapack, BLAS (on IBM SP) o Canned routines for standard tasks (matrix diagonalization, FFT …) o Tuned for specific hardware to extract maximum speed o Using the libraries saves coding time and often improves performance also

20. 23/08/2001 SERC Orientation Programme : HPC 19 Building an Application

21. 23/08/2001 SERC Orientation Programme : HPC 20 The Compiler o Compiler translates your programme to a machine-understandable code. o Fortran/C/C++ compilers are available in all Unix systems at SERC o Example : f77 my_prog.f -o my_prog creates an executable called my_prog from my_prog.f

22. 23/08/2001 SERC Orientation Programme : HPC 21 Compiler Flags (options) For specific needs :  -O, -O2, -O3 for optimization  -c for creating object files  -g for debugging  -l for including a library

23. 23/08/2001 SERC Orientation Programme : HPC 22  A perfectly compiled code may exit while running saying core dumped o Such run-time errors may be caused by various reasons, such as : m Divide by zero m Accessing an array element beyond its declared dimension m Number overflow or underflow Run-time Errors

24. 23/08/2001 SERC Orientation Programme : HPC 23 Debuggers o Debuggers will be the savior if you get run-time error on a complex code  Compile with -g flag. o Run the executable through debugger: dbx exe_filename  This will bring you to (dbx) prompt o Check system-specific manuals

25. 23/08/2001 SERC Orientation Programme : HPC 24 Alternative to Debuggers o For small codes, include some print statements in your code and run. o This may zero in the source of error. o For Fortran fanatics : m Check spelling of your variable names.  Put implicit none at the top line of your code, and explicitly declare all variables to avoid spelling mistakes.

26. 23/08/2001 SERC Orientation Programme : HPC 25 Timing your Code  Use the timex command to know how much CPU time your code takes. % timex a.out real 10.06 user 3.01 sys 0.03 The time command in the cshell gives a more detailed output.

27. 23/08/2001 SERC Orientation Programme : HPC 26 Profilers o The profilers will give you detailed time statistics of your code.  Compile with -p and run. o Post-Mortem profile with: prof > prof.dat  Complete profiler information is stored in prof.dat

28. 23/08/2001 SERC Orientation Programme : HPC 27 Alternative to Profilers o Instead of profilers, you can also manually time code segments. o E.g. in IBM systems : integer t1,t2,time_in_sec t1 = mclock() call subprog(x,y,z) t2 = mclock() time_in_sec = (t2-t1)/100

29. 23/08/2001 SERC Orientation Programme : HPC 28 An Extra Tool : gprof o If you have a large and complicated code with many subprograms you might require to know the call structures and profiles : gprof is tool which helps you do that. o Compile with –pg option and run the code as usual. o Ex. % cc –pg test.c % a.out % gprof a.out > test.gprof

30. 23/08/2001 SERC Orientation Programme : HPC 29 How big is the code? o The size of the code is an very important parameter. A code which fits into the memory runs faster as it avoids paging. o Too large a code also might result in insufficient memory errors. o size command gives the executable program size. % size a.out

31. 23/08/2001 SERC Orientation Programme : HPC 30 What to do in case of size problem? o Check for user limits using ulimit command. If not sufficient increase them to maximums. o Try running the code on a system with larger memory. o If everything fails you might have to reduce problem size.

32. 23/08/2001 SERC Orientation Programme : HPC 31 Thriving for Speed o Need to improve the performance of you code if you are going to run it again and again. o This can be achieved in two ways : m Through compiler flags for optimization m By modifying your code appropriately o Modify code to save resources (CPU time, memory, disk space …)

33. 23/08/2001 SERC Orientation Programme : HPC 32 Optimization through Compiler  Use -O[n] compiler flag for optimization; n signifies optimization level (different in each architecture) o Use architecture-specific tuning flags E.g. for IBM 59x Servers -qarch=pwr2 -qtune=pwr o High levels of optimization may sometime change the order of expression evaluation, and cause wrong results  Check the Compiler man pages for details on the optimization flags

34. 23/08/2001 SERC Orientation Programme : HPC 33 Optimization by Code Tuning A few General Rules : o Break up complex expressions into simpler sub-expressions o Pull out constant operations from a loop o Access array elements in proper order o Maintain Locality of Reference o Cut down redundant arrays

35. 23/08/2001 SERC Orientation Programme : HPC 34 Accessing Array Elements o Fortran stores arrays in column-wise, C stores in row-wise format. o Your access pattern should conform to this. o Access consecutive elements of an array within a close proximity in your code.

36. 23/08/2001 SERC Orientation Programme : HPC 35 Access : Fortran Example 6 Wrong Access : DO I = 1, N DO J = 1,N A(I,J)=2*A(I,J) ENDDO 4 Right Access : DO J = 1, N DO I = 1,N A(I,J)=2*A(I,J) ENDDO For a 4000 x 4000 array, the Right access pattern improves the speed by a factor of 40 !!!

37. 23/08/2001 SERC Orientation Programme : HPC 36 More Help on Optimization o Optimization and Tuning Guide -- IBM Manual o Man pages of compilers o High Performance Computing, by Kevin Dowd (O’Reilly & Assoc.)

38. 23/08/2001 SERC Orientation Programme : HPC 37 Running an Application

39. 23/08/2001 SERC Orientation Programme : HPC 38 How to use Servers ? o About 20 IBM Servers in SERC. o Servers run multiple background jobs o Difficult to figure out which server is least loaded at a given point of time. o Do not submit jobs on severs manually. o Use the LoadLeveler utility to submit jobs on servers.

40. 23/08/2001 SERC Orientation Programme : HPC 39 The LoadLeveler Utility o IBM LoadLeveler is a software layer that schedules user jobs on a distributed environment o Avialble on IBM RS/6000 Servers and on IBM RS/6000 SP o Predefined job classes (queues) are available depending on requested resource (cpu time, memory..)

41. 23/08/2001 SERC Orientation Programme : HPC 40 Using LoadLeveler  Include /home/loadl/bin in your search path (for IBM systems)  Create a file (say, myjob ) with the following lines : #@ executable = a.out #@ error = err.log #@ output = out.log #@ class = q120s #@ queue  Submit as : llsubmit myjob

42. 23/08/2001 SERC Orientation Programme : HPC 41 More LoadLeveler Commands  llq lists job status  llcancel cancels a job  llclass lists available queues  llstatus shows status of all machines  xloadl A graphical interface (GUI) to do all the above

43. 23/08/2001 SERC Orientation Programme : HPC 42 Load Sharing Facility (LSF) o LSF is another batch processing tool (similar to LoadLeveler) o Available on Ultra 60 and XP 1000 Workstation, Power Challenge, Dec – alpha Jwala and ES40 servers Common LSF commands :  bsub : submits a job  bkill : cancels a job  bjobs : lists job status  bqueues : lists available queues  bhosts : lists available hosts  xbsub : GUI to do all the above

44. 23/08/2001 SERC Orientation Programme : HPC 43 Supercomputing @ SERC

45. 23/08/2001 SERC Orientation Programme : HPC 44 Supercomputers (Parallel Machines) o For a large problem, the capability of a single CPU becomes insufficient o Such problems could be handled by harnessing the power of many CPU’s o Supercomputer has many CPU’s across which an application is distributed o Different parts of the code run on different CPU’s simultaneously

46. 23/08/2001 SERC Orientation Programme : HPC 45 Classes of Supercomputers Shared Memory (SMP) Many CPU’s share a single bank of memory Systems @ SERC m IBM SP(3) High Node m Compaq ES40 m SGI Power Challenge m Dec Turbolaser 8400 Distributed Memory (DMP) Each CPU has it’s own memory, and accesses other cpu’s memory over network Systems @ SERC m IBM SP(2) m C-DAC PARAM 10000

47. 23/08/2001 SERC Orientation Programme : HPC 46 Shared Memory Paradigm o Single memory, single address space o Easy to program o Auto-parallelizers render serial code to parallel o Program performance does not scale beyond 8- 16 CPU’s (most cases) o Still suitable for medium- sized problems CPU 1 CPU 2CPU 3CPU 4 BUS MEMORY

48. 23/08/2001 SERC Orientation Programme : HPC 47 IBM RS/6000 SP(3) High Node o 16-CPU node o Power3-II 375 MHz CPU o 32 GB Main Memory o AIX Parallelizing Compilers o Configured to accept batch jobs only.

49. 23/08/2001 SERC Orientation Programme : HPC 48 IBM RS/6000 SP(3) Thin Node o 4-CPU node o Power3-II 375 MHz CPU o 8 GB Main Memory o AIX Parallelizing Compilers o For test runs & development for codes to be run on the high node.

50. 23/08/2001 SERC Orientation Programme : HPC 49 IBM RS/6000 SP(3) : Shared Memory Parallel Libraries o SP(3) has both serial and parallel versions of scientific and linear algebra libraries ESSL and PESSL available. o To use the any particular library correct path has to be specified while linking. o For example if shared memory parallelized version of ESSL is to be used add -L/usr/lpp/essl.rte.smp43 –lessl while linking (or compiling)

51. 23/08/2001 SERC Orientation Programme : HPC 50 IBM RS/6000 SP(3) : Distributed Memory Parallelization o SP(3) can also be used in distributed memory mode with each processor having 2GB memory of its own. o For the program to be run in the distributed memory mode user must add specific calls to message passing library like MPI. o Details of MPI subroutines and how to use them is available with documentation for SP. o User has to use –lmpi option while linking the program which has mpi subroutine calls. o User s can take help of POE when running the distributed memory parallel code.

52. 23/08/2001 SERC Orientation Programme : HPC 51 IBM RS/6000 SP(3) : Serial Code Optimizations o It is power3 architecture computer so the optimization option to be used are -qarch=pwr3 and –qtune=pwr3 o All other optimizations are like IBM-590 servers. o Options like –bmaxdata and –bamxstack might have to be specified for large sized codes.

53. 23/08/2001 SERC Orientation Programme : HPC 52 Compaq Alpha Server ES40 o 4-CPU Shared Memory System o Alpha 667 MHz CPU o 8 GB Main Memory o DEC Compilers o KAP Fortran/C Auto- parallelizer o MPI Support

54. 23/08/2001 SERC Orientation Programme : HPC 53 Compaq Alpha Server ES40 o Kuck and Associates pre-compiler (KAP) is available this pre-compiler helps user in analysing the code for optimizations and also in distributed memory parallelization. o Usage is kapf. o Manual information on the kapf and kapc is available.

55. 23/08/2001 SERC Orientation Programme : HPC 54 Compaq Alpha Server ES40 o For shared memory parallelization using OpenMP standard directives can be used by using option –omp while compiling. o Out of six alpha servers only one is configured for login – rest five are configured to accept LSF batch jobs. o For interactive usage and code development for the alpha-servers one node alphas4 is to be used.

56. 23/08/2001 SERC Orientation Programme : HPC 55 SGI Power Challenge (everest) o 6-CPU Shared Memory System. o 1 GB of Memory o MIPS R8000 CPU o > 1 GFlops peak speed o MIPS Compilers and Auto-parallelizers

57. 23/08/2001 SERC Orientation Programme : HPC 56 DEC TurboLaser 8400 (jwala) o 4-CPU Shared Memory System o 1 GB of total Memory o DEC Alpha 333 MHz CPU o > 4 GFlops peak speed. o DEC Compilers o KAP Fortran/C Auto-Parallelizers o This is predecessor of ES40 alpha-servers.

58. 23/08/2001 SERC Orientation Programme : HPC 57 Distributed Memory Paradigm o New programming technique : need to use message-passing calls (MPI) in Fortran/C code to access remote memory o Scalable : No bottleneck on memory, performance improves by adding more and more CPU’s o Only choice for really large-scale problems CPU 1 CPU 2CPU 3CPU 4 MEM NETWORK SWITCH / HUB

59. 23/08/2001 SERC Orientation Programme : HPC 58 IBM RS/6000 SP (2) o 32-CPU Distributed Memory System o Power2 77/135 MHz CPU o 10GB total Memory o > 8 GFlops peak speed o High Performance Switch o Tuned MPI Library for improved communication

60. 23/08/2001 SERC Orientation Programme : HPC 59 IBM RS/6000 SP (2) o No automatic parallelizers.User has to add explicit message passing subprogram calls. o Optimized MPI and PVM both are available. o Of 30(now available) 2 machines are configured for logins, 8 machines are (including 2 login ones) are configured to take interactive jobs and can be used for code development and test runs. Rest all are configured to take only loadleveler jobs. o Another way to use multiple processors is to use Parallel Engineering and Scientific subroutine library (PESSL).

61. 23/08/2001 SERC Orientation Programme : HPC 60 C-DAC PARAM 10000 o 4-node dual-CPU cluster (total 8 CPUs) o SUN Ultra 400 MHz CPU o PARAMNet and Fast Ethernet Connectivity o Supports MPI o Also provides optimized MPI (CMPI) and lightweight messaging protocol (KSHIPRA)

62. 23/08/2001 SERC Orientation Programme : HPC 61 C-DAC PARAM 10000 o User explicit parallelization across nodes using MPI however since each node is a shared memory dual processor, for single node auto parallelizing and shared memory parallelization through Open MP is possible. o The help regarding how to use the MPI and CMPI on PARAM10000 is given when a user is validated. o It has several application software packages for chemistry, fluid dynamics etc. ported and available. o It has sun workshop program development environment. o Several CDAC developed packages add to functionality these packages are parallel usage visualizer, parallel debugger, bottle-neck analyzer etc.

63. 23/08/2001 SERC Orientation Programme : HPC 62 Troubleshooting

64. 23/08/2001 SERC Orientation Programme : HPC 63 Problems : Search Path Q. How do I add a directory in my search path ? A. For C-shell users : set path = ($path /directory) For Bourne- and K-shell users : export PATH=$PATH:/directory

65. 23/08/2001 SERC Orientation Programme : HPC 64 Problems : Disk Space Q. I need more disk space than what was allotted. What to do ? A. Try the following : m See if you can cut down some I/O statements and save space.  Use the /tmp or /temp directory. m If all these fails, apply for disk quota increase (form available in Room 104)

66. 23/08/2001 SERC Orientation Programme : HPC 65 Usage of /tmp Space  /temp is available in all systems. Also each system has its own /tmp o Always create a directory of your own and work within this directory  /temp and /tmp are purged regularly. Anything more than 7 days old can get deleted without notice.  Use /temp for data files only.

67. 23/08/2001 SERC Orientation Programme : HPC 66 Problems : Memory Q. My problem requires large memory.  Find out how much memory (bytes) are needed by : size exe_filename m Try to reduce memory by cutting some unnecessary arrays. Do not over- dimension arrays. m Use the servers with larger memory. m Migrate to a parallel system.

68. 23/08/2001 SERC Orientation Programme : HPC 67 Sources of Help o Man pages on the systems. o On-line Hypertext (through help icon in CDE) in the machines. o Hardcopy manuals in SERC library. o Through SERC web pages. o Many other tutorials and e-books generally available on Net.

69. 23/08/2001 SERC Orientation Programme : HPC 68 Information on the Web o WWW is the best way to keep you updated on the latest. o Check out SERC web site : http://www.serc.iisc.ernet.in/ (click on Computing Facilities ) o Many of the lecture notes in this series will also be made available under User Information in SERC web pages.

70. 23/08/2001 SERC Orientation Programme : HPC 69 SERC Wishes you a Happy Computing Experience over the years to Come!!