1. IBM India Presentation subtitle: 20pt Arial Regular, teal R045 | G182 | B179 Recommended maximum length: 2 lines Confidentiality/date line: 13pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Disclaimer information may also be appear in this area. Place flush left, aligned at bottom, 8-10pt Arial Regular, white IBM logo must not be moved, added to, or altered in any way. Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements Presentation title: 28pt Arial Regular, black Recommended maximum length: 2 lines Group name: 17pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white © 2007 IBM Corporation Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Compiling and Executing Applications on BlueGene Subba R Bodda and Ravi K Komanduri IBM India Systems and Technology Laboratory

2. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 2 Agenda BlueGene setup Compiling applications on BlueGene Executing applications on BlueGene

3. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 3 Redbooks Link: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=blue+AND+gene Blue Gene System Administration, SG24-7417-00 IBM System Blue Gene Solution: System Administration, SG24-7178-04 IBM System Blue Gene Solution: Configuring and Maintaining Your Environment, SG24-7352-00 Blue Gene/L: Hardware Overview and Planning, SG24-6796-02 IBM System Blue Gene Solution: Hardware Installation and Serviceability, SG24-6743-00 Blue Gene Safety Considerations, REDP-4257-00 Blue Gene/L: Safety Considerations, REDP-3983-01 Blue Gene/L: Performance Analysis Tools, SG24-7278-00 IBM System Blue Gene Solution: Application Development, SG24-7179-04 Unfolding the IBM eServer Blue Gene Solution, SG24-6686-00 GPFS Multicluster with the IBM System Blue Gene Solution and eHPS Clusters, REDP-4168-00 IBM System Blue Gene Solution Problem Determination Guide, SG24-7211-00 Evolution of the IBM System Blue Gene Solution, REDP-4247-00

4. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 4 BlueGene setup bglfen01(10.16.28.30),bglfen02 is the front end node bglsn is the service node Login to bglfen and compile your code Use mpirun or LL to submit your jobs bglsn has the service daemons which would spawn your jobs on actual BG nodes

5. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 5 IBM Compilers for Blue Gene IBM compilers (Fortran, C, C++) for Blue Gene are located on the frontend node in /opt/ibmcmp Fortran: /opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf /opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf90 /opt/ibmcmp/xlf/bg/10.1/bin/blrts_xlf95 C: /opt/ibmcmp/vac/bg/8.0/bin/blrts_xlc C++: /opt/ibmcmp/vacpp/bg/8.0/bin/blrts_xlC Note : Users can add the respective directories to their PATH.

6. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 6 GNU Compilers for Blue Gene GNU compilers (Fortran, C, C++) for Blue Gene are located in /opt/blrts-gnu/ Fortran: /opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-g77 C: /opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-gcc C++: /opt/blrts-gnu/bin/ powerpc-bgl-blrts-gnu-g++

7. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 7 MPI on Blue Gene MPI implementation on Blue Gene is based on MPICH-2 from Argonne National Laboratory. Include files mpi.h and mpif.h are at the location: -I/bgl/BlueLight/ppcfloor/bglsys/include Libraries to link for MPI are: -L/bgl/BlueLight/ppcfloorbglsys/lib -lmpich.rts -lmsglayer.rts -lrts.rts - ldevices.rts Note : The linking order for the MPI libraries should be as mentioned above.

8. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 8 Compiler Optimizations

9. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 9 Compiler Optimizations

10. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 10 IBM ESSL libraries The IBM ESSL (Engineering Scientific Subroutine Library) is located in /opt/ibmmath/essl/4.2/

11. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 11 Executing jobs using mpirun cd /gpfs/fcHome/scratch/userid mpirun –np 32 –partition R000 -cwd /gpfs/fcHome/scratch/ -exe /gpfs/fcHome/scratch/hello_c Common options: -args : List of arguments to the executables in double quotes -env : List of environment variables in double quotes. “VARIABLE=value” -mode : CO or VN (coprocessor/virtual-node) -only_test_protocol : This options is used only to test communication protocol coprocessor mode : one MPI process per node, 1 GB limit per process. Default mode virtual-node mode : two MPI processes per node, 512 MB limit per process; L3 cache, memory, networks, are shared For more details type the following at a command prompt : mpirun -h Note : A user can have only one job per Blue Gene partition

12. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 12 Executing jobs using submit_job Login to bglsn cd /bgl/BlueLight/ppcfloor/bglsys/bin./mmcs_db_console mmcs$ allocate M03B_J214_64 mmcs$ submitjob M03B_J214_64 /bgl/hello.rts /bgl/ OK jobId=25 mmcs$ waitjob M03B_J214_64 25 OK mmcs$ free M03B_J214_64 OK mmcs$ quit

13. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 13 Environment Variables Environment variables can be specified using –env “XYZ=123 ABC=456” Some of the environment variables are  BGLMPI_EAGER  BGL_APP_L1_WRITE_THROUGH  BGL_APP_L1_SWOA  BGLMPI_COPRO_SENDS  BGLMPI_INTERRUPT

14. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 14 Redbooks Link: http://www.redbooks.ibm.com/cgi-bin/searchsite.cgi?query=blue+AND+gene Blue Gene System Administration, SG24-7417-00 IBM System Blue Gene Solution: System Administration, SG24-7178-04 IBM System Blue Gene Solution: Configuring and Maintaining Your Environment, SG24-7352-00 Blue Gene/L: Hardware Overview and Planning, SG24-6796-02 IBM System Blue Gene Solution: Hardware Installation and Serviceability, SG24-6743-00 Blue Gene Safety Considerations, REDP-4257-00 Blue Gene/L: Safety Considerations, REDP-3983-01 Blue Gene/L: Performance Analysis Tools, SG24-7278-00 IBM System Blue Gene Solution: Application Development, SG24-7179-04 Unfolding the IBM eServer Blue Gene Solution, SG24-6686-00 GPFS Multicluster with the IBM System Blue Gene Solution and eHPS Clusters, REDP-4168-00 IBM System Blue Gene Solution Problem Determination Guide, SG24-7211-00 Evolution of the IBM System Blue Gene Solution, REDP-4247-00

15. IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations IBM India © 2007 IBM Corporation Optional slide number: 10pt Arial Bold, white Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 15 Summary Compilation IBM XLF/XLC and GNU Compilers ESSL Libraries submit_job mpirun LL Environment Variables