1. Advanced Topics: MPI jobs
The EPIKH Project
(Exchange Programme to advance e-Infrastructure Know-How)
Advanced Topics: MPI jobs
Diego Scardaci
INFN Catania
Joint CHAIN/GISELA/EPIKH Grid School for Application Porting
29th November - 9th Dicember 2010

2. Table of Contents MPI and its implementations
Wrapper script for mpi-start
Hooks for mpi-start
Defining the job and executable
Running the MPI job
References
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

3. MPI and its implementations
The Message Passing Interface (MPI) is a de-facto standard for writing parallel application.
There are two versions of MPI, MPI-1 and MPI-2;
Two implementations of MPI-1:
LAM;
MPICH.
Two implementations of MPI-2:
OpenMPI;
MPICH2.
Individual sites may chose to support only a subset of these implementations, or none at all.
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

4. Requirements to submit MPI jobs
Proxy certificate.
Permissions to submit job in a specific site.
The mpi jobs can run only in one Grid site. 3
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

5. Get information from the information sys
Find the sites that support MPICH and MPICH2 4
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

6. Get information from the information sys
Find the sites that support MPICH and its available CPUs 5
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

7. Get information from the information sys
Find the sites who have shared home MPI directory
What happens when there is a MPI shared directory in the site?
The file is compiled in one WN, then the file read by the other WNs in the same site.
What happens when is not MPI shared directory?
The file is compiled in one WN, then the file must be copied to the others WNs.
Tag=MPI-START 6
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

8. MPI and its implementations Wrapper script for mpi-start
Hooks for mpi-start
Defining the job and executable
Running the MPI job
References
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

9. mpi-start
mpi-start is a recommended solution to hide the implementation details for jobs submission.
The design of mpi-start was focused in making the MPI job submission as transparent as possible from the cluster details!
It was developed inside the Int.EU.Grid project
The RPM to be installed in all WNs can be found here
Using the mpi-start system requires the user to define a wrapper script that set the environment variables and a set of hooks.
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

10. Wrapper script for mpi-start
#!/bin/bash
# Pull in the arguments.
MY_EXECUTABLE=`pwd`/$1
MPI_FLAVOR=$2
# Convert flavor to lowercase for passing to mpi-start.
MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'`
# Pull out the correct paths for the requested flavor.
eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH`
# Ensure the prefix is correctly set. Don't rely on the defaults.
eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH
export I2G_${MPI_FLAVOR}_PREFIX
# Touch the executable.
#It exist must for the shared file system check.
# If it does not, then mpi-start may try to distribute the executable
# when it shouldn't.
touch $MY_EXECUTABLE
# Setup for mpi-start.
export I2G_MPI_APPLICATION=$MY_EXECUTABLE
export I2G_MPI_APPLICATION_ARGS=
export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER
export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh
export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh
# If these are set then you will get more debugging information.
export I2G_MPI_START_VERBOSE=1
#export I2G_MPI_START_DEBUG=1
# Invoke mpi-start.
$I2G_MPI_START
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

11. MPI and its implementations Wrapper script for mpi-start
Hooks for mpi-start
Defining the job and executable
Running the MPI job
References
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

12. Hooks for mpi-start /1
The user may write a script which is called before and after the MPI executable is run.
The pre-hook script can be used, for example, to compile the executable itself or download data;
The post-hook script can be used to analyze results or to save the results on the grid.
The pre- and post- hooks script may be defined in separate files, but the name of the functions named exactly “pre_run_hook” and “post_run_hook”
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

13. Hooks for mpi-start /2 #!/bin/sh
# This function will be called before the MPI executable is started. #
pre_run_hook () {
# Compile the program.
echo "Compiling ${I2G_MPI_APPLICATION}"
# Actually compile the program.
cmd="mpicc ${MPI_MPICC_OPTS} -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c"
echo $cmd
$cmd
if [ ! $? -eq 0 ]; then
echo "Error compiling program. Exiting..."
exit 1 fi
# Everything's OK.
echo "Successfully compiled ${I2G_MPI_APPLICATION}"
return 0 }
# This function will be called before the MPI executable is finished.
# A typical case for this is to upload the results to a Storage Elem.
post_run_hook () {
echo "Executing post hook."
echo "Finished the post hook."
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

14. MPI and its implementations Wrapper script for mpi-start
Hooks for mpi-start
Defining the JDL for the job and executable
Running the MPI job
References
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

15. Defining the job and executable /1
Running the MPI job itself is not significantly different from running a standard grid job. JobType = “Normal";
CpuNumber = 2;
Executable = "mpi-start-wrapper.sh";
Arguments = "mpi-test MPICH";
StdOutput = "mpi-test.out";
StdError = "mpi-test.err";
InputSandbox = {"mpi-start-wrapper.sh", "mpi-hooks.sh","mpi-test.c"};
OutputSandbox = {"mpi-test.err","mpi-test.out"};
Requirements = Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment)
&& Member(“MPICH", other.GlueHostApplicationSoftwareRunTimeEnvironment);
The JobType must be “Normal” and the attribute CpuNumber must be defined
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

16. Structure of a MPI job in the Grid with mpi-start
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

17. Structure of a MPI job in the Grid with mpi-start8
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

18. Defining the job and executable /2
#include "mpi.h"
#include <stdio.h>
int main(int argc, char *argv[]) {
int numprocs; /* Number of processors */
int procnum; /* Processor number */
/* Initialize MPI */
MPI_Init(&argc, &argv);
/* Find this processor number */
MPI_Comm_rank(MPI_COMM_WORLD, &procnum);
/* Find the number of processors */
MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
printf ("Hello world! from processor %d out of %d\n", procnum, numprocs);
/* Shut down MPI */
MPI_Finalize();
return 0; }
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

19. Hook Helpers
The shell variable $MPI_START_SHARED_FS can be checked to figure out if the current site has a shared file system or not.
The mpi_start_foreach_host shell function can be used to iterate over all the available machines in the current run. do_foreach_node () { # the first parameter $1 contains the hostname }
post_run_hook () {
...
mpi_start_foreach_host do_foreach_node
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

20. MPI and its implementations Wrapper script for mpi-start
Hooks for mpi-start
Defining the job and executable
Running the MPI job
References
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

21. Running the MPI job is no different from any other grid job.
If the job ran correctly, then the standard output should contain something like the following: -<START PRE-RUN HOOK>
[…]
-<STOP PRE-RUN HOOK>
=[START]=========================================================
Hello world! from processor 1 out of 2
Hello world! from processor 0 out of 2
=[FINISHED]======================================================
-<START POST-RUN HOOK>
-<STOP POST-RUN HOOK>
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

22. Running MPI job in the Grid without mpi-start12
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

23. MPI and its implementations Wrapper script for mpi-start
Hooks for mpi-start
Defining the job and executable
Running the MPI job
References
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

24. http://applications.eu- eela.eu/grid_faq_MPICH.php?l=40&n=14
References
GISELA MPI FAQ:
eela.eu/grid_faq_MPICH.php?l=40&n=14
EGEE Mpi guide [ link ]
EGEE MPI WG [ link ]
MPI-START Documentation [ link ]
Site config for MPI [ link ]
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,

25. Thank you for your kind attention !
Any questions ?
Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting,