Presentation is loading. Please wait.

Presentation is loading. Please wait.

Www.eu-eela.org E-science grid facility for Europe and Latin America Porting MPI Applications use cases: OpenFOAM, Fluent Marcello Iacono Manno Consorzio.

Similar presentations


Presentation on theme: "Www.eu-eela.org E-science grid facility for Europe and Latin America Porting MPI Applications use cases: OpenFOAM, Fluent Marcello Iacono Manno Consorzio."— Presentation transcript:

1 E-science grid facility for Europe and Latin America Porting MPI Applications use cases: OpenFOAM, Fluent Marcello Iacono Manno Consorzio COMETA JOINT EELA2-EGEE3 TUTORIAL4TRAINERS Catania, 2008, July 4 th

2 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Outline OpenFOAM to the Grid – Porting Issues – Submission procedure – Developed Tools – Future Developments Fluent – Porting Issues – Submission procedure – Developed Tools – Future Developments Abaqus Developed tools Future Development

3 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Porting (1/5) OpenFOAM is an open-source SW for computational fluid dynamics (CFD) it is a parallel, multi – stage application several solvers, C-based libraries PORTING OPENFOAM to the Grid: a “forerunner” experience – static installation – the gcc4.2.1/pgcc compiler – the 32/64 bit issue – LAM / MPICH – which mpirun – machines file – preparing parallel execution

4 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Porting (2/5) Installation – compressed package is about 300 MB – dynamical installation not feasible  “stable” package  jobs can also be “short” (test)  environment problem not solved  … but some further testing is required on this particular issue – static installation  software manager directly on the WNs installation job (lcg-asis)  check for compatibility middleware other applications

5 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – – Two ways to install the software:  Static Installation (local or shared fixed installation) Advantage: speeds up job execution (less work to do) Disadvantages: ocompatibility problems with middleware and/or other applications omore complex modifying and updating the application orequires Software Manager (SWM) role privileges Usage: ohuge and/or stable SW packages oonly MPI binary executable available (difficultly modifiable)  Dynamic Installation (“local” installation during job pre – processing) Advantages: omore robust and flexible jobs oeasier modifying and updating oNo SWM privileges required Disadvantages: slows down job execution (more work to do) Usage: olittle and/or frequently modified SW packages oMPI launching script (easily modifiable) [Static vs. Dynamic Installation]

6 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Porting (3/5) Gcc4.1.2 – different middleware versions  OpenFOAM-1.4 requires at least gcc3.4.6 (4.1.2 suggested)  ”old” WNs run 32 bits Scientific Linux (SLC) with gcc3.2.6  … INCOMPATIBILITY – 64-bit installation  full-64 OpenFOAM-1.4 bit is available  “new” WNs have a 64 bit-architecture with SLC4 and gcc4.x.x 32/64 bit – no 64-bit user interface available up to … yesterday  pre and post – processing must run (as jobs) on the WN  autonomous jobs or part of a greater job (pre / post) ?  alternatively cross-pre-processing on a 64-bit machine, then transferring the OpenFOAM “user” directory

7 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Porting (4/5) LAM / MPICH – different protocols  OpenFOAM-1.4 is declared MPICH-compliant (by official docs)  only libPstream needed to be recompiled (thanks to the forum!)  no “native” mpirun allowed (Grid incompatible) Which mpirun – new 64-bit (Grid compliant) mpirun script – different libraries for MPICH/MPICH2, InfiniBand/GigaBit – device p4  ch_p4 Machines files – specified by the middleware (transparent to the user) – all the WNs of a CE

8 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Porting (5/5) Preparing parallel execution – OpenFOAM user workspace is “local”: – a sub – directory called user-1.4 under $PWD – $PWD/user-1.4/run/tutorials/ / /system –  solver (icoFoam) –  geometry (cavity) – system  running system information – (file) decomposeParDict  mesh decomposition – decomposePar icoFoam cavity – (file) proc0…n  processing element information Current pre-processing – needs to become a job or – alternatively, a 64-bit UI process

9 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – OpenFOAM.grid]$ cat openfoam.jdl Type = "Job"; JobType = "MPICH"; Executable = "/opt/exp_soft/cometa/OpenFOAM/OpenFOAM-1.4/applications/bin/ linuxGcc4DPOpt/icoFoam"; Arguments = ". cavity -parallel"; NodeNumber = 4; StdOutput = "openfoam.out"; StdError = "openfoam.err"; InputSandbox = {"mpi.pre.sh","mpi.post.sh","cavity.tar"}; OutputSandbox = {"openfoam.out","openfoam.err","result.tgz"}; RetryCount = 7; Requirements = ( other.GlueCEUniqueID == "infn-ce-01.ct.pi2s2.it:2119/jobmanager-lcglsf-infinite" ); Submit (1/10)

10 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Submit (2/10) OpenFOAM.grid]$ cat mpi.pre.sh #!/bin/bash date echo "openfoam pre-processing file" hostname -f tar xzf cavity.tar ls -l echo "content of $PWD/cavity" ls -l $PWD/cavity

11 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Submit (3/10) OpenFOAM.grid]$ ll /home/marcello/JobOutput/marcello_- fGr9naNAp6Phj3P_GacQw total 88 -rw-rw-r-- 1 marcello marcello 0 Jul 2 16:02 openfoam.err -rw-rw-r-- 1 marcello marcello Jul 2 16:02 openfoam.out OpenFOAM.grid]$ head -n 100 /home/marcello/JobOutput/marcello_- fGr9naNAp6Phj3P_GacQw/openfoam.out Mon Jul 2 15:54:03 CEST 2007 openfoam pre-processing file infn-wn-10.ct.pi2s2.it WM_LINK_LANGUAGE=c++ WM_ARCH=linux WM_JAVAC_OPTION=Opt WM_PROJECT_VERSION=1.4 WM_COMPILER_LIB_ARCH= WM_PROJECT_INST_DIR=/opt/exp_soft/cometa/OpenFOAM WM_PROJECT_DIR=/opt/exp_soft/cometa/OpenFOAM/OpenFOAM-1.4 WM_COMPILER_ARCH=-64 shared installation space 64-bit architecture

12 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Submit (4/10) WM_PROJECT=OpenFOAM WM_COMPILER=Gcc4 WM_MPLIB=MPICH WM_COMPILER_DIR=/opt/exp_soft/cometa/OpenFOAM/linux/gcc WM_COMPILE_OPTION=Opt WM_SHELL=bash WM_DIR=/opt/exp_soft/cometa/OpenFOAM/OpenFOAM-1.4/wmake WM_PROJECT_USER_DIR=/home/cometa005/globus-tmp.infn-wn /.mpi/https_3a_2f_2finfn-rb-01.ct.pi2s2.it_3a9000_2f- fGr9naNAp6Phj3P_5fGacQw/user-1.4 WM_OPTIONS=linuxGcc4DPOpt WM_PROJECT_LANGUAGE=c++ WM_PRECISION_OPTION=DP “native” gcc compiler double precision

13 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Submit (5/10) content of: /home/cometa005/globus-tmp.infn-wn /.mpi/https_3a_2f_2finfn-rb- 01.ct.pi2s2.it_3a9000_2f-fGr9naNAp6Phj3P_5fGacQw total 5536 lrwxrwxrwx 1 cometa005 cometa 130 Jul 2 15:54 cometa > /home/cometa005/globus- tmp.infn-wn /.mpi/https_3a_2f_2finfn-rb-01.ct.pi2s2.it_3a9000_2f- fGr9naNAp6Phj3P_5fGacQw/user-1.4 -rw-r--r-- 1 cometa005 cometa 0 Jul 2 15:53 host rw-r--r-- 1 cometa005 cometa 64 Jul 2 15:53 mpi.post.sh -rw-r--r-- 1 cometa005 cometa 547 Jul 2 15:53 mpi.pre.sh -rw-r--r-- 1 cometa005 cometa 0 Jul 2 15:54 openfoam.err -rw-r--r-- 1 cometa005 cometa 878 Jul 2 15:54 openfoam.out drwxr-xr-x 5 cometa005 cometa 4096 Jun 18 13:15 user-1.4 -rw-r--r-- 1 cometa005 cometa Jul 2 15:53 user-1.4.tar content of /opt/exp_soft/cometa/OpenFOAM total 12 -rw-rw-rw- 1 cometa001 cometa 382 Jun 20 10:09 env_config drwxrwxrwx 3 cometa001 cometa 4096 Jun 6 08:48 linux drwxrwxrwx 10 cometa001 cometa 4096 Jun 8 18:23 OpenFOAM-1.4 home dir on the “master” WN pre and post – processing scripts user dir

14 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Submit (6/10) content of /home/cometa005/globus-tmp.infn-wn /.mpi/https_3a_2f_2finfn- rb01.ct.pi2s2.it_3a9000_2f-fGr9naNAp6Phj3P_5fGacQw/user-1.4/run/tutorials total 16 -rwxr-xr-x 1 cometa005 cometa 1308 Jun 19 16:31 controlDict -rwxr-xr-x 1 cometa005 cometa 1363 Jun 29 13:53 decomposeParDict -rwxr-xr-x 1 cometa005 cometa 1479 Jun 19 16:31 fvSchemes -rwxr-xr-x 1 cometa005 cometa 1305 Jun 19 16:31 fvSolution /* *\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: 1.4 | | \\ / A nd | Web: | | \\/ M anipulation | | \* */ […] mesh decomposition

15 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Submit (7/10) [1] Date : Jul [1] Time : 15:54:22 [1] Host : infn-wn-10.ct.pi2s2.it [1] PID : [4] Date : Jul [4] Time : 15:54:22 [4] Host : infn-wn-06.ct.pi2s2.it [4] PID : [3] Date : Jul [3] Time : 15:54:22 [3] Host : infn-wn-10.ct.pi2s2.it […] [MPI Pstream initialized with: floatTransfer : 1 nProcsSimpleSum : 0 scheduledTransfer : 0

16 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Submit (8/10) Exec : /opt/exp_soft/cometa/OpenFOAM/OpenFOAM- 1.4/applications/bin/linuxGcc4DPOpt/icoFoam /home/cometa005/globus-tmp.infn-wn /.mpi/https_3a_2f_2finfn-rb-01.ct.pi2s2.it_3a9000_2f- fGr9naNAp6Phj3P_5fGacQw/cometa /run/tutorials/icoFoam cavity –parallel [0] 7 [0] ( [0] infn-wn-10.ct.pi2s2.it [0] infn-wn-10.ct.pi2s2.it [0] infn-wn-10.ct.pi2s2.it [0] infn-wn-06.ct.pi2s2.it [0] infn-wn-06.ct.pi2s2.it [0] infn-wn-06.ct.pi2s2.it [0] infn-wn-06.ct.pi2s2.it [0] ) [0] Create time Process IDs

17 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Submit (9/10) Create mesh for time = 0 [7] Date : Jul [7] Time : 15:54:22 [7] Host : infn-wn-06.ct.pi2s2.it [7] PID : [7] Root : /home/cometa005/globus-tmp.infn-wn /.mpi/https_3a_2f_2finfn-rb- 01.ct.pi2s2.it_3a9000_2f-fGr9naNAp6Phj3P_5fGacQw/cometa /run/tutorials/icoFoam [7] Case : cavity [7] Nprocs : 8 Reading transportProperties Reading field p Reading field U Reading/calculating face flux field phi Starting time loop

18 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Submit (10/10) Time = Courant Number mean: 0 max: 0 DILUPBiCG: Solving for Ux, Initial residual = 1, Final residual = e-06, No Iterations 11 DILUPBiCG: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0 DICPCG: Solving for p, Initial residual = 1, Final residual = e-07, No Iterations 47 time step continuity errors : sum local = e-09, global = e-10, cumulative = e-10 DICPCG: Solving for p, Initial residual = , Final residual = e-07, No Iterations 46 time step continuity errors : sum local = e-08, global = e-10, cumulative = e-12 ExecutionTime = 0.06 s ClockTime = 0 s

19 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Developed Tools(1/2) #!/bin/bash VO=cometa APP=openfoam SE=infn-se-01.ct.pi2s2.it lfc-rm -r "/grid/$VO/$APP/$1" 2> filelist FILELIST=`sed 's/[[:space:]]/_/g' filelist` REVERSELIST="" for item in $FILELIST; do REVERSELIST=$item" "$REVERSELIST done echo $REVERSELIST ROOT_DIR="$1" Recursive Up/Download from Data Catalog – middleware extension (bulk operation) – recursively descends the directory’s tree  on the local file system to upload files on a Data Catalog  on the Data Catalog to download files on the local file systems

20 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Developed Tools(2/2) if [ "$ROOT_DIR" != "" ]; then #mkdir $PWD/$ROOT_DIR for FILE in $REVERSELIST; do FILE=${FILE#1*_} #echo "FILE="$FILE LFC_DIR=`echo "$FILE" | grep 'Directory_not_empty'` if [ "$LFC_DIR" != "" ]; then LFC_DIR=`echo ${LFC_DIR%':_Directory_not_empty'}` #echo "Dir: $LFC_DIR" LOC_DIR=`echo ${LFC_DIR##"/grid/$VO/$APP/"}` #echo "LOC_DIR=$LOC_DIR" DIR_TEST=`ls $PWD/$LOC_DIR` 2>>/dev/null if [ "$DIR_TEST" != "" ]; then echo "$PWD/$LFC_DIR is an existing directory" else mkdir "$PWD/$LOC_DIR" echo "created dir: $PWD/$LOC_DIR" fi else FILE=`echo ${FILE%':_File_exists'}` #echo "File: $FILE" LOC_FILE=`echo ${FILE##"/grid/$VO/$APP/"}` lcg-cp --vo $VO lfn:"$FILE" file:"$PWD/$LOC_FILE" echo "created file: $PWD/$LOC_FILE" fi done fi

21 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Future Developments and then…? – pre and post - processing  environment exportation in mpi.pre.sh requires middleware modification may be useful for other applications  mesh decomposition as a job required only for a few complex cases  data recollection from slaves already developed the required tools integration needed  full 64 – bit implementation (almost complete) – “project” as a whole  GUI integration/development  (general purpose) data storage

22 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Fluent: generality Fluent is a commercial SW for computational fluid dynamics (CFD) it is a parallel, multi – stage application several solvers, C-based libraries PORTING FLUENT to the Grid: a “lucky” experience – MPI wrapper supplied – MPICH compliant – InfiniBand net supported – already runs on a “MVAPICH” cluster – static installation – actual submission in mpi.pre.sh – “dummy” invocation of the “regular” MPI wrapper

23 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Fluent: submission procedure “canonic” procedure would be different … – recompile SW with pgcc – use the Grid “built-in” MPI wrapper (MPICH/2 over IB) – non – sense if a SW package already runs at (almost) first shot – but …. the reasons for a unique MPI wrapper are nonetheless valid – standardization: heterogeneous (proprietary) solutions could be difficult to manage in the future (with many more users) – security: user should not be allowed to use resources by- passing the usual submission mechanism (CE unaware) – so …. we are looking for a trade-off solution

24 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Fluent: license Fluent is a licensed SW – a license server – application contacts the license server – the accounting system is under development the submission procedure for Fluent (sequential or parallel) jobs is available at: https://grid.ct.infn.it/twiki/bin/view/PI2S2/SubmitFLUENT

25 Catania (Italy), Joint EELA/EGEEIII Tutorial for Trainers, – Any Questions ? Thank you very much for your kind attention!


Download ppt "Www.eu-eela.org E-science grid facility for Europe and Latin America Porting MPI Applications use cases: OpenFOAM, Fluent Marcello Iacono Manno Consorzio."

Similar presentations


Ads by Google