Submit Host Setup (user) Tc.data file Pool.config file Properties file Vdl-gen file Input file Exitcode checking script

Tc.data file TG-SDSC FOAM./foam1.5ifc8.1gm INSTALLED INTEL32::LINUX GLOBUS::jobtype=mpi,count=8; TG-SDSC globus-url-copy /usr/local/apps/globus gcc- r5/bin/globus-url-copy INSTALLED INTEL32::LINUX ENV::GLOBUS_LOCATION=/usr/local/apps/globus gcc- r5;ENV::LD_LIBRARY_PATH=/usr/local/apps/globus gcc- r5/lib/;ENV::PATH=/usr/local/apps/globus gcc-r5/bin/ TG-SDSC transfer /users/nefedova/bin/transfer INSTALLED INTEL32::LINUX ENV::GLOBUS_LOCATION=/usr/local/apps/globus gcc- r5;ENV::LD_LIBRARY_PATH=/usr/local/apps/globus gcc- r5/lib;ENV::PATH=/usr/local/apps/globus gcc- r5/bin;GLOBUS::maxwalltime=15 TG-SDSC PostProcess /users/nefedova/bin/PostProc INSTALLED INTEL32::LINUX ENV::PATH=/users/nefedova/bin:/usr/bin:/bin TG-SDSC DirGen /users/nefedova/bin/DirGen INSTALLED INTEL32::LINUX ENV::PATH=/work/ux455036:/usr/bin:/bin

Pool.config file /gpfs/nefedova /home/nefedova/FOAM/SDSC- Production/exitcode

Properties file vds.replica.mode rls #vds.rls.url rlsn://smarty.isi.edu vds.rls.url rls://evitable.uchicago.edu vds.pool.mode=xml vds.rls.query multiple vds.tc.file=/home/nefedova/FOAM/NCSA/tc.data vds.pool.file=/home/nefedova/FOAM/NCSA/pool.config vds.scheduler.remote.projects TG=TG-ATM050001T vds.db.vdc.schema AnnotationSchema vds.tc.mode=File vds.db.ptc.schema InvocationSchema vds.db.*.driver Postgres vds.db.*.driver.url jdbc:postgresql:foamvdc vds.db.*.driver.user nefedova vds.db.*.driver.password nefedova vds.db.vdc.schema.file.store /home/nefedova/FOAM/test/vdc.db vds.transfer.thirdparty.pools=local,TG vds.transfer.mode multiple vds.scheduler.condor.output.stream false vds.scheduler.condor.error.stream false vds.site.selector Group

Vdl-gen file #! /bin/sh if /usr/bin/test ! $# -eq 2 then echo "Syntax: vdl-gen FileName Anomstart" exit 1 fi cat >./foam-gen.vdl <<EOFF TR FOAM( input RunParamsName, output DirAtmos, output DirOcean, output DirCoupl, input WorkDirNam e, none vds_workdir, none vds_group ) { argument = \${RunParamsName}; profile globus.maxWallTime = "210"; profile vds.workdir =\${vds_workdir}; profile vds.group=\${vds_group}; } TR PostProcess( input DirName, none entityName, none Finished, none id, output entityFile, none v ds_group, none vds_workdir, none F1_5 ) { argument = \${entityName}; argument = \${Finished}; argument = \${id}; argument = \${F1_5}; profile globus.maxWallTime = "7"; profile vds.workdir =\${vds_workdir}; profile vds.group=\${vds_group}; } TR DirGen( input Finished, input WorkDir, input Anomstart, output WorkDirName ) { argument = \${Finished}; argument = \${WorkDir}; argument = \${Anomstart}; profile globus.maxWallTime = "3"; } EOFF

Vdl-gen file (contd.) ….. while read x; do if /usr/bin/test $i -eq "1" then MEM=$x i=0 WorkDir=`echo $x | awk 'BEGIN {FS="."} {print $3}'` else fileno=`expr $x + 1` cat >>./foam-gen.vdl <<EOFF DV DirGen$NEM->DirGen( ); DV FOAM_RUN$NEM->FOAM( vds_workdir="foam-$WorkDir", vds_group="Foam$NEM" ); DV Atmos$NEM->PostProcess( entityName="atmos", id="ha", Finished="$x", F1_5="$MEM", vds_workdir="foam-$WorkDir", vds_group="Foam$NEM" ); DV Ocean$NEM->PostProcess( entityName="ocean", id="ho", Finished="$x", F1_5="$MEM", vds_workdir="foam-$WorkDir", vds_group="Foam$NEM" ); DV Coupl$NEM->PostProcess( entityName="coupl", id="hc", Finished="$x", F1_5="$MEM", vds_workdir="foam-$WorkDir", vds_group="Foam$NEM" ); EOFF cat >> dax-dv-input <<EOFF Atmos$NEM Ocean$NEM Coupl$NEM EOFF

Generating VDL file Input file: F1_5.4yrensNovpos.mem F1_5.4yrensNovpos.mem F1_5.4yrensNovpos.mem ………………………..{any number of members} Syntax:./vdl-gen InputFileName Anomstart

Compute node setup Directory generation step FOAM run Post-processing jobs (m2ynce.py)

Directory setup DirGen Tc.data file: TG-SDSC DirGen /users/nefedova/bin/DirGen INSTALLED INTEL32::LINUX ENV::PATH=/work/ux455036:/usr/bin:/bin #! /bin/sh if /usr/bin/test ! $# -eq 3 then echo "Syntax: DirGen Finished WorkDir ANOMSTART" exit 1 fi CURRENT_DIR=`pwd` mkdir -p foam-$2 cd foam-$2 mkdir -p history restart cd history mkdir -p atmos coupl ocean cd../ cp /users/ux455036/foame/foam1.5ifc8.1 foam1.5ifc8.1gm cat >> run_params <<EOFF RESTFRQ: 1440 HISTFRQ: 30 PREFIX:./ STORAGE:./ TIME_INV: /gpfs/nefedova/f1.5data FINISHED: $1 RUNLNG: 1440 ANOMSTART: $3 EOFF cp /gpfs/nefedova/atmos_params. cd restart globus-url-copy gsiftp://wiggum.mcs.anl.gov/home/climate1/jacob/ensrestart/rs.F1_5.75co2ctlLE.$1.tar file://$CURRENT_DIR/foam-$2/restart/rs.F1_5.75co2ctlLE.$1.tar tar -xf rs.F1_5.75co2ctlLE.$1.tar echo Done

FOAM run Executable must be compiled on the host it is running on Location: –tc.data: TG-SDSC FOAM./foam1.5ifc8.1gm INSTALLED INTEL32::LINUX GLOBUS::jobtype=mpi,count=8; –DirGen: cp /users/ux455036/foame/foam1.5ifc8.1 foam1.5ifc8.1gm Time Limit: –Can be set in tc.data as GLOBUS::jobtype=mpi,count=8,maxwalltime=210; –Can be set at vdl-gen step: TR FOAM( input RunParamsName, output DirAtmos, output DirOcean, output DirCoupl, input WorkDirName, none vds_workdir, none vds_group ) { argument = \${RunParamsName}; profile globus.maxWallTime = "210"; profile vds.workdir =\${vds_workdir}; profile vds.group=\${vds_group}; }

Post Processing jobs PostProc In tc.data: TG-SDSC PostProcess /users/nefedova/bin/PostProc INSTALLED INTEL32::LINUX ENV::PATH=/users/nefedova/bin:/usr/bin:/bin #! /bin/sh if /usr/bin/test ! $# -eq 4 then echo "Syntax: PostProc EntityName Finished Id F1_5" exit 1 fi cd history/$1 fileno=`expr $2 + 30` m2ncye.py $3 $fileno 1 $ mv *.nc../../ Echo Done M2ncye.py executable (and others that it calls) should be in the path

Transfer results to the storage This step is done automatically, the transfer nodes are generated by VDS. Tc.data file: TG-SDSC transfer /users/nefedova/bin/transfer INSTALLED INTEL32::LINUX ENV::GLOBUS_LOCATION=/usr/local/apps/globus gcc- r5;ENV::LD_LIBRARY_PATH=/usr/local/apps/globus gcc- r5/lib;ENV::PATH=/usr/local/apps/globus gcc- r5/bin;GLOBUS::maxwalltime=15 Pool.config file: –TG-SDSC pool: –Local pool:

Exitcode checking script To report the exitcode To create correct rescue DAG #! /bin/sh grep 'Done' $1 > /dev/null 2>/dev/null Pool.config: /home/nefedova/FOAM/SDSC- Production/exitcode

FOAM N-Member Ensemble Run under VDT Remote Directory Creation for Ensemble Member 1 Remote Directory Creation for Ensemble Member 2 Remote Directory Creation for Ensemble Member N FOAM run for Ensemble Member 1 FOAM run for Ensemble Member 2 FOAM run for Ensemble Member N Atmos Postprocessing Ocean Postprocessing for Ensemble Member 2 Coupl Postprocessing for Ensemble Member 2 Atmos Postprocessing for Ensemble Member 2 Coupl Postprocessing for Ensemble Member 2 Results are transferred back to the storage

Actual DAG (for 1 member- ensemble) ###################################################################### # GRIPHYN VDS CONCRETE DAG GENERATOR # DAG test # Index = 0, Count = 1 ###################################################################### JOB DirGen_ID DirGen_ID sub SCRIPT POST DirGen_ID /home/nefedova/FOAM/SDSC-Production/exitcode DirGen_ID out JOB FOAM_ID FOAM_ID sub SCRIPT POST FOAM_ID /home/nefedova/FOAM/SDSC-Production/exitcode FOAM_ID out JOB PostProcess_ID PostProcess_ID sub SCRIPT POST PostProcess_ID /home/nefedova/FOAM/SDSC-Production/exitcode PostProcess_ID out JOB PostProcess_ID PostProcess_ID sub SCRIPT POST PostProcess_ID /home/nefedova/FOAM/SDSC-Production/exitcode PostProcess_ID out JOB PostProcess_ID PostProcess_ID sub SCRIPT POST PostProcess_ID /home/nefedova/FOAM/SDSC-Production/exitcode PostProcess_ID out JOB new_rc_tx_PostProcess_ID000003_0 new_rc_tx_PostProcess_ID000003_0.sub SCRIPT POST new_rc_tx_PostProcess_ID000003_0 /home/nefedova/FOAM/SDSC-Production/exitcode new_rc_tx_PostProcess_ID000003_0.out JOB new_rc_tx_PostProcess_ID000004_0 new_rc_tx_PostProcess_ID000004_0.sub SCRIPT POST new_rc_tx_PostProcess_ID000004_0 /home/nefedova/FOAM/SDSC-Production/exitcode new_rc_tx_PostProcess_ID000004_0.out JOB new_rc_tx_PostProcess_ID000005_0 new_rc_tx_PostProcess_ID000005_0.sub SCRIPT POST new_rc_tx_PostProcess_ID000005_0 /home/nefedova/FOAM/SDSC-Production/exitcode new_rc_tx_PostProcess_ID000005_0.out PARENT DirGen_ID CHILD FOAM_ID PARENT FOAM_ID CHILD PostProcess_ID PARENT FOAM_ID CHILD PostProcess_ID PARENT FOAM_ID CHILD PostProcess_ID PARENT PostProcess_ID CHILD new_rc_tx_PostProcess_ID000003_0 PARENT PostProcess_ID CHILD new_rc_tx_PostProcess_ID000004_0 PARENT PostProcess_ID CHILD new_rc_tx_PostProcess_ID000005_0 ###################################################################### # End of DAG ######################################################################

How to run ?./vdl-gen inputfile 1 {2,3,4} vdlt2vdlx foam-gen.vdl > foam-gen.xml updatevdc -Dvds.properties=/path/to/properties foam- gen.xml gendax -Dvds.properties=/path/to/properties -verbose -o foam-gen.dax -L dax-dv-input (dax-dv-input is generated each time by vdl-gen, so it’s a good idea to delete it before each vdl-gen, so it does not get corrupted, etc) gencdag -Dvds.properties=/path/to/properties --verbose - force -dax foam-gen.dax -p TG-SDSC -o local condor_submit_dag test_0.dag