1 Support for Parameter Study applications in the P-GRADE Portal Cevat Şener Dept. Of Computer Engineering, METU

2 General structure of a Parameter Study (PS) applications Algorithm 2 SPMD parallelism: Single Program Multiple Data

3 Advanced PS applications Algorithm 2 Algorithm 1 Cut input into smaller pieces

4 Advanced PS applications Algorithm 2 Algorithm 1 Algorithm 3 Cut input into smaller pieces Aggregate result

5 PS applications in P-GRADE Portal 2.5 Complete workflow Files in the same LFC catalog (e.g. /grid/gilda/sipos/myinputs ) Results produced in the same catalog

6 Advanced PS applications in P-GRADE Portal 2.5 Generator component(s) Initial input data Generate or cut input into smaller pieces Collector component(s) Aggregate result Files in the same LFC catalog (e.g. /grid/gilda/sipos/myinputs ) Results produced in the same catalog Complete workflow

7 Each job can be a parallel program – Parallel execution inside a workflow node (SIMD/MIMD/MISD) – Parallel execution among workflow nodes (SIMD/MIMD/MISD) Multiple jobs run parallel – Parameter study execution of the workflow (SIMD) Multiple instances of the same workflow process different data files Third level of parallelism

8 Turning a WF into a PS By turning at least of the open input ports into a “PS Input port” the WF is turned into a Parameter Study

9 Turning a WF into a PS /grid/gilda/sipos/InputImages Image.0 Image.1 /grid/gilda/sipos/XCoordinates XCoordinate.0 XCoordinate.1 /grid/gilda/sipos/YCoordinates YCoordinate.0 YCoordinate.1 /grid/gilda/sipos/Output ImagePart.0 ImagePart x 2 x 2 = 8 execution of the whole workflow

10 PS workflows PS port: 4 instances of the input file PS port: 3 instances of the input file 1 PS workflow execution = 4 x 3 normal workflow execution This provides the 3 rd level of parallelism resulting a very large demand for Grid resources

11 PS workflow execution

12 PS workflow detailed view eWorkflows Total = Init + Submitted + Rescue + Error + Finished

13 Generators Auto generator Pre defined program logic (static binary) Generates text files User can control text file content by templates and patterns Custom generator User provides generator program logic Useful to generate binary content (e.g. image files, audio files, …) Generate input files for parameter study workflows Saves these files on SEs, register them with LFNs into the LFC catalog

14 Collector Collects output files and perform collective operation on them. E.g. –Standard deviation –Average –Statistics –Evaluation and find the “best” result –… User provides the program logic Portal provides data transfer –Refer in your code to input files as local files –No need to use any Grid API in your code

15 Workflow management window showing Generator and Collector Collector phase is inactive until each eWorkflow has terminated

16 Workflow management window showing Generator and Collector The Output(s) of the collector(s) can be downloaded.

17 The generic PS workflow application First phase: Generators executed Third phase: collectors executed Second phase: eWorkflows executed

18 Introduction to the practicals

19 Hands-on exercise Matrix multiplication Multiplication job Matrix

20 Hands-on exercise Matrix multiplication Multiplication job Auto generator Input files registered in the LFC catalog X 9 <= X <=15, step Matrix2 Output files registered in your the LFC catalog

21 Hands-on exercise Turn the matrix operations WF into a PS AB[*,0] T x AB[*,1] AB A * B A * B [ *, 0 ]A * B [ *, 1 ] B ( A * B [ *, 0 ] T ) * ( A * B [ *, 1 ] ) A * B [ *, 0 ] T