1 Support for parameter study applications in the P-GRADE Portal Gergely Sipos MTA SZTAKI (Hungarian Academy of Sciences)

2 General structure of a PS applications Algorithm 2

3 Advanced PS applications Algorithm 2 Algorithm 1 Cut input into smaller pieces

4 Advanced PS applications Algorithm 2 Algorithm 1 Algorithm 3 Cut input into smaller pieces Aggregate result

5 PS applications in P-GRADE Portal 2.5 Complete workflow Files in the same LFC catalog (e.g. /grid/gilda/sipos/myinputs ) Results produced in the same catalog

6 Advanced PS applications in P-GRADE Portal 2.5 Generator component(s) Initial input data Generate or cut input into smaller pieces Collector component(s) Aggregate result Files in the same LFC catalog (e.g. /grid/gilda/sipos/myinputs ) Results produced in the same catalog Complete workflow

7 Each job can be a parallel program – Parallel execution inside a workflow node (SIMD/MIMD/MISD) – Parallel execution among workflow nodes (SIMD/MIMD/MISD) Multiple jobs run parallel – Parameter study execution of the workflow (SIMD) Multiple instances of the same workflow process different data files Third level of parallelism

8 Turning a WF into a parameter study By turning at least of the open input ports into a “PS Input port” the WF is turned into a Parameter Study

9 Turning a WF into a parameter study /grid/gilda/sipos/InputImages Image.0 Image.1 /grid/gilda/sipos/XCoordinates XCoordinate.0 XCoordinate.1 /grid/gilda/sipos/YCoordinates YCoordinate.0 YCoordinate.1 /grid/gilda/sipos/Output ImagePart.0 ImagePart x 2 x 2 = 8 execution of the whole workflow

10 Parameter sweep (PS) workflows PS port: 4 instances of the input file PS port: 3 instances of the input file 1 PS workflow execution = 4 x 3 normal workflow execution This provides the 3 rd level of parallelism resulting a very large demand for Grid resources

11 PS Workflow execution

12 PS workflow detailed view - eWorkflows Total = Init + Submitted + Rescue + Error + Finished

13 Generators Auto generator Pre defined program logic (static binary) Generates text files User can control text file content by templates and patterns Custom generator User provides generator program logic Useful to generate binary content (e.g. image files, audio files, …) Generate input files for parameter study workflows Saves these files on SEs, register them with LFNs into the LFC catalog

14 Collector Collects output files and perform collective operation on them. E.g. –Standard deviation –Average –Statistics –Evaluation and find the “best” result –… User provides the program logic Portal provides data transfer –Refer in your code to input files as local files –No need to use any Grid API in your code

15 Workflow Management Window showing Generator and Collector Collector phase is inactive until each eWorkflow has terminated

16 Workflow Management Window showing terminated Generator and Collector The Output(s) of the collector(s) can be downloaded.

17 The generic PS workflow application First phase: Generators executed Third phase: collectors executed Second phase: eWorkflows executed

18 Introduction to the practicals

19 Hands-on exercise Matrix multiplication PS Multiplication job Matrix

20 Hands-on exercise Matrix multiplication PS Multiplication job Auto generator Input files registered in your GILDA LFC catalog X 9 <= X <=15, step Matrix2 Output files registered in your GILDA LFC catalog

21 Hands-on exercise 2 Turn the matrix operations WF into a parameter study AB[*, 0] T * AB[*, 1] AB A * B A * B [ *, 0 ]A * B [ *, 1 ] B ( A * B [ *, 0 ] T ) * ( A * B [ *, 1 ] ) A * B [ *, 0 ] T

22 Infrastructure for the hands-on Use the same account that during the workflow hands-on yesterday! GILDA P-GRADE Portal (SZTAKI, Budapest) based on v2.5 Machines at the school Proxy download GILDA Grid services GILDA MyProxy server GSI + gLite Proxy 1 Use the same account that during the workflow hands-on yesterday!

23 Open the parameter study practicals from the agenda page Use the same portal and MyProxy accounts than yesterday!