1 P-GRADE Portal and GEMLCA Legacy Code Architecture Peter Kacsuk MTA SZTAKI

2 Outline Motivation of our work P-GRADE (Parallel Grid Run-time and Application Development Environment) Portal in a nutshell Workflow application support Parameter sweep application support GEMLCA P-GRADE Portal and Grid interoperability Conclusions

3 E-scientists concerns The P-GRADE Grid Portal gives you all the answers! How to concentrate on my own research if the technology I would like to use is in continuous change? Which is the most suitable grid for me? How can I learn and understand the usage of that technology? How can I develop applications? How can I execute applications? How to tackle performance issues? How to use several Grids at the same time? How to migrate my application from one grid to another? How can I collaborate with fellow researchers?

4 Easy-to-use, middleware-neutral workflow-oriented Grid portal Graphical tools for workflow-oriented application development, execution and monitoring Sequential & parallel components can be integrated into large Grid applications Direct or broker-based resource allocation The P-GRADE Portal hides differences of Grids Globus 2 and 4 – LCG2 – gLite Grid interoperability Transparent switching between Grid technologies Your code does not have to contain grid specific calls

5 Proxy management P-GRADE portal in a nutshell Grid and Grid resource management Graphical editor for workflow creation Job mapping to Grid and Grid resources Workflow management and execution visualization

6 What is a P-GRADE Portal workflow? a directed acyclic graph where –Nodes represent jobs (batch programs to be executed on a computing element) –Ports represent input/output files the jobs expect/produce –Arcs represent file transfer operations and job dependencies semantics of the workflow: –A job can be executed if all of its input files are available

7 The typical user scenario Part 1 - development phase Certificate servers Portal server Grid services START EDITOR OPEN & EDIT or DEVELOP WORKFLOW SAVE WORKFLOW

8 Certificate servers Portal server Grid services TRANSFER FILES, SUBMIT JOBS DOWNLOAD (SMALL) RESULTS The typical user scenario Part 2 - execution phase VISUALIZE JOBS and WORKFLOW PROGRESS MONITOR JOBS DOWNLOAD PROXY CERTIFICATES

9 Demo Workflow editing Workflow submission Workflow execution monitoring Examples: –VO site test –Equation Solver solving the equation A*x = B

10 Users feedback Users want parameter study (PS) support at workflow level It means: –If the user has an existing workflow in a repository, he would like to run the same workflow (without any change) with many different parameters Goal of the work: –Enable users to easily turn their workflow application into a parameter sweep application

11 Parameter sweep (PS) workflow execution based on the black box concept PS port: 4 instances of the input file PS port: 3 instances of the input file 1 PS workflow execution = 4 x 3 normal workflow execution This provides the 3 rd level of parallelism resulting a very large demand for Grid resources

12 Introducing three levels of parallelism Each job can be a parallel program – Parallel execution inside a workflow node – Parallel execution among workflow nodes Multiple jobs run parallel – Parameter study execution of the workflow Multiple instances of the same workflow with different data files

13 PS_1_2 PS_1_1 PS_2_2 PS_2_1 PS_1_1PS_2_1PS_1_2PS_2_1PS_1_1PS_2_2PS_1_2PS_2_2 WF One or more Parameter Sets are defined. Each has different parameters. (Sets are represented by remote file directories, parameters by files) PS DATA FLOW Content of Parameter Sets are combined (Descartes product) The workflow is executed as many times as many elements are in the Descartes product of input sets Result_1_1 Result_2_1 Result_1_2 Result_2_2 Store the results of the calculations (Results are represented by remote files of one or more common directories)

14 Data Model of advanced PS Gen A_Gen Input Coll Output Coll Output Coll Output The Autogenerator is a special job: May have no input Runs on the Portal

15 Implementation of the example Auto Generator

16 Parameter study workflow GEN SEQ COLL SEQ Parameter sweep grid jobs Grid job generates input parameter space Collector grid job evaluates the results of the simulation P-GRADE Portal

17 Summary of PS execution First phase: executing ones all the Generators Last phase: executing ones all the Collectors Second phase: executing all generated eWorkflows in parallel

18 Workflow Management Window showing Generator and Collector Collector phase is inactive until each eWorkflow has terminated

19 PS-portal Demo PS-Workflow editing PS-Workflow submission PS-Workflow execution monitoring Example: –Equation Solver

20 GEMLCA/P-GRADE Portal Motivations Need for using legacy codes in the Grid –Code from the past, maintained because it works –Often supports business critical functions –Complex dependencies, licence and/or hardware restrictions cannot be (modified) gridified GEMLCA – Grid Execution Management for Legacy Code Architecture –To deploy legacy code applications as Grid services without reengineering the original code and with only minimal user effort GEMLCA/P-GRADE Portal integration –To create Grid workflows where components can also be legacy code applications –To make these functions available from a Grid Portal

21 Publish legacy applications as application services Grid(s)/VO(s) Legacy applications Grid 1 Grid 2... GEMLCA P-GRADE Portal Server Desktop 1 Web browser LC LC GEMLCA Administration Tool portlet

22 Integrate legacy application services into your workflow Grid(s)/VO(s) Legacy applications Grid 1 Grid 2... GEMLCA P-GRADE Portal Server Desktop 2 Web browser Result Input LC 1 LC 2

23 Combine legacy codes with new codes inside the same workflow! Code invocation Job submission Combining legacy and non-legacy (job) components Publish successfully executed new code as legacy code

24 Sharing jobs with other GEMLCA portal users New button for sucessfully executed jobs: create a legacy code service from this jobs

25 GEMLCA P-GRADE Portal server Sharing jobs with other GEMLCA portal users Job Repository Put job into Portal level repository

26 Grid interoperability 1 P-GRADE portal supports simultaneous job submission in several Grids: –GT2 (OSG, TeraGrid and UK NGS) –LCG-2, gLite – EGEE, SEEGRID –NorduGrid P-GRADE Portal enables: –Connecting production Grids and make them interoperable at workflow level

27 User P-GRADE Portal WPI Portal Server Connecting various production Grids gLite broker Certificate server Proxy Workflow ANL Fermi Lab OSG (GT2) SEEGRID (gLite) Job Budapest Istambul Athens Supports both direct and brokered job submission

28 Grid interoperability 2 P-GRADE Portal with GEMLCA supports legacy code service invocation in 2 nd and 3 rd generation Grids: –GT2, GT4 – (OSG, TeraGrid and UK NGS) –LCG-2, gLite – (EGEE, SEEGRID) Combination of jobs and legacy code services enables GEMLCA/P-GRADE Portal to support: –Extending 2 nd generation Grids with legacy code services –Extending 2 nd generation Grids with GT4 sites –Connecting 2 nd generation and 3 rd generation Grids

29 Connecting 2 nd generation and 3 rd generation production Grids Job Repository (GEMLCA) Fermi Lab User ANL GEMLCA P-GRADE Portal GIN Portal Server Executable OSG (GT2) WestFocus GT4 Grid UoW Brunel Poznan Budapest EGEE LCG/gLite Executable Service Invocation Executable

30 GIN VO Portal

31 References P-GRADE Portal service is available for –SEE-GRID infrastructure –Central European VO of EGEE –GILDA: Training VO of EGEE –Many national Grids (UK National Grid Service, HunGrid, etc.) –US Open Science Grid, TeraGrid –Economy-Grid, Swiss BioGrid, Bio and Biomed EGEE VOs, BioInfoGrid, BalticGrid –GIN VO (see the presentation at the EGEE, TeraGrid and Hungarian Grid booths)

32 P-GRADE portal installations

33 How to learn the P-GRADE portal? –Take a look at (manuals, slide shows, installation procedure, etc.) –Visit or request a training event! (event list also on homepage) Lectures, demos, hands-on tutorials, application development support –Get an account for one of its production installations: OSG portal - WPI SEEGRID portal – SZTAKI GILDA portal – SZTAKI NGS portal – University of Westminster –If you are the administrator of a Grid/VO then contact SZTAKI to get your own P-GRADE Portal! –If you know the administrator of a P-GRADE Portal you can ask him/her to give access to your Grid through his/her portal installation! (Multi-Grid portal)

34 Thank you! Learn once, use everywhere - portal Develop once, execute anywhere - application