Running Workflows on Clouds and Grids Gabor Terstyanszky, University of Westminster T. Fahringer, P. Kacsuk, J. Montagnat, I. Taylor e-Science Workshop, Oxford, 15 March 2012 SHIWA is supported by the FP7 Capacities Programme under contract No. RI

2 Workflow Interoperability Challenge

33 Computer and Automation Research Institute, Hungarian Academy of Sciences MTA SZTAKIHungary University of InnsbruckUIBKAustria Charité - Universitätsmedizin BerlinC-UBGermany French National Centre for Scientific ResearchCNRSFrance University of WestminsterUOWUnited Kingdom Cardiff UniversityCUUnited Kingdom Academic Medical Centre of the University of AmsterdamAMCNetherlands University of Southern CaliforniaUSCUSA SHIWA Project Partners: Duration: July 2010 – June 2012 (extended to September 2012)

Project Objectives and Services Objectives: To create an environment which enables seamless execution of workflows of different workflow systems through workflow interoperability Services: To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows To provide coarse- and fine-grained workflow interoperability solutions To offer Distributed Computing Infrastructure interoperability solution To provide desktop computer and portal interfaces to manage workflows Key actors: researchersworkflow engine developers workflow developers 4

5 Workflow description abstract workflow plain text which describes the inputs and outputs and explains what the workflow does Workflow implementations concrete workflow It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine. Workflow configurations It contains pre-defined input files and parameters or references to these files and parameters and other data/metadata of a workflow. Workflow engines It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to them SHIWA Data Structure

6 J2 WF4 J1 J3 GEMLCA client GEMLCA Service Workflow Engine B Workflow of Workflow Engine B Workflows of Workflow Engine A Coarse-Grained Interoperability DCI native jobs (or workflows):J1, J2, J3 non-native workflows:WF4 - black boxes which are managed as legacy code applications Workflow Engine A

7 SHIWA Science GatewayResources native WEWS-PGRADElocal resources: invocation of locally deployed WEs portalWS-PGRADE v3.3.2WE submission to local cluster repositoryGEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitterGEMLCA with GIBWEs to gLite and Globus DCIs proxy managementSHIWA Proxy Server SHIWA Portal Legacy Code GUI SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor gLite DCI Globus DCI pre-deployed- WEs MOTEUR WE Kepler WE Taverna WE Triana WE local cluster MOTEUR WE GWES WE ASKALON WE SHIWA VO ASKALON WE GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server CGI Infrastructure ProActive WE

8 SHIWA Portal: Editing Workflows

9 SHIWA Portal: Configuring Workflows

10 SHIWA Portal: Executing Workflows

11 SHIWA Workflow Repository Browse Workflows Browse Implementations Search Sort

12 SHIWA Workflow Repository

13 SHIWA RepositorySHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WE + WF WF1WFm GEMLCA with GIB WF list WS-PGRADE Workflow engine WS-PGRADE Workflow editor edit WF s2 search WF s1 s5 s4 gLite DCI MOTEUR WE GWES WE Globus DCI pre-deployed- WEs MOTEUR WE Kepler WE Taverna WE Triana WE local cluster ASKALON WE SHIWA VO ASKALON WE user invoke WE s7 CGI User Scenario: Native WE Legacy Code GUI SHIWA Proxy Server Proxy Server s3 s7 submit WE s6 ProActive WE

14 SHIWA RepositorySHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB WS-PGRADE Workflow Engine WS-PGRADE Workflow Editor search WF s1 WE + WF s5 gLite DCI MOTEUR WE GWES WE Globus DCI MOTEUR WE Kepler WE Taverna WE Triana WE local cluster ASKALON WE SHIWA VO ASKALON WE user invoke WEs7 CGI User Scenario: Non-native WE Legacy Code GUI GEMLCA Client MOTEUR Workflow Engine MOTEUR Workflow editor s2 s3 GEMLCA UI SHIWA Proxy Server Proxy Server s6 s4 ProActive WE

15 DCI Interoperability JSDL Translator Workflow Engine DCI Bridge Metabroker Proxy Server DCI workflow for DCI B J2 J1 J4 J3 jobs in JSDL J2 J1 J4 J3 jobs in non-JSDL

16 DCI Bridge Features: web service based app which provides standard access to different DCIs: clouds - EC2, OpenNebula clusters - LSF, PSB desktop grids - BOINC, OurGrid, XtremWeb service grids - ARC, gLite, Globus, Unicore workflow jobs are submitted to the BES Factory Service of the DCI Bridge where jobs are described in JSDL Components: resource registry - configuration interface of DCIs application manager - BES management runtime system - job execution management via BES Factory monitoring - logs and messages handling and visualisation

17 DCI Bridge WS-PGRADE Workflow Engine WS-PGRADE Workflow Editor Portal BES Factory Service Job Registry Input Queue Upload Marnager Plug-in Manager Plug inA Plug inB Plug inC inputs & executables outputs clouds desktop grids service grids DCI Bridge proxy server s1 s2s3 s4 s5 s6 s7 s8 s10 s9 s11

18 Fine-Grained Interoperability front-end plug-in: converts workflow of workflow engine A into IWIR (Intermediate Workflow Interpreter Representation) back-end plug-in:converts from IWIR into workflow of workflow engine B IWIR workflow J2 J1 J4 J3 WFA ->IWIR frontend plug-in J2 J1 J4 Workflow of Workflow Engine A J3 WFA ->IWIR backend plug-in Workflow Engine B DCI J2 J1 J4 J3 Workflow of Workflow Engine B ASKALON MOTEUR PGRADE Triana

SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway WFn WE1WFp GEMLCA Repository WF1WFm gLite DCI Globus DCI DCIs ARC DCI Unicore DCI GEMLCA submitter Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE WS-PGRADE Workflow engine WS-PGRADE Workflow editor PGRADE WE pre-deployed- WEs SSP DCI & FGI Infrastructure SHIWA Science GatewayResources native WEASKALON, MOTEUR, WS-PGRADElocal WEs: Kepler, MOTEUR, Taverna & Triana WEs portalWS-PGRADE v3.4.1 submit to the local cluster repositoryGEMLCA + SHIWA remote WEs: ASKALON, GWES & MOTEUR WEs GEMLCA ARC, gLite, Globus and Unicore DCIs proxy server DCI Bridge Meta Broker Proxy Server JSDL Translator ProActive WE

20 SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway WFn WE1WFp GEMLCA Repository WF1WFm gLite DCI Globus DCI DCIs ARC DCI Unicore DCI WS-PGRADE Workflow engine DCI Interoperability Scenario DCI Bridge Meta Broker Proxy Server JSDL Translator search WF s1 s6 s4 s2 edit workflow s7 submit workflow s5 s3 WS-PGRADE Workflow editor

21 SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway WFn WE1WFp GEMLCA Repository WF1WFm DCIs WS-PGRADE Workflow Engine WS-PGRADE Workflow Editor FGI Interoperability Scenario c c DCI Bridge Meta Broker Proxy Server JSDL Translator search WF s1 Workflow Engine Workflow Editor s4 FP1 FP2 FPn c c BP1 BP2 BPm s3 s2 edit WF submit WF s7 gLite DCI Globus DCI ARC DCI Unicore DCI s5 s6

22 SHIWA Simulation Platform: Access SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) joint portal and repository account given by the science gateway administrator DCI resources certificate given by the relevant DCI administrator Access to the SHIWA Portal shiwa-portal.cpc.wmin.ac.uk/liferay-portal-6.05 Access to the SHIWA Repository shiwa-repo.cpc.wmin.ac.uk

23 Conclusions the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform the simulation platform provides access to major European e- infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources