SHIWA: Is the Workflow Interoperability a Myth or Reality PUCOWO, June 2011, London Gabor Terstyanszky, Tamas Kiss, Tamas Kukla University of Westminster

2 Workflow Interoperability Challenge

3 SHIWA Aims and Objectives E-scientists Aims: seamlessly “recycle” existing workflows of different workflow systems Objectives: To be able to search and browse available workflows To be able to execute non-native workflows and create and submit embedded workflows Workflow and workflow engine developers Aims: publish workflows to enable their sharing Objectives: To be able develop and test workflows To be able upload and share workflows

4 SHIWA Workflow Interoperability Solution

55 SHIWA concept of coarse-grained workflow interoperability WFs are black boxes and we do not interpret them Only inputs and outputs are known WFs are executed by their engines remotely Such WFs can be embedded as jobs into WFs of other systems forming embedded or heterogeneous meta WFs. Native Job MOTEUR WF Askalon WF 021 3

SHIWA Simulation Platform v0 (SSP v0):October 2010 portal:PGRADE repository: GEMLCA Legacy Repo workflow:native – PGRADE workflow engines are legacy codes and workflows are their parameter SHIWA Simulation Platform v1 (SSP v1):March 2011 repository: SHIWA Workflow Repo both workflow engines and workflows are legacy codes SHIWA Simulation Platform v2 (SSP v2):March 2012 portal:WS-PGRADE 3.3 repository: seamless data transfer among the GEMLCA and SHIWA repos workflow:native – ASKALON + MOTEUR SHIWA Simulation Platform (SSP)

SSP v1: Architecture SHIWA RepositorySHIWA Portal WF1 GEMLCA admin SHIWA Science Gateway gLite DCI WFn WE1WEp GEMLCA Repository WF1WFm MOTEUR WE PGRADE Workflow engine PGRADE Workflow editor GWES WE Globus DCI pre-deployed- WEs MOTEUR WE Kepler WE Taverna WE Triana WE local cluster ASKALON WE SHIWA VO ASKALON WE GEMLCA Service GEMLCA with GIB

SSP v1: SHIWA Science Gateway SHIWA RepositorySHIWA Portal WF1 GEMLCA admin SHIWA Science Gateway WFn WE1WEp GEMLCA Repository WF1WFm PGRADE Workflow engine PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Portal: P-GRADE 2.4 portal technology certificate/proxy and DCI resource management access to different DCI information systems integrated with the P-GRADE Workflow System (used as native workflow engine) administration of GEMLCA services GEMLCA Service: converts legacy applications such as workflows and workflow engines into Grid services invokes locally or remotely pre-deployed workflow engines or submits workflow engines to local or remote resources to execute workflows GEMLCA Repository workflow engine (WE) and workflow (WF) data supporting execution SHIWA Repository: create, add, edit and delete workflow metadata upload and download workflows with their implementations and configurations.

Workflow description plain text describing inputs and outputs and explains what the workflow does Workflow implementations all implementation files or references to these files data/metadata necessary to run the workflow on its associated workflow engine. Workflow configurations pre-defined input files and parameters or references to these, and other data/metadata of a workflow. Workflow engines files and other data/metadata necessary to execute a workflow engine on a grid site or references to them SHIWA Repository Data Structure

SHIWA Workflow Repository

SHIWA RepositorySHIWA Portal WF1 GEMLCA admin SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WE+ LCID WF1WFm GEMLCA with GIB PGRADE Workflow engine PGRADE Workflow editor WE developer create LCID upload WE s1 s2 Actor: WE developer Frequency: once for a particular version of a WE Process: step 1 - create and upload workflow engine’s legacy code description (LCID) using the GEMLCA admin portlet. step 2 - upload the workflow engine’s binary to the GEMLCA Repository (WE 1, …WE p ) SSP v1: User scenarios - Workflow engine upload

Actor: WF developer Frequency: once for a particular version of a WF Process: step 1 - create and upload the workflow’s LCID description using the GEMLCA admin portlet step 2 - upload the workflow descriptor file to the GEMLCA Repository (WF1, …WFm). step 3 - define the workflow metadata using the SHIWA Repository GUI step 4 - upload the workflow descriptor file to SHIWA Repository (WF1, …WFn). SSP v1: User scenarios - Workflow upload SHIWA RepositorySHIWA Portal WF1 GEMLCA admin SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF + LCID WF1WFm GEMLCA with GIB PGRADE Workflow engine PGRADE Workflow editor WF developer create LCID upload WF create WF metadata upload WF s1 s2 s3 s4

SSP v1: User scenarios – workflow editing & execution SHIWA RepositorySHIWA Portal WF1 e-scientist SHIWA Science Gateway invoke WE GEMLCA Service WFn WE1WEp search WF GEMLCA Repository edit WF WE + WF WF1WFm GEMLCA with GIB s1 s2 s3 s5 s2 s4 WF list PGRADE Workflow engine PGRADE Workflow editor gLite DCI MOTEUR WE GWES WE Globus DCI pre-deployed- WEs MOTEUR WE Kepler WE Taverna WE Triana WE local cluster ASKALON WE SHIWA VO ASKALON WE GEMLCA admin step 1 – browse/search the SHIWA repository and select workflows to execute step 2 - create embedded workflows by adding the selected non-native workflows using their IDs step 3 - P-GRADE workflow engine submits the embedded workflow through GEMLCA Service step 4 - GEMLCA Service combines the workflow and its corresponding workflow engine. step 5 - the GEMLCA Service invokes the pre-deployed workflow engine to execute the workflow or submits the workflow engine with the workflow submit WE s5

Register with the SHIWA VO: aim: to get authorisation to access GEMLCA resource, WF engines, compute and data resources Get an account to the SHIWA portal aim: to get access to the portal Register with the SHIWA repository aim: to be authorised to upload and modify workflows repository can be browsed and workflows can be downloaded without account Detailed how-to-get-access guide: SSP v1 – How to get access?

15 SHIWA Repository SHIWA Portal WF1 GEMLCA admin SHIWA Science Gateway WFn WE1WFp GEMLCA Repository WF1WFm GEMLCA client c c gLite DCI Globus DCI ARC DCI Unicore DCI GEMLCA submitter Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE MOTEUR GEMLCA UI ASKALO N WE ASKALON GEMLCA UI PGRADE Workflow engine PGRADE Workflow editor PGRADE WE pre-deployed- WEs DCI Bridge SSP v2: Architecture

16 SHIWA Repository SHIWA Portal WF1 GEMLCA admin SHIWA Science Gateway WFn WE1WFp GEMLCA Repository WF1WFm GEMLCA client GEMLCA submitter PGRADE Workflow engine PGRADE Workflow editor DCI Bridge SSP v2: Science Gateway SHIWA Portal: WS-PGRADE 3.3 DCI Bridge: access to multiple DCIs SHIWA Repository: seamless data transfer among the GEMLCA and SHIWA repositories

17 SSP v2: User scenarios – workflow editing & execution SHIWA Repository SHIWA Portal WF1 e-scientist GEMLCA admin SHIWA Science Gateway access non- native WF WFn WE1WFp GEMLCA Repository edit WF WE + WF WF1WFm GEMLCA client s2 s3 s4 c c gLite DCI Globus DCI DCIs ARC DCI Unicore DCI search WF s1 GEMLCA submitter Kepler WE GWES WE Taverna WE Triana WE invoke WE s5 MOTEUR WE MOTEUR GEMLCA UI ASKALO N WE ASKALON GEMLCA UI PGRADE Workflow engine PGRADE Workflow editor PGRADE WE pre-deployed- WEs