Peter Kacsuk MTA SZTAKI kacsuk@sztaki.hu SHIWA technology to enable the integration and collaboration of various workflow systems used in Europe Peter Kacsuk MTA SZTAKI kacsuk@sztaki.hu

Who are the members of an e-science community from WF applications point of view? WF System Developers (20-40) Develop WF systems Writes technical, user and installation manuals WF Application Developers (500-1000) Develop WF applications Publish the completed WF applications for end-users End-users (e-scientists) (5000-50000) Execute the published WF applications with custom input parameters by creating application instances using the published WF applications as templates 2

What does a WF developer need? Portal Using a portal/desktop to parameterize and run these applications, and to further develop them WF App. Repository Access to a large set of ready-to-run scientific WF applications accessing a large set of various DCIs to make these WF applications run Clouds Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems E-science infrastructure

WF developers worked in an isolated way in the past Portal/desktop Using a portal/desktop to develop WF applications accessing a single DCI to make these WF applications run Cluster based service grids (SG) (ARC.) Grid systems E-science infrastructure

Collaboration between WF application developers based on SHIWA technology Publish WF applications in the repository to be continued by other appl. developers Application developers use the portal/desktop to develop complex applications (executable on various DCIs) for various end-user communities SHIWA App. Repository Application developers SSP Portal Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems Local clusters Supercomputers Clouds

SHIWA Simulation Platform

Advantages for the various types of user communities using SHIWA WF system developers Better visibility: much more WF developers can access and use their WF system than before (through the applications stored in the SHIWA repo) The joint impact is much bigger than the individual WF systems can achieve WF developers They can collaborate: share and re-use existing WF applications WF application development can be accelerated More complex WFs can be created in shorter time They will access many different DCIs (their WF will be more popular) End-users much bigger set of usable and more sophisticated WF applications These applications can run on various DCIs

What to do for a WF system developer to use SHIWA technology? CGI approach using the SSP Provide the WF engine as a legacy code for SHIWA developers SHIWA developers will make it usable inside SSP Estimated work: 1 month Result: WF system A WFs can be used as components WFs in the meta-workflows constructed by the SSP

WF Engine A integrated into the SSP SHIWA Science Gateway SHIWA VO SHIWA Portal SHIWA Repository WF1 WFn gLite DCI Globus DCI local cluster GEMLCA admin GEMLCA Repository Triana WE WS-PGRADE Workflow editor WF1 WFm Taverna WE WE1 WEp WS-PGRADE Workflow engine MOTEUR WE GEMLCA with GIB MOTEUR WE GWES WE ASKALON WE Kepler WE GEMLCA Service ASKALON WE SHIWA Proxy Server Proxy Server WF A WE

What to do for a WF system developer to use SHIWA technology? CGI approach making WF system A as the native WF development and execution environment Connect GEMLCA to WF system A Estimated work: ~2 months Result: In WF system A meta-workflows can be constructed and executed as now in the SSP

WF system A as the native WF dev. environment SHIWA VO local cluster gLite DCI Globus DCI SHIWA Repository WF1 WFn search WF s1 Triana WE Taverna WE GEMLCA Repository MOTEUR WE WF1 WFm WE + WF s5 WE1 WEp ASKALON WE researcher Kepler WE ASKALON WE MOTEUR WE GWES WE s2 invoke WE s6 pre-deployed- WEs GEMLCA with GIB WF A Workflow editor s4 GEMLCA Service s3 s4 WF A Workflow Engine GEMLCA Client GEMLCA UI 11

What to do for a WF system developer to use SHIWA technology? DCI Bridge approach enables WF system A to run applications on various DCIs Connect WF system A engine to DCI Bridge service WF system A engine should be able to submit jobs as BES jobs using JSDL Estimated work: We do not know yet how big work Result: In WF system A workflows can run on various DCIs

Using DCI Bridge by WF System A JSDL job submission DCI Bridge Clouds Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems Various DCIs will be usable

What to do for a WF system developer to use SHIWA technology? SHIWA WF Bundle enables common WF representation in WF repositories Implement SHIWA WF bundle in order to easily store your WF in the SHIWA WF repositories (and in other WF repositories, e.g. myExperiment) Make various WF repositories interoperable Estimated work: 1-2 months Result: WF system A workflows will be easily accessible by other WF systems

What to do for a WF system developer to use SHIWA technology? SHIWA IWIR enables common WF representation to transform WF A into WF B Implement WF A to IWIR translator Implement IWIR to WF A translator Estimated work: 2-3 months each Result: WF system A workflows will be easily executable by other WF systems Workflows written in other WF systems will be easily executable by WF system A

Conclusions SHIWA brings advantage for all the 3 kinds of user communities: WF system developers: higher visibility WF developers: sharing and re-using WFs End-users: much bigger set of usable WF applications These applications can run on various DCIs With relatively little effort WF system developers can adapt SHIWA technology Or WF systems can join the SSP