Introduction to SHIWA Technology Peter Kacsuk MTA SZTAKI and Univ.of Westminster

What is a multi-workflow simulation? A simulation workflow where nodes of the simulation are themselves workflows potentially based on different workflow languages A practical example: LINGA application (outGRID project) –Combining several workflows (2 CIVETs+FreeSurfer+STAT) –Heterogeneous workflow systems (LONI/MOTEUR) It also should enable the execution in different DCIs. LINGA application: –LONI Cluster (USA) –gLite-based neuGRID infrastructure (Europe) –CBRAIN HPC infrastructure (Canada) 2

Experiment setup 3 CBRAIN LONI Pipeline 151 input data items neuGRID LONI Pipeline 146 input data items CRANIUM LONI Pipeline EGI MOTEUR Outputs of both CIVETs

SHIWA solution for LINGA Sub-Workflows Management 4 Multi- Workflow

5 Start date: 01/07/2010 Duration: 27 months Total budget: 2,101,980 € Funding from the EC: 1,800,000 € Total funded effort in person-months: 231 Web site: Coordinator: Prof. Peter Kacsuk, SHIWA ( SHaring Interoperable Workflows for Large-Scale Scientific Simulations on Available DCIs) project

6 Motivations 1 In many cases large simulations are organized as scientific workflows that run on DCIs However, there are too many different WF formalism WF languages WF engines If a community selected a WF system it is locked into this system: They can not share their WFs with other communities (even in the same scientific field) They can not utilize WFs developed by other communities

WF Ecosystem 7

Who are the members of an e-science community from WF applications point of view? End-users (e-scientists) ( ) Execute the published WF applications with custom input parameters by creating application instances using the published WF applications as templates WF Application Developers ( ) Develop WF applications Publish the completed WF applications for end-users WF System Developers (50-100) Develop WF systems Writes technical, user and installation manuals

accessing a large set of various DCIs to make these WF applications run Clouds Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems E-science infrastructure What does a WF developer need? WF App. Repository Access to a large set of ready-to-run scientific WF applications Portal Using a portal/desktop to parameterize and run these applications, and to further develop them

accessing a single DCI to make these WF applications run Cluster based service grids (SG) (e.g. ARC) Grid system In the past: WF developers worked in an isolated way, on a single DCI Portal/desktop Using a portal/desktop to develop WF applications As a result if a community selected a WF system it is locked into this DCI Porting the WF to another DCI required large effort Parallel execution of the same WF in several DCIs is usually not possible

After SHIWA: Collaboration between WF application developers SHIWA App. Repository SSP Portal Clouds Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems Application developers Publish WF applications in the repository to be continued by other appl. developers Application developers use the portal/desktop to develop complex applications (executable on various DCIs) for various end-user communities

Project objectives Enable user communities to share their WFs –Publish the developed WFs –Access and re-use the published WFs –Build multi-workflows from the published WFs Toolset: –SHIWA Simulation Platform WF Repository (production) SHIWA Portal (production) SHIWA Desktop (prototype) 12

Coarse-grained interoperability (CGI) CGI = Nesting of different workflow systems to achieve interoperability of WF execution frameworks Multi-workflow

Export to IWIR Import from IWIR WF B WF A Interoperable Workflow Intermediate Representation IWIR Fine-grained interoperability (FGI)

Tools for CGI SHIWA services SHIWA repository to: –Describe workflows –Share workflows SHIWA portal to: –Access and enact registered workflows –Compose and enact multi-workflows –Monitor workflows and multi-workflows execution in various DCIs –Retrieve results of the execution

SHIWA Repository facilitates publishing and sharing workflows Supports: Abstract workflows with multiple implementations of over 10 workflow systems Storing execution specific data Available: from the SHIWA Portal standalone service at: repo.shiwa-workflow.eu

EGI Community Forum, Munich, March 28, 2012 Scenario: Find and test WFs SHIWA Repository : Analyze description, inputs and outputs of published WFs SHIWA Portal : Instantiate WF from repo, execute with given sample data (inside WS-PGRADE workflow used as the Master WF system) 17

18 Title: Work Package SA1...Author:. G Terstyanszky.. v.: SHIWA Portal: Workflow Editor

19 Title: Work Package SA1...Author:. G Terstyanszky.. v.: SHIWA Portal: Configuring Workflow

20 Title: Work Package SA1...Author:. G Terstyanszky.. v.: SHIWA Portal: Executing Workflow

21 SHIWA RepositorySHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WE + WF WF1WFm GEMLCA with GIB WF list WS- PGRADE Workflow engine WS-PGRADE Workflow editor edit WF s2 search WF s1 s5 s4 gLite DCI MOTEUR WE GWES WE Globus DCI pre-deployed- WEs MOTEUR WE Kepler WE Taverna WE Triana WE local cluster ASKALON WE SHIWA VO ASKALON WE researcher invoke WE s6 CGI User Scenario with WS-PGRADE as master SHIWA Proxy Server Proxy Server s3 s6 submit WE

Advantages for the various types of user communities using SHIWA WF system developers –Better visibility: much more WF developers can access and use their WF system than before (through the applications stored in the SHIWA repo) –The joint impact is much bigger than the individual WF systems can achieve WF developers –They can collaborate: share and re-use existing WF applications –WF application development can be accelerated –More complex WFs can be created in shorter time –They will access many different DCIs (their WF will be more popular) End-users –much bigger set of usable and more sophisticated WF applications –These applications can run on various DCIs 22

Conclusions SHIWA brings advantage for all the 3 kinds of user communities: –WF system developers –WF developers –End-users With relatively little effort –WF systems can join the SSP –WF system developers can adapt SHIWA technology Further information: 23