1. Sharing, integrating and executing different workflows in heterogeneous multi-cloud systems Peter Kacsuk MTA SZTAKI kacsuk@sztaki.hu SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI-283481 1

2. 2 Motivations 1 In many cases large simulations are organized as scientific workflows that run on Distributed Computing Infrastructures (DCIs) However, there are too many different WF formalism WF languages WF engines If a community selected a WF system it is locked into this system: They can not share their WFs with other communities (even in the same scientific field) They can not utilize WFs developed by other communities

3. 3 Motivations 2 A WF system engine is typically connected to one particular DCI (Distributed Computing Infrastructure) As a result, if a community selected a WF system it is locked into this DCI Porting the WF to another DCI requires extra effort Parallel execution of the same WF in several DCIs is usually not possible

4. Part of WF Ecosystem 4

5. er What do we want to achieve? 5 XSE DE BOIN C Amaz on Bio1 Bio2 BioN Cyberspace Workflows Infrastructures Users should be able to access and use any WF and any infrastructure in an interoperable way no matter which is their home WF system Taverna Galaxy Kepler WF systems

6. What does WF interoperability mean? If a user developed WF Y in WF system B she (or other users) should be able to 1.Re-use WF Y as part of another WF (e.g. WF X) developed in WF system A (Coarse- grained interoperability – CGI) 6

7. Coarse-grained interoperability CGI Coarse-grained interoperability (CGI) = Nesting of different workflow systems to achieve interoperability of WF execution frameworks DCI 1 DCI 2 DCI 3 7 A Y

8. Features of CGI approach Advantages –No restrictions on the embedded WFs –You can run the embedded WFs in their native DCI (even in parallel -> easy to achieve high degree of DCI parallelism) –Easy to implement and connect a new WF system Drawbacks –Black-box approach: you cannot modify the embedded workflow control and observe the internal operation of the embedded WF 8

9. What does WF interoperability mean? If a user developed WF Y in WF system B she (or other users) should be able to 1.Run WF Y under another WF system (e.g. WF system A) (Fine-grained interoperability – FGI) 2.Further develop WF Y under another WF system (e.g. WF system A) (FGI) 9

10. Transform WF Y into IWIR WF Y Interoperable Workflow Intermediate Representation (IWIR) Transform IWIR into WF X WF X Fine-grained interoperability (FGI) or white box WF interoperability: Enables to transform one WF to another WF system and further develop it in the new system 10

11. Features of FGI approach Advantages –White-box approach: you can modify the embedded workflow control and observe the internal operation of the embedded WF Drawbacks –There are some restrictions on the WFs that can be transformed –You can run the embedded WFs only in the native DCIs of the target WF system –Not easy to implement and connect a new WF system 11

12. What does infrastructure interoperability mean? If a user developed WF X in WF system A she (or other users) should be able to –Run WF X in any DCI without significant porting effort –Run different nodes of WF X in different DCIs (if these nodes are in parallel branches then they can simultaneously run in different DCIs) 12 Cloud1 Cloud N

13. EU Projects that develop solutions for these goals SHIWA –To solve WF and DCI interoperability issues –Duration: 2 years (July 2010 – June 2012) SCI-BUS –To provide the required gateway technology –Duration: 3 years (Oct 2011 – Sep 2014) 13

14. accessing a large set of various DCIs to make these WF applications run Clouds Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems E-science infrastructure What does a WF developer need? WF App. Repository Access to a large set of ready-to-run scientific WF applications Portal Using a portal/desktop to parameterize and run these applications, and to further develop them

15. Reference production infrastructure of SHIWA SHIWA App. Repository Application developers Publish WF applications in a repository to be continued/used by other appl. Developers SHIWA Portal Clouds Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems Use the portal/desktop to develop complex applications (executable on various DCIs) based on WFs stored in the repository

16. facilitates publishing and sharing workflows Supports: Abstract workflows with multiple implementations of 10 workflow systems Storing execution specific data Available: from the SHIWA Portal standalone service at: repo.shiwa-workflow.eu SHIWA Repository 16

17. SHIWA Bundle and SHIWA Desktop for WF interoperability 17 SHIWA Bundle SHIWA App. Repository WS-PGRADE Triana MOTEUR ASKALON SHIWA Desktop

18. SHIWA Bundle and SHIWA Desktop SHIWA Bundle: –object (stored as a zip file) containing everything needed to expose a workflow for use –Provides a common language/format for workflow engines Workflows are stored as SHIWA bundle in the SHIWA Repository SHIWA Desktop connects a user’s desktop workflow environment to the SHIWA Repository 18

19. Extension of WF interoperability with DCI interoperability 19 SHIWA Bundle SHIWA App. Repository SHIWA Desktop WS- PGRADE SHIWA Desktop Triana SHIWA Desktop MOTEUR SHIWA Desktop ASKALON Cloud Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) Supercomputer based SGs Grid systems DCI Bridge BES interface

20. Production service JSDL Translator Workflow Engine DCI Bridge workflow for DCI B J2 J1 J4 J3 jobs in non-JSDL J2 J1 J4 J3 jobs in JSDL DCI n DCI 1 Accessing DCI Bridge 20 BES requires JSDL for job submission Therefore we need a JSDL generator to help WF engines to create the JSDL for the jobs generated for WF nodes

21. Extension of WF interoperability with DCI interoperability (2) 21 SHIWA Bundle SHIWA App. Repository SHIWA Desktop WS- PGRADE SHIWA Desktop Triana SHIWA Desktop MOTEUR SHIWA Desktop ASKALON Cloud Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) Supercomputer based SGs Grid systems DCI Bridge BES interface JSDL Translator

22. Where are we? Workflow interoperability done by –SHIWA Bundle –SHIWA Desktop –SHIWA Repository DCI interoperability done –DCI Bridge –JSDL Translator All of them are production services What else do we need? –A reference service through which anyone can try the technology The reference service is the SHIWA portal 22

23. SHIWA portal: WS-PGRADE/gUSE Generic-purpose gateway framework 23 Based on Liferay General purpose Workflow-oriented portal framework Supports the development and execution of workflow-based applications Enables the multi-cloud, multi-DCI execution of any WF Provides access to internal repository external SHIWA Repository

24. Creating and running WS-PGRADE workflows 24 Step 1: Edit workflow

25. Step 2: Configuring the workflow Cloud1Cloud N

26. Step 3: Running workflow instance 26

27. Scalable architecture based on collaborating services

28. Seamless access to various types of DCIs WF Graph editor WEB-UI (HTML) Liferay WS- PGRADE portal Liferay WS- PGRADE portal Information System WF Storage File Storage WF Repository WF Interpreter GT5 plugin Glite plugin ARC plugin Unicore plugin BOINC plugin GT5 Grid DCI-Bridge Client machine Portal Server machine DCIs BES interface BOINC Grid ARC Grid Cloud plugin Cloud Broker

29. WFs in the clouds This issue is solved by the SCI-BUS project by integrating WS-PGRADE/gUSE with CloudBroker Platform Motivation: –Cloud resources are getting more and more popular –Clouds are more reliable than grids –WFs with cloud access are capable of satisfying compute needs of complex scientific computations –Clouds can provide a vast amount of resources Aim: –Provide access to cloud resources in a transparent way 29

30. WS-PGRADE/gUSE and SCI-BUS 30

31. Multi-cloud Integrated WS-PGRADE/CloudBroker Platform to access multi-clouds Cloud Broker Platform WS- PGRADE n IaaS Cloud 1 IaaS Cloud N SEQ WS- PGRADE 1 Supported clouds: Amazon, IBM, OpenStack, Eucalyptus, OpenNebula SaaS solution: Preregistered services/jobs can run from WS-PGRADE (Supported from gUSE 3.5.0) IaaS solution: any services/jobs can be submitted from WS-PGRADE (Supported from gUSE 3.5.1) 31

32. CloudBroker Platform Web-based application repository for the deployment and execution of scientific and technical software in the cloud Offers these stored applications as SaaS service for end users On demand, pay per use, browser / programmatic / command-line access, cross- domain Uses infrastructure as a service (IaaS) from resource providers and offers these IaaS resources for users 32

33. User Tools Java Client Library CloudBroker Platform Architecture 20.09.2012 / CloudBroker Platform Amazon Cloud Open- Stack Cloud … Cloud Chemistry Appli- cations Biology Appli- cations Health Appli- cations Web Browser UI … Appli- cations REST Web Service API CloudBroker Integration End Users, Software Vendors, Resource Providers CLI Engineerin g Appli- cations IBM Cloud Euca- lyptus Cloud 33

34. Integrated architecture

35. Integration features Support for commercial clouds with costs (prices configured in CloudBroker Platform): –Estimated job cost before submission –Actual job and workflow cost after execution

36. Accessible Cloud Resources Access provided by the CloudBroker Platform Commercial: –Amazon EC2 –IBM OpenSource/Free: –OpenStack –OpenNebula –Eucalyptus Currently, within SCI-BUS accessible: –MTA SZTAKI OpenNebula (400 cores) –BIFI OpenStack (50 cores)

37. Collaboration within and among communities based on gUSE 37 SHIWA Repository gUSE Portal Cloud 1 OpenNebula Cloud 2 Amazon Cloud n OpenStack gUSE Portal WF upload as SHIWA bundle gUSE WF Repo

38. Success story: SHIWA solution for the LINGA experiment Sub-Workflows Management 38 Multi- Workflow

39. Maturity of implementation Production services: –SHIWA Repository –SHIWA Bundle and SHIWA Desktop –CGI approach - Connected WF systems: ASKALON, Galaxy, MOTEUR, Pegasus, Taverna, Triana, WS- PGRADE –SHIWA portal based on gUSE –CloudBroker Platform - Connected clouds: Amazon, IBM, Eucalyptus, OpenNebula, OpenStack Prototype services –FGI approach - Connected WF systems: –ASKALON, MOTEUR, Triana, WS-PGRADE New EU project ER-Flow supports 6 user communities 39

40. Recent WS-PGRADE/gUSE releases History since v3.4.0 Nov 2011: v3.4.0 (DCI Bridge) Feb 2012: v3.4.1 (usage statistics portlet) March 2012: v3.4.2 (support for new EMI release) April 2012: v3.4.3 (support for Liferay 6.1) … Aug 2012: v3.5.0 (SaaS cloud access via CBP) Sep 2012: v3.5.1 (IaaS cloud access via CBP) Oct 2012: v3.5.2 (SHIWA workflow repository export/import) March 2013: v3.5.3 (REST support, EMI-UI v1/v2 support, …) April 2013: v3.5.4 (cloud cost estimation/reporting) April 2013: v.3.5.5 (robot certificates) May 2013: v.3.5.6 (Improved SHIWA workflow repository export/import)

41. gUSE sourceforge statistics 41

42. 42 Where to find further information? gUSE/WS-PGRADE: –http://www.guse.hu/ gUSE on sourceforge –http://sourceforge.net/projects/guse/ –http://sourceforge.net/projects/guse/forums/forum/ –http://sourceforge.net/projects/guse/develop SCI-BUS web page: –http://www.sci-bus.eu/ SHIWA web page: –http://www.shiwa-workflow.eu/http://www.shiwa-workflow.eu/ ER-Flow web page: –http://www.erflow.euhttp://www.erflow.eu

43. 43

44. Summary We have created a technology that enables to combine many different WFs, WF systems and DCIs in many different ways It is like a puzzle where you can put together the required pieces to create the final picture 44 gUSE WF system OpenNebula Cloud Kepler WF Galaxy WF