Co-ordination & Harmonisation of Advanced e-Infrastructures for Research and Education Data Sharing Co-funded by the European Commission under its 7 th Framework Programme The CHAIN-REDS Project: Cloud Activities Giuseppe LA ROCCA CHAIN-REDS School on Cloud Computing, April 2015

Outline 2  The CHAIN-REDS project: general information and vision  Some activities in the CHAIN-REDS project  Catania Science Gateway Framework (CSFG)  The new JSAGA Adaptor for OCCI  The CHAIN-REDS Cloud Test-bed  Orchestration of different clouds with Clever  Summary and conclusions 13-18/04/2015CHAIN-REDS School on Cloud Computing, Catania

3 General Information 13-18/04/2015CHAIN-REDS School on Cloud Computing, Catania  Co-ordination & Harmonisation of Advanced eInfrastructures for Research and Education Data Sharing  Research Infrastructures – Support Action  Grant Agreement n  Total Costs of € 2.3 M  Max. EC contribution: € 1.52 M  Start date: 1 December 2012  Duration: 30 Months

4  INFN (IT) – Coordinator  CIEMAT (ES) – WP4 Leader  GRNET (GR) – WP3 Leader  CESNET (CZ) – WP5 Leader  UBUNTUNET (MW) – Africa  CLARA (UR) – Latin America  IHEP (CN) – China  ASREN (DE) – Arab States  SIGMA ORIONIS (FR) – WP2 Leader  C-DAC (IN) – India The CHAIN-REDS Project ( /04/2015CHAIN-REDS School on Cloud Computing, Catania

5  Promote and support technological and scientific collaboration across different e-Infrastructures established and operated in various continents to facilitate their uptake and use by established and emerging Virtual Research Communities (VRCs) but also by single researchers 13-18/04/2015CHAIN-REDS School on Cloud Computing, Catania The Project Strategic Vision  Not only disseminate, exchange and reinforce the best practices currently adopted in Europe and other continents, but also promote the progress of interoperability among different regional e-Infrastructures  Study and define a path towards a global e-Infrastructure ecosystem that will allow VRCs, research groups and even single researchers to access and efficiently use worldwide distributed resources  Promote and support technological and scientific collaboration across different e-Infrastructures established and operated in various continents to facilitate their uptake and use by established and emerging Virtual Research Communities (VRCs) but also by single researchers

CHAIN-REDS School on Cloud Computing, Catania 6 Activities in the CHAIN-REDS project 13-18/04/2015

7 CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 Virtual Research Community (VRC) Grid/Cloud Infrastructure e-Infrastructure Network Infrastructure The e-Infrastructure vision e-Infrastructure «an environment where research resources can be shared and accessed to promote better and more efficient research» The Research Grid/Cloud infrastructure provides a distributed environment for sharing computing and storage capacities through the appropriate SW «middleware» The Research Network infrastructure provides fast interconnection and advanced services for Research and Education institutes of different countries

8 CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 Some barriers limit the e-Infrastructure adoption

9 The “non-global” middleware CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 Genesis II Interoperability and easiness of access are issues

10 A very “cloudy” sky … CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015

11 Science Gateway Science Gateway Administrator Scientist Cloud tenant Users belonging to Identity Federations Users belonging to Identity Federations Grid/Cloud Engine (based on SAGA) Orchestrator (based on OCCI) Cloud #2 Cloud #n Cloud #1 Single domain HPC Clusters MyCloud / The Catania Science Gateway Framework’s high-level architecture 13-18/04/2015CHAIN-REDS School on Cloud Computing, Catania

12 The Catania Science Gateway Framework’s components 13-18/04/2015CHAIN-REDS School on Cloud Computing, Catania  Authentication and Authorization  SAML, LDAP  Application e-Infrastructure independent  SAGA, OCCI  Secure Grid & Cloud transactions  Robot Certificates, PKCS#11  Standard Java Technology  JSR 168/286  Portal Framework  Liferay portlet contained

13 The AuthN & AuthZ Schema CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 Federation 1. Try to login 3. Identity attributes 2. Forwarded to the IdP 4. Check authorisations Sync user roles Retrieve e-Infrastructure credentials Science Gateway VAMP Workshop 2013 – Helsinki, 30/9-1/10/2013 Local Cluster Grid More details on the next Marco’s presentation

14 The high-level architecture of the eToken servlet CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015  The eToken servlet is a standard based solution developed by INFN for central management of robot certificates and provisioning of proxies to get seamless access to computing infrastructures supporting the X-509 standard for AuthN

15 How does the eToken work ? CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015

16 The eToken web interface (protected access) CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 Use the VOMS-ADMIN APIs to get the list of FQANs

17 The eToken web interface (protected access) CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 Enable / Disable long-term proxy Enable RFC / Full-legacy proxyAdding additional CN (for accounting)

18 Some RESTful APIs to request proxies / list robots CHAIN-REDS School on Cloud Computing, Catania13-18/04/ d11f?voms=fedcloud.egi.eu:/fedcloud.egi.eu&proxy-renewal=true&disable-voms-proxy=false&rfc- proxy=true&cn-label=LAROCCA Create RFC 3820 complaint proxies Create full-legacy globus proxies d11f?voms=fedcloud.egi.eu:/fedcloud.egi.eu&proxy-renewal=true&disable-voms-proxy=false&rfc- proxy=false&cn-label=Empty eTokenServer host & port MD5Sum Options FQANs Create plain proxies (without VOMS ACs) Get the list of available robot certificates Create full-legacy globus proxies (with more FQANs) d11f?voms=fedcloud.egi.eu:/fedcloud.egi.eu&proxy-renewal=true&disable-voms-proxy=true&rfc- proxy=false&cn-label=Empty 010?voms=vo.eu-decide.eu:/vo.eu-decide.eu/Role=Neurologist+vo.eu-decide.eu:/vo.eu- decide.eu&proxy-renewal=true&disable-voms-proxy=false&rfc-proxy=false&cn-label=Empty

19 New eToken installations (being) supported by CHAIN-REDS CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 in preparation

20 SAGA in a nutshell CHAIN-REDS School on Cloud Computing, Catania13-18/04/  Simple APIs for Grid Applications (SAGA)SAGA  The SAGA OGF Standard Specification [1][1]  SAGA does NOT want to target middleware developers!  SAGA allows creating an unique interface to different middleware stacks such as “Cloud”, Grid and local batch schedulers.  It is independent of the underlying infrastructure

21 SAGA in a nutshell CHAIN-REDS School on Cloud Computing, Catania13-18/04/  SAGA is composed by:  SAGA Adaptors: libraries providing access to the underlying infrastructures  SAGA Core Libraries: containing the SAGA base system, the runtime and the API packages (file management, job management, etc.)

22 SAGA in a nutshell CHAIN-REDS School on Cloud Computing, Catania13-18/04/  Several SAGA implementations are available  A C++ and a Java implementation developed at the Louisiana State University / CCT and Vrije Universiteit Amsterdam (  A Java implementation developed at CCIN2P3 (  A Python implementation based on those above

23 The Catania Grid & Cloud Engine CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 Users Tracking DB Science GW Interface SAGA/JSAGA API Job Engine Data Engine Users Track & Monit. Grid/Cloud/Local MWs User Tracking DB eToken Server 23

24 The JSAGA Adaptor for OCCI CHAIN-REDS School on Cloud Computing, Catania13-18/04/ Science Gateway Catania Grid/Cloud Engine Security context Job management Stage-in/out 2) kill VM SG stage-in & run app VM1) stage-out SG VM SG start VM robot certificate

25 Running scientific applications on different cloud providers CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015

26 CHAIN-REDS School on Cloud Computing, Catania13-18/04/ Enable advanced settings Specify the advanced settings for the application Start VM Select the VM profile Enable notification Select the VM template Job description Running scientific applications on different cloud providers

27 The “interoperability” demo during the EGI-TF 2013 in Madrid CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015

CHAIN-REDS School on Cloud Computing, Catania 28  The Weather Research and Forecasting (WRF) model is a next- generation meso-scale numerical weather prediction system designed to serve both atmospheric research and operational forecasting needs. WRF in a nutshell

13-18/04/2015CHAIN-REDS School on Cloud Computing, Catania 29 // Define the list of available e-Infrastructure(s) for the application InfrastructureInfo infrastructures[] = new InfrastructureInfo[NMAX]; infrastructures[0] = new InfrastructureInfo("SSH", "ssh", LOGIN, PASSWD, wmsList); infrastructures[1] = new InfrastructureInfo("CHAIN-REDS","rocci", "", wmsList, ETOKENSERVER, PORT, ROBOTID, VONAME, ROLE, true); infrastructures[2] = new InfrastructureInfo("EUMED", wrf_TOPBDII, eumed_wmsList, eumed_ETOKENSERVER, eumed_PORT, eumed_ROBOTID, eumed_VONAME, eumed_ROLE, "VO-" + eumed_VONAME + "-" + wrf_SOFTWARE); // Define the parameters for the application MultiInfrastructureJobSubmission CloudMultiJobSubmission = new MultiInfrastructureJobSubmission(); CloudMultiJobSubmission.setExecutable("/bin/bash"); CloudMultiJobSubmission.setArguments(Arguments); [..] Looking under the hood

13-18/04/2015CHAIN-REDS School on Cloud Computing, Catania 30 // Add the list of e-Infrastructures CloudMultiJobSubmission.addInfrastructure(infrastructures[0]); CloudMultiJobSubmission.addInfrastructure(infrastructures[1]); CloudMultiJobSubmission.addInfrastructure(infrastructures[2]); // Get the infra InfrastructureInfo infrastructure = CloudMultiJobSubmission.getInfrastructure(); // Submit the application CloudMultiJobSubmission. submitJobAsync ( infrastructure, username, addr.getHostAddress()+":8162", Integer.valueOf(wrf_APPID), _Parameters[4]); Looking under the hood

13-18/04/2015CHAIN-REDS School on Cloud Computing, Catania 31 MyJobs

CHAIN-REDS School on Cloud Computing, Catania13-18/04/ Standard-based SG developed in the context of projects collaborating with CHAIN-REDS 32

CHAIN-REDS School on Cloud Computing, Catania 33 Orchestrate different clouds in CHAIN-REDS 13-18/04/2015

34 CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 CLEVER in a nutshell (1/2)  Cloud-Enabled Virtual EnviRonment (CLEVER) is a Virtual Infrastructure ManagerCLEVER  It is able to federate different clouds using OCCI standard  The CLEVER project is maintained by the University of Messina - Italy

35 CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 CLEVER in a nutshell (2/2)  Host Manager (HM) performs the operations needed to monitor the physical resources and the instantiated VEs; moreover, it runs the VEs on the physical hosts (downloading the VE image) and performs the migration of VEs  Cluster Manager (CM) acts as an interface between the clients (software entities which can exploit the cloud) and the HM agents  Distributed Database is merely the database containing the overall set of information related to the middleware  XMPP Communication System is the channel used to enable the interaction among the middleware components.

36 CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 The CHAIN-REDS Cloud Testbed (current status)

37 CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 The CHAIN-REDS Cloud Testbed (evolution)

38 CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 Current functionalities: Federated authentication Fine-grained authorisation Single/multi-deployment of VMs on a cloud and across clouds, including the EGI FedCloud Single/multi-move of VMs across clouds Single/multi-deletion of VMs on a cloud and across clouds SSH connection to VMs Direct web access to VMs hosting web services VMs are Moveable across Clouds Available VMs Multi/Single instance Available VMs Multi/Single instance Orchestration of different clouds with Clever (1/2)

39 Orchestration of different clouds with Clever (2/2) CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015 Turn On/Off the VM

Summary and conclusions 40  CHAIN-REDS can be considered a seminal project  The CHAIN-REDS project has heavily contributed to e- Infrastructure harmonisation  CHAIN-REDS has successfully demonstrated interoperability at user application level thanks to standard adoption (SAGA, SAML, OCCI, JSR286, etc.) CHAIN-REDS School on Cloud Computing, Catania13-18/04/2015

Co-ordination & Harmonisation of Advanced e-Infrastructures for Research and Education Data Sharing Co-funded by the European Commission under its 7 th Framework Programme Thank you !