1. www.egi.eu EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 CAP4SME and EGI

2. www.egi.eu Outline Introduction to EGI EGI Federated Cloud RTD collaboration CAP4SME and EGI Federated Cloud 2 17 December 2013

3. www.egi.eu Outline Introduction to EGI EGI Federated Cloud RTD collaboration CAP4SME and EGI Federated Cloud 3 17 December 2013

4. www.egi.eu EGI.eu Coordination for European Grid resources –Established February 8th 2010 –Central policy & services needed to run a grid –Sustainable small coordinating organisation Based in Amsterdam –Coordinating core (~25 people) in Amsterdam –Technical services from partners (~20 people) CAP4SME and EGI Federated Cloud 4 EGI and EGI.eu: Supported by the EGI-InSPIRE project 17 December 2013

5. www.egi.eu European Grid Infrastructure 2013 Integrated EGI-InSPIRE Partners and EGI Council Participants Internal/External RPs being integrated External RP Peer RP CAP4SME and EGI Federated Cloud 5 17 December 2013

6. www.egi.eu European Grid Infrastructure (EGI) 6 337 Resource Centres in 34 National Grid Initiatives/EIROs 430,000 logical CPU cores 190 PB disk, 180 PB tape 1.2 M job/day, EGI-InSPIRE PY3: +44.7% increase of CPU wall clock time used (HS-06 h) 53 High activity projects supported by EGI EGI-InSPIRE EC Project (2010-2014) supporting 50 project partners (EGI.eu, 38 NGIs, 2 EIROs) Plus 9 APAC partners EGI and NGI operations Outreach and policy development Software validation and verification Federated cloud CAP4SME and EGI Federated Cloud 17 December 2013

7. www.egi.eu 10 years in production 7 LS: 116.7 Million hours AA: 110.3 Million hours HEP: 4.5 Billion hours EGI-InSPIRE CAP4SME and EGI Federated Cloud 17 December 2013

8. www.egi.eu Outline Introduction to EGI EGI Federated Cloud RTD collaboration CAP4SME and EGI Federated Cloud 8 17 December 2013

9. www.egi.eu EGI Federated Cloud: From Pilot to Production Objectives: – Identify and investigate the capabilities needed to federate private clouds – Identify the technical solutions, deploy proof of concepts in a pre production testbed, test the solutions with real use cases Capabilities 1.Manage VM instances 2.Data access/transfer interface 3.Cloud service information federation 4.Resource consumption management 5.Cloud service availability 6.Notification & Automation Setup 09.2011 Consolidation 09.2012 Integration 03.2013 Preproduction 09.2013 Production 05.2014 CAP4SME and EGI Federated Cloud 9 7.Federated AAI 8.VM Image Management 9.Brokering 10.Contextualisation 11.SLA & Business models 17 December 2013

10. www.egi.eu Participation – Nov 2013 Members ~102 individuals ~45 institutions >15 countries Members ~102 individuals ~45 institutions >15 countries Stakeholders 23 Resource Providers 10 production 10 Technology Providers 10 User Communities 4 Liaisons Technologies OpenNebula, StratusLab, OpenStack, Synnefo, WNoDeS, Cloudstack,* PERUN, SlipStream. Technologies OpenNebula, StratusLab, OpenStack, Synnefo, WNoDeS, Cloudstack,* PERUN, SlipStream. BSC CNRS LMU OeRC Masaryk INFN-BARI IFAE Cyfronet 100%IT CESNET RADICAL SRCE DANTE FZJ GRNET GWDG STFC SARA KTH INFN- CNAF FCTSG EGI.eu ISRGrid CESGA CETA IFCA IGI IPHC IN2P3 SZTAKI IISAS SixSq CSC 10 BIFI 17 December 2013 CAP4SME and EGI Federated Cloud

11. www.egi.eu EGI Vision To support the digital European Research Area through a pan-European research infrastructure based on an open federation of reliable services that provide uniform access to national computing, storage and data resources. CAP4SME and EGI Federated Cloud 11 EGI federated Cloud vision for 2020: 10M cores Cloud compute 1 EB Cloud storage 17 December 2013

12. www.egi.eu EGI Platform architecture Core Infrastructure CAP4SME and EGI Federated Cloud 12 17 December 2013

13. www.egi.eu EGI Core Infrastructure Services that federate and integrate the functional services deployed in the production infrastructure CAP4SME and EGI Federated Cloud 13 AAI Service Registry Service Monitoring Accounting Information Service Messaging Metrics visualisation 17 December 2013

14. www.egi.eu EGI Platform architecture Federated IaaS Cloud Core Infrastructure CAP4SME and EGI Federated Cloud 14 17 December 2013

15. www.egi.eu EGI’s Cloud Infrastructure EGI Federated Operations Federated AAI Service Discovery & Status MonitoringAccounting EGI Cloud Infrastructure Platform Cloud Management Stacks (OpenStack, OpenNebula, …) VM Mgmt Storage Mgmt EGI Collaboration Tools EGI Application DB Image Repository Enable an open ecosystem of services OCCI CDMI UR GLUE2 OVF X.509 CAP4SME and EGI Federated Cloud 15 17 December 2013

16. www.egi.eu EGI Platform architecture Federated IaaS Cloud Community PaaS VRE BrokerVRE Core Infrastructure Virtual Research Environment (VRE) VRE CAP4SME and EGI Federated Cloud 16 EGI members (define, deploy, operate) EGI members (define, deploy, operate) EGI (define, deploy, operate) EGI (define, deploy, operate) Customers (define, deploy, operate) Customers (define, deploy, operate) Customers (define, deploy, operate) Customers (define, deploy, operate) Customers (define, deploy, operate) Customers (define, deploy, operate) Customers (define, deploy, operate) Customers (define, deploy, operate) 17 December 2013

17. www.egi.eu Federated Cloud service tiers Federated IaaS Cloud CAP4SME and EGI Federated Cloud 17 Tier 1: Reliable Infrastructure Cloud Tier 4: Zero ICT Infrastructures Tier 3: Professional research infrastructure platforms Tier 2: General-purpose platform services VRE DB aaS Hadoop aaS VRE Secure storage Key Mgmt Encryptio n ACL mgmt Virtual eLaboratory 17 December 2013

18. www.egi.eu Outline Introduction to EGI EGI Federated Cloud RTD collaboration CAP4SME and EGI Federated Cloud 18 17 December 2013

19. www.egi.eu Membership EGI membership –National Grid Initiatives (academic), EIROs –Full member –Affiliate member (no voting rights) EGI Federated Cloud membership –Lead by EGI (through OeRC and EGI.eu) –Technology Providers –Resource & Service Providers –User Communities –Some funding through EGI-InSPIRE –Remainder self-funded activities –Academic, public sector & SME –Different resources through different business models CAP4SME and EGI Federated Cloud 19 17 December 2013

20. www.egi.eu Integrating Resource Providers 3 step process 1.Configure/extend CMF to support mandated Cloud interfaces 2.Integrate with EGI Core Platform (Accounting Monitoring Information publishing) 3.Operate to EGI’s OLA requirements Detailed documentation available CAP4SME and EGI Federated Cloud 20 17 December 2013

21. www.egi.eu Accessing EGI’s IaaS services (1) Current pre-production process: 1.Obtain e-Science certificate (X.509 based) http://www.egi.eu/how-to/get_a_certificate.html 2.Create or join Virtual Organisation –Lightweight group membership management –EGI runs a SaaS: PERUN (https://perun.metacentrum.cz/)https://perun.metacentrum.cz/ 3.Find EGI resource provider to support you –Decide on business model and supporting providers 4.Register VM images in EGI AppDB –http://appdb.egi.eu/http://appdb.egi.eu/ –Not yet supporting private images! –Currently a manual process CAP4SME and EGI Federated Cloud 21 17 December 2013

22. www.egi.eu Accessing EGI’s IaaS services (2) 5.Read the EGI Federated Clouds FAQ 6.Set up a Command Line environment https://wiki.egi.eu/wiki/Fedcloud-tf:CLI_Environment 5.Join the EGI Federated Clouds group –Create EGI SSO account http://www.egi.eu/ssohttp://www.egi.eu/sso –Send Email to Michel with your EGI SSO name CAP4SME and EGI Federated Cloud 22 17 December 2013

23. www.egi.eu Interoperability & Migration Open Standards enable interoperability –Between Cloud Providers –Reduced multi-provider integration effort Hypervisors: strategic choice Cloud object storage is format agnostic –CDMI metadata are name/value strings Cloud block storage usually raw or qcow2 CAP4SME and EGI Federated Cloud 23 17 December 2013

24. www.egi.eu Summary EGI: Community of resource providers with long-term tradition in providing federated ICT services for research EGI Federated Cloud: Paving the way for a federated cloud in Europe –Open standards, open technology –Open membership, open processes Driver for innovation in Europe CAP4SME and EGI Federated Cloud 24 17 December 2013

25. www.egi.eu BACKUP 17 December 2013 CAP4SME and EGI Federated Cloud 25

26. www.egi.eu Federated Cloud Users New services to meet user demand –Use case driven Proof of concepts being successfully demonstrated (http://go.egi.eu/PoC)http://go.egi.eu/PoC –Hosting of services for data dissemination (SaaS) – ENVRI, EISCAT_3D, BNCWeb/CLARIN –Digital Libraries and Digital Preservation services for memory institutions and human science - DCH-RP –Integrated Cloud Iaas-SaaS services to avoid large data transfers (ESA) –Virtual laboratories (PaaS and IaaS) – BioVel, LifeWatch –Hadoop clusters on demand (PaaS) – Peachnote, BioVel –Researcher training in safe, low cost environments on community tools – WeNMR –Repeatable and controllable software development and build environments – SCI-BUS –Science gateway hosting and access to resources – CSGF –… 26 CAP4SME and EGI Federated Cloud 17 December 2013

27. www.egi.eu PlatformDescriptionKey servicesUse cases Data dissemination/Open data Cloud storage provides an infrastructure to collect and disseminate scientific data. Data intake is curated, data access is inherently open (anonymous) or tracked (social identity?) SaaS catalogues tailored to user communities (ex. EO, Biology, etc…) Custom Data ACL Federated AAI for data access Monitoring of data usage ENVRI GeoCatalogue EISCAT 3D Peachnote Secure StorageStored data is protected even from RP access (e.g. through encryption). Safe for storing (personal) confidential data. Keys management Server-side encryption Secure data deletion Data sharing with ACL Arvados Location-based computing Input data is stored on the cloud, processing is on the same cloud or in federated data-centers to have high data access throughput with own applications; “ Integrated Cloud Iaas-SaaS services to avoid large data transfer IaaS to provide processing flexibility Input data stored openly to attract different communities ESA G-POD Arvados Virtual LaboratoriesTools to customize and manage virtual laboratories for different communities. Laboratories have shared tools to access data from different sources. User communities manages the instruments, EGI operates the underlying infrastructure and provides the generic tools to access storage (cloud storage, etc...) Common shared tools to access data from different resources Tools to ease laboratories setup Underlying IaaS and SaaS to support running of the laboratories and store user data storage (with easy scalability) Simple user interface to request laboratories access BioVeL virtual e-laboratory LifeWatch virtual laboratory Use cases 1 27 17 December 2013 CAP4SME and EGI Federated Cloud

28. www.egi.eu PlatformDescriptionKey servicesUse cases Data preservationData long-term preservation. To provide consolidation, persistency, integrity, redundancy, and usability over long periods of time. Data consolidation (ensure all the data is harmonized in terms of format, nomenclature, access, etc…) Data integrity and redundancy (ensure no loss of data) Data access preservation (ensuring software to read and analyze the data is maintained) EISCAT 3D DCH-RP HEP Platform-as-a- Service Pre configured processing facilities with integrated access to data, running on top of cloud IaaS and SaaS solutions. Possible PaaS services are: Grid Computing Hadoop Clusters on demand Generic High Avaliability service Scalability (adapt to the workload) High Availability (resources are always available) Resource sharing (different services share the same underlying physical resources) Peachnote BioVeL HA Use cases 2 28 17 December 2013 CAP4SME and EGI Federated Cloud

29. www.egi.eu Value proposition Open standards, open technologies –Use of Open Standards is key to the establishment of an effective, fair and transparent cloud market in Europe –Open Source components raise the barrier for hidden backdoors, thus lead to more trusted services Firmly rooted in Europe –Strong public sector involvement through NRENs, NGIs, EIROs are EGI’s members –European commercial Cloud resource providers, including SME A single cross-border market –Reaching out for research, government & business sectors –Level playing field for innovation and services on multi-service tiers 29 17 December 2013 CAP4SME and EGI Federated Cloud