1 EGI Federated Cloud Architecture Matteo Turilli Senior Research Associate, OeRC, University of Oxford Chair – EGI Federated Clouds Task Force EGI Technical Forum 2012 – Prague, 17/21 Sep

2 Task Force Mandate and Organisation Mandate: 18 months, September 2011 – March Activities: divided into 3 blocks of 6 months each. Workgroups: –working units within the Task Force. –Investigate a specified set of capabilities for a federation of clouds. –Have leaders and collaborators. Workbenches: sections of the TF Wiki site collecting and documenting the activities of each workgroup. Meetings: weekly conference call. F2F every 3 months.

3 What users are asking for IaaS and control: running directly VMs for multiple purposes. VM images: pre-defined, tailored, easy to maintain VM images. Specialised hardware: 24+ core machines. Portals and tailored services: running portals/tailored services on an IaaS in order to get the usual benefits of Cloud Computing. Multiple hypervisors: they may have already VMs to deploy. Data-intensive distributed applications: i.e. Hadoop framework, large storage provision. Integration: CC is not all-or-nothing.

4 Federation Test bed – Sep 2012 At the moment, the test bed is composed of 4 services, 2 management interfaces, and 7 cloud infrastructures operated by 6 Resource Providers. 3 more providers are in the process of being federated.

5 Management Interfaces Virtual Machines (OCCI) Used to instantiate, destroy, list and inspect Virtual Machines across the federated cloud infrastructures. In principle, OCCI offers an unified and standardised management interface independently from the cloud technology chosen by the Resource Providers. The test bed successfully uses the OCCI implementations available for OpenNebula (rOCCI) and OpenStack. Data Objects (CDMI) Used to upload, download, delete and list file objects on multiple repositories hosted by the Resource Providers, possibly independently from the provision of other virtualised resources. The test bed uses the CDMI server and client developed by the Venus-c project.

6 Services Information system. Each cloud infrastructure exposes a LDAP server publishing information by means of a customised GLUE2 schema. Each LDAP server is polled by a top-BDII server. ldap://test03.egi.cesga.es:2170

7 Services Monitoring. A standard Nagios installation is used to monitor the availability of the management interfaces exposed by each cloud infrastructure. Probes to test the state of the federated services are under development.

8 Services Accounting. Each cloud infrastructure generates usage records based on an extended version of the EGI UR format recommendation. Records are uploaded to a central server by means of a client customised for each type of infrastructure.

9 Services Marketplace. A repository were Resource Providers and EGI can publish metadata about images from which virtual machines can be instantiated. When needed, a single image can be signed and then endorsed by multiple providers.

10 Conclusions Integration Third cycle of the TF, 6 months into March Requirements for the integration of the test bed into the EGI infrastructure. OCCI client and first broker deployment. Focus on use cases. Opening of the test bed to early adopters. Task Force resources Mailing List: Wiki site: GitHub: Indico site:

11 Thank you. Matteo Turilli Chair EGI FedCloud Task Force Senior Research Associate, OeRC, University of Oxford

12 Resource Providers – User View Resource Provider GWDG (ON/OS) Resource Provider GWDG (ON/OS) OCCI 1.1 CDMI 1.0 Information GLUE 2.0 LDAP Information GLUE 2.0 LDAP Monitoring Nagios Monitoring Nagios Accounting OGF UR UR+ & StAR Accounting OGF UR UR+ & StAR VM metadata Marketplace VM metadata Marketplace Resource Provider CESNET (ON) Resource Provider CESNET (ON) Resource Provider CYFRONET (ON) Resource Provider CYFRONET (ON) Resource Provider KTH (ON) Resource Provider KTH (ON) Resource Provider CESGA (ON) Resource Provider CESGA (ON) Resource Provider Venus-C Resource Provider Venus-C OCCI 1.1 CDMI 1.0 OCCI 1.1 CDMI 1.0 OCCI 1.1 CDMI 1.0 Resource Provider FZJ (OS) Resource Provider FZJ (OS) OCCI 1.1 User x.509 SSH key Lgn/Pswrd User x.509 SSH key Lgn/Pswrd ON = OpenNebula. OS = OpenStack.

13 Information GLUE 2.0 LDAP Information GLUE 2.0 LDAP Monitoring Nagios Monitoring Nagios Accounting OGF UR UR+ & StAR Accounting OGF UR UR+ & StAR Message Bus VM metadata Marketplace VM metadata Marketplace Resource Provider Venus-C Resource Provider Venus-C CDMI 1.0 Resource Providers – Federation View Resource Provider GWDG (ON/OS) Resource Provider GWDG (ON/OS) OCCI 1.1 CDMI 1.0 MP/UR Clients Resource Provider CESNET (ON) Resource Provider CESNET (ON) OCCI 1.1 CDMI 1.0 LDAP MP/UR Clients Resource Provider CYFRONET (ON) Resource Provider CYFRONET (ON) OCCI 1.1 LDAP MP/UR Clients Resource Provider KTH (ON) Resource Provider KTH (ON) OCCI 1.1 CDMI 1.0 LDAP MP/UR Clients Resource Provider CESGA (ON) Resource Provider CESGA (ON) OCCI 1.1 LDAP MP/UR Clients Resource Provider FZJ (OS) Resource Provider FZJ (OS) OCCI 1.1 LDAP MP/UR Clients LDAP ON = OpenNebula. OS = OpenStack. MP = Marketplace. UR = Usage Records.