1. Ian Bird CMS Computing & Software CERN, 15 th October 2015 15 Oct 2015 Ian Bird; CMS Offline & Computing1

2. Cloud experiences  Many experiences:  Through all LHC experiments  With many labs and WLCG sites  Majority of usage has been with simulation use cases  Not much experience with data intensive use  We know we can successfully use cloud resources, and make them useful  Even have had some reasonably large scale uses  However, largely supported by grants, free access, and etc.  Amazon grants, Google offers, Helix Nebula project, including a small scale procurement, etc. 15 Oct 2015 Ian Bird; CMS Offline & Computing2

3. Why are clouds interesting?  Must define what we mean  Private clouds Technology – better/different implementation of resource provisioning Mechanism for connecting to external resources  Commercial clouds Why are they relevant? Cost (actual cost, elasticity/flexibility, operations cost, …)  Bottom line is only how we can maximise the amount of computing we can buy 15 Oct 2015 Ian Bird; CMS Offline & Computing3

4. Open issues  What is the cost?  How can we manage data in commercial clouds?  How do we manage network policies?  What would be a model that easily allows us to use commercial clouds?  What are the procurement issues?  Where can we procure?  Is large-scale joint procurement possible?  How to procure spot pricing, “backfill”, paying for what is actually used, etc. And how this is verified 15 Oct 2015 Ian Bird; CMS Offline & Computing4

5. Cloud use  CERN-specific  What is long term evolution of the CERN DC? Wigner-style facility, essentially classic DC extension of CERN Hosted dedicated facility? Simply buy compute and storage services – paid by use?  How can commercial cloud services supplement or (partially) replace HEP-owned and managed facilities  (When) will cost drive this?  What is the model?  Cloud treated as a separate site like a Tier 1, or 2?  Cloud treated as an extension of a site – as part of the pledge  Jointly procured large resource pool?  NB. Funding and procurement mechanisms will drive this 15 Oct 2015 Ian Bird; CMS Offline & Computing5

6. Science Cloud – past, present  Helix Nebula EC project in Europe (together with other sciences)  CERN is doing 2 procurements to understand market prices and usage models:  200 KCHF for compute (essentially simulation)  400 KCHF for data intensive use cases (needs market survey)  To be procured as extensions of the CERN cloud  Eventual use cases – investigate alternatives to “Wigner-style” extensions for next round; dynamic expansion of CERN resources  This capacity is in addition to pledges  PCP project has been approved and will start in January 2016  Derogation of CERN procurement rules to allow compliance with EC procurement (EC member states) agreed by CERN Council last week  Will allow a joint procurement across Tier 1s – understand if we can obtain economies of scale   see next few slides 15 Oct 2015 Ian Bird; CMS Offline & Computing6

7. Science cloud …  And parallel discussions with other commercial providers  AWS, Microsoft Azure  Openlab project with Rackspace on federating clouds and using federated ID’s (in openstack) 15 Oct 2015 Ian Bird; CMS Offline & Computing7

8. The Helix Nebula Initiative Strategic Plan  Establish multi- tenant, multi- provider cloud infrastructure  Identify and adopt policies for trust, security and privacy  Create governance structure  Define funding schemes To support the computing capacity needs for the LHC experiments Setting up a new service to simplify analysis of large genomes, for a deeper insight into evolution and biodiversity To create an Earth Observation platform, focusing on earthquake and volcano research To improve the speed and quality of research for finding surrogate biomarkers based on brain images Adopters Suppliers Additional Users: 8Ian Bird; CMS Offline & Computing 15 Oct 2015

9. Example HNX procurement CERN price enquiry for a small fraction of the resources necessary to run the ATLAS experiment simulation software  Several valid responses were received and Atos (data centre in Tenerife, Spain) was awarded the contract accessed via the SlipStream broker and provisioning engine (from SixSq SME) and delivered over the GÉANT network  1.2 million CPU hours over 6 weeks producing 11.5 million GEANT4 tt ̅ events with a high level of efficiency for CPU intensive workload that confirmed  the use of commercial IaaS is technically feasible for simulation workloads  variations in RESTful APIs can be overcome with reasonable technical effort  independent monitoring of the quality of service and resources consumed is essential Ian Bird; CMS Offline & Computing9 15 Oct 2015

10.  Hybrid – link public research organisations, e-Infrastructures & commercial cloud services  Use GEANT network to link Research Infrastructures, repositories (EUDAT, OpenAIRE), EGI, PRACE etc. to commodity commercial cloud services (multiple providers)  A cornerstone of the Open Science Commons*  Trust - Researchers keep control of the cloud and their data  Guarantee a copy of all the data is kept on public resources  Ensure long-term preservation of the data  Insulate users from changes of service supplier and technology  Economy - Must be cheaper than the ‘build our own’ approach  Avoid separate ‘silos’ for each Research Infrastructure/Community  Profit from the economies of scale in commercial data centres * http://go.egi.eu/osc http://go.egi.eu/osc Towards the European Open Science Cloud http://dx.doi.org/10.5281/zenodo.16140 15 Oct 2015 Ian Bird; CMS Offline & Computing10

11. 11 Why PCP? Commercial IaaS exists but not certified, integrated with public e-infrastructures, offering std interfaces with suitable SLA and contractual terms & conditions. PPI Potential follow-on project if this PCP project is successful 15 Oct 2015 Ian Bird; CMS Offline & Computing

12. HNSciCloud H2020 PCP Project The group of buyers have committed ~1.6M€ of funds (generating ~6M€ total funds) Manpower Applications & Data In-house IT resources To procure innovative IaaS cloud services integrated into a hybrid cloud model Commercial cloud services European e-Infrastructures GEANT, eduGain, EGI In-house IT resources Procured services will be made available to end-users from many research communities Ian Bird; CMS Offline & Computing12 15 Oct 2015

13. Preparation Analysis of requirements, current market offers and relevant standards Build stakeholder group Develop tender material ImplementationSharing Best practises Recommendations Training Oct’ 16 PCP project phases Launch of tender Pilots tested & assessed 9 months 15 months6 months Jan’ 16 Jan’ 18 Jun’ 18 All procuring organisations must sign letter to re-confirm commitment of funds DesignPrototypePilot

14. User Groups The cloud resources procured will be made available to user groups during the pilot phase  LHC experiments via WLCG Procured resources will count against the buyers’ pledges (during pilot phase) CERN will provide the interface via Tier-0 (OpenStack in tender spec.)  ELIXIR Managed by EMBL-EBI via the ELIXIR Compute platform  Other research communities via EGI Fed Cloud Request OCCI interface in tender spec.  Local users at each buyers site Each buyer is responsible for integration Request web GUI interface in tender spec. Sites can also use OpenStack or OCCI interface as well Each buyer decides what fraction of their procured resources is made available to each user group but cannot assign only to their local users Collectively the users will form a user group with a role in the project to define requirements and provide feedback on pilot deployments BBMRI DARIAH EISCAT_3D EPOS INSTRUCT LifeWatch Long-tail 15 Oct 2015 Ian Bird; CMS Offline & Computing14

15. 15 Oct 2015 Ian Bird; CMS Offline & Computing15

16. Open Science Cloud and the EC Questions posed by RJ Smits during visit to CERN with Commissioner Moedas in January 2015: “The Commission will launch a European Cloud initiative including cloud services certification, contracts, switching of cloud services providers and a research open science cloud.” The draft H2020 Work Programme 2016 – 2017 (annex 4) includes funding call INFRADEV-04- 2016 European Open Science Cloud for Research (10M€) Commissioner Moedas announced in June 2015: CERN produced a paper Towards the European Open Science Cloud that answered these questions http://dx.doi.org/10.5281/zenodo.16140 http://dx.doi.org/10.5281/zenodo.16140 15 Oct 2015 Ian Bird; CMS Offline & Computing16

17. 15 Oct 2015 Ian Bird; CMS Offline & Computing17

18. Summary  A lot of ongoing work exploring how to use commercial (cloud) resources for various workloads  And how to procure and provision such resources  This is needed in order to understand and plan for how to most cost-effectively procure compute and storage resources in the future  EC has a vision of a European Science Cloud  This might develop out of some of the work we are involved in  We have to understand whether that vision matches how we see the LHC computing models evolving 15 Oct 2015 Ian Bird; CMS Offline & Computing18