Pre-Commercial Procurement proposal - HNSciCloud 13 May 2015 Bob Jones, CERN

Pre-Commercial Procurement proposal A Pre-Commercial Procurement proposal was submitted to the H2020 ICT 8a call in April 2015 for innovative IaaS cloud services The proposal is currently under evaluation by the EC Official feedback expected in September These slides give an overview of the proposal

The Helix Nebula Initiative Strategic Plan Establish multi-tenant, multi-provider cloud infrastructure Identify and adopt policies for trust, security and privacy Create governance structure Define funding schemes To support the computing capacity needs for the LHC experiments Setting up a new service to simplify analysis of large genomes, for a deeper insight into evolution and biodiversity To create an Earth Observation platform, focusing on earthquake and volcano research To improve the speed and quality of research for finding surrogate biomarkers based on brain images Additional Users: Adopters Suppliers Helix Nebula was formed in 2011 by leading research organisations in collaboration with Europe’s commercial cloud suppliers to develop such a platform. It has grown over time with more research organisations joining (demand-side) and more service suppliers (supply-side) both commercial and publicly funded. Note the high proportion of SMEs that are members (30% of members in 2014) Bob Jones, CERN

H2020 Coordination and Support Action Total budget: 500 K€ Duration: 18 months Start date: 1 Oct. 2014 Partners: Public research organisations have well established procurement processes that are tailored to buying and operating computing H/W and S/W but not the ‘rental’ of cloud services.

Example HNX procurement CERN price enquiry for a small fraction of the resources necessary to run the ATLAS experiment simulation software Several valid responses were received and Atos (data centre in Tenerife, Spain) was awarded the contract accessed via the SlipStream broker and provisioning engine (from SixSq SME) and delivered over the GÉANT network 1.2 million CPU hours over 6 weeks producing 11.5 million GEANT4 tt̅ events with a high level of efficiency for CPU intensive workload that confirmed the use of commercial IaaS is technically feasible for simulation workloads variations in RESTful APIs can be overcome with reasonable technical effort independent monitoring of the quality of service and resources consumed is essential Limited scale procurements are being made via this platform demonstrating the feasibility of the approach. The SlipStream broker is an open source solution developed with the aid of the EC funding via the StratusLab FP7 project. Bob Jones, CERN

Towards the European Open Science Cloud Hybrid – link public research organisations, e-Infrastructures & commercial cloud services Use GEANT network to link Research Infrastructures, repositories (EUDAT, OpenAIRE), EGI, PRACE etc. to commodity commercial cloud services (multiple providers) A cornerstone of the Open Science Commons* Trust - Researchers keep control of the cloud and their data Guarantee a copy of all the data is kept on public resources Ensure long-term preservation of the data Insulate users from changes of service supplier and technology Economy - Must be cheaper than the ‘build our own’ approach Avoid separate ‘silos’ for each Research Infrastructure/Community Profit from the economies of scale in commercial data centres * http://go.egi.eu/osc So clearly there is a need to have a platform on which this data can be managed. Sharing of data is necessary, increasing and must be encouraged by the platform but it has to respect the cultures and investments made by the research communities. Researchers have fears, some of them irrational, about using commercial cloud services. But telling someone who is afraid of flying that their fear is irrational does not mean they will get on the plane. We have to make them feel secure. Here the hybrid cloud model can help by ensuring that a copy of their precious data is kept safe on publicly managed resources. Economy – the price comparison must take into account all costs, not just the purchase price of h/w. http://dx.doi.org/10.5281/zenodo.16140

PCP PPI Why PCP? Commercial IaaS exists but not certified, integrated with public e-infrastructures, offering std interfaces with suitable SLA and contractual terms & conditions. PPI Potential follow-on project if this PCP project is successful

Proposed Joint Pre-Commercial Procurement The group of buyers have committed >1.6M€ of funds Manpower Applications & Data In-house IT resources To procure innovative cloud services Integrated into a hybrid cloud model: Commercial cloud services e-Infrastructures GEANT network eduGAIN Fed. Id mgmt. EGI Fed Cloud Potentially host data services from EUDAT, INDIGO-Datacloud, etc. Made available to end-users including BBMI, DARIAH, ELIXIR, EISCAT_3D, EPOS, INSTRUCT, LifeWatch, LHC, etc. Research organisations from 7 countries have proposed to work together to develop a cross-border joint procurement of innovative cloud services. This is an important change: never before have research organisations across Europe pooled their funds and resources to procure cloud services to support their scientific programmes. eduGAIN federated identity mgmt. should include commercial identity providers taking into account multiple levels of assurance. Bob Jones, CERN

Pilots tested & assessed PCP project phases Preparation Analysis of requirements, current market offers and relevant standards Build stakeholder group Develop tender material Implementation Sharing Best practises Recommendations Training 6 months 18 months Design Prototype Pilot Launch of tender Pilots tested & assessed Jan’16 Oct’16 Jan’18 Jun’18 All procuring organisations must sign letter to re-confirm commitment of funds

PCP project implementation phase Design Prototype Pilot Bidders have 3 months to respond to bid Bidders provide written design report including architecture and technical design of components together with unit costs Contractors build prototypes including all components and make it accessible to experts from buyers group that perform functionality and security tests select ≥3 designs taking into account tender criteria Evaluation committee includes representatives from all buyers Multiple iterations with feedback to suppliers in prototype and pilot phases Contractors deploy expanded prototypes that are made available firstly to experts who perform scalability tests and then to end-users select ≥2 prototypes taking into account feedback from experts

PCP project implementation phase 3 phases: Solution design ≥3 designs 3 months 15% tender budget Prototype development ≥3 prototypes 6 months 25% tender budget Deployment of limited scale prototypes ≥2 pilot deployments 60% tender budget Total procurement funds available: ~5M€ Each phase is competitive. Bids evaluated/tested against published criteria

Project organisation CERN CERN CNRS DESY INFN EGI.eu TRUST-IT The EC funding covers the procurement costs, project and procurement process administration as well as work package leadership All the buyers provide technical effort to evaluate designs/prototypes/pilots

User Groups The cloud resources procured will be made available to user groups during the pilot phase LHC experiments via WLCG Procured resources will count against the buyers’ pledges (during pilot phase) CERN will provide the interface via Tier-0 (OpenStack in tender spec.) ELIXIR Managed by EMBL-EBI via the ELIXIR Compute platform Other research communities via EGI Fed Cloud Request OCCI interface in tender spec. Local users at each buyers site Each buyer is responsible for integration Request web GUI interface in tender spec. Sites can also use OpenStack or OCCI interface as well Each buyer decides what fraction of their procured resources is made available to each user group but cannot assign only to their local users Collectively the users will form a user group with a role in the project to define requirements and provide feedback on pilot deployments BBMRI DARIAH EISCAT_3D EPOS INSTRUCT LifeWatch Long-tail

Next Steps CERN will perform another limited scale procurement in preparation for the PCP More details at the 6th Helix Nebula General Assembly @ CERN 24-25 June restricted to NDA signatories https://indico.cern.ch/event/369721/ 26 June open day http://indico.cern.ch/e/OpenDay Assuming the PCP proposal is funded September-December 2015: Approval of changes to CERN procurement rules to support EC co-funded PCP Prepare EC contract and consortium agreement Make more detailed planning of tasks Start work on tender material Investigate to support data-intensive workloads Project start January 2016