1. Bringing Private Cloud Computing to HPC and Science EGI Technical Forum 2013 Madrid, Spain, September 17th, 2013 Ignacio M. Llorente Project Director © OpenNebula Project. Creative Commons Attribution-NonCommercial-ShareAlike License

2. 2/30 Bringing Private Cloud Computing to HPC and Science Contents Building Private Cloud Computing to HPC and Science This presentation is about: The Private HPC Cloud Use Case Main Challenges for Private HPC Cloud Private HPC Cloud Case Studies Private Cloud Trends in Industry About Grid and Clou d

3. 3/30 Bringing Private Cloud Computing to HPC and Science The Private HPC and Science Cloud Use Case The Pre-cloud Era LRMS (LSF, PBS, SGE…) Grid Middleware Access Provision

4. 4/30 Bringing Private Cloud Computing to HPC and Science The Private HPC and Science Cloud Use Case OpenNebula as an Infrastructure Tool – Enhanced Capabilities Virtual Worker Nodes LRMS (LSF, PBS, SGE…) Grid Middleware Access Provision Service Common interfaces Grid integration Custom environments Dynamic elasticity Consolidation of WNs Simplified management Physical – Virtual WNs Dynamic capacity partitioning Faster upgrades Service/Provisioning Decoupling

5. 5/30 Bringing Private Cloud Computing to HPC and Science The Private HPC and Science Cloud Use Case OpenNebula as an Provisioning Tool – Enhanced Capabilities Pilot Jobs, SSH… IaaS Interface Access Provision Service Simple Provisioning Interface Raw/Appliance VMs Dynamic scalable computing Custom access to capacity Not only batch workloads Not only scientific workloads Improve utilization Reduced service management Cost efficiency

6. 6/30 Bringing Private Cloud Computing to HPC and Science Main Challenges for Private HPC Cloud Main Demands from Engineering, Research and Supercomputing Flexible Definition of Multi-tier Applications Resource Management Scale-out and Provisioning Application Performance

7. 7/30 Bringing Private Cloud Computing to HPC and Science Main Challenges for Private HPC Cloud Using the Cloud – Execution of Multi-tiered Applications Management of interconnected multi-VM applications: Definition of application flows Catalog with pre-defined applications Sharing between users and groups Management of persistent scientific data Automatic elasticity Front-end Worker Nodes { "name": ”Computing_Cluster", "deployment": "straight", "roles": [ { "name": "frontend", "vm_template": 0 }, { "name": "worker", "parents": frontend, "cardinality": 2, "vm_template": 3, "min_vms" : 1, "max_vms" : 5, "elasticity_policies" : { ”expressions" : ”CPU> 90%”, "type" : "CHANGE", "adjust" : 2, "period_number" : 3, "period" : 10 }, …

8. 8/30 Bringing Private Cloud Computing to HPC and Science Main Challenges for Private HPC Cloud Using the Cloud – Performance Penalty as a Small Tax You Have to Pay Overhead in Virtualization Single processor performance penalty between 1% and 5% NASA has reported an overhead between 9% and 25% (HPCC and NPB) 1 Growing number of users demanding containers (OpenVZ and LXC) Need for Low-Latency High-Bandwidth Interconnection Lower performance, 10 GigE typically, used in clouds has a significant negative (x2- x10, especially latency) impact on HPC applications 1 FermiCloud has reported MPI performance (HPL benchmark) on VMs and SR- IOV/Infiniband with only a 4% overhead 2 The Center for HPC at CSR has contributed the KVM SR-IOV Drivers for Infiniband 3 (1) An Application-Based Performance Evaluation of Cloud Computing, NASA Ames, 2013 (2) FermiCloud Update, Keith Chadwick!, Fermilab, HePIX Spring Workshop 2013 (3) http://wiki.chpc.ac.za/acelab:opennebula_sr-iov_vmm_driver, 2013 Overhead in Input/Output Growing number of Big Data apps Support for multiple datastores including automatic scheduling

9. 9/30 Bringing Private Cloud Computing to HPC and Science Main Challenges for Private HPC Cloud Operating the Cloud – Resource Management Optimal Placement of Virtual Machines Automatic placement of VM near input data Striping policy to maximize the resources available to VMs Fair Share of Resources Resource quota management to allocate, track and limit resource utilization. Management of Different Hardware Profiles Resource pools (physical clusters) with specific Hw and Sw profiles, or security levels for different workload profiles (HPC and HTC) Isolated Execution of Applications Full Isolation of performance-sensitive applications

10. 10/30 Bringing Private Cloud Computing to HPC and Science Main Challenges for Private HPC Cloud Operating the Cloud – Scale out and Provisioning Multi-tier Deployment Management of multiple cloud instances that may be hosted in different sites Provide VOs with Isolated Cloud Environ Automatic provision of Virtual Data Centers Hybrid Cloud Computing Cloudbursting to address peak or fluctuating demands for no critical and HTC workloads

11. 11/30 Bringing Private Cloud Computing to HPC and Science Private HPC Cloud Case Studies One of Our Main User Communities Supercomputing Centers Research Centers Distributed Computing Infrastructures Industry

12. 12/30 Bringing Private Cloud Computing to HPC and Science Private HPC Cloud Case Studies FermiCloud NodesKVM on 23 nodes (1 TB RAM - 368 cores) Koi Computer NetworkGigabit and Infiniband StorageCLVM+GFS2 on shared 120TB NexSAN SataBeats AuthNX509 LinuxScientific Linux InterfaceSunstone Self-service and EC2 API App ProfileLegacy, HTC and MPI HPC http://www-fermicloud.fnal.gov/ Typical Workloads Scientific stakeholders get access to on- demand VMs Developers & integrators of new Grid applications

13. 13/30 Bringing Private Cloud Computing to HPC and Science Private HPC Cloud Case Studies CESGA Cloud NodesKVM on 27 nodes (0.5 TB RAM – 216 cores) HP ProLiant Network2 x Gigabit (1G and 10G) Storagessh from remote EMC storage server AuthNX509 and core password LinuxScientific Linux InterfaceSunstone Self-service and OCCI App ProfileIndividual VMs and virtualised computing clusters Typical Workloads 103 users Genomic, rendering… Grid services on production at CESGA Node at FedCloud project UMD middleware testing http://cloud.cesga.es/

14. 14/30 Bringing Private Cloud Computing to HPC and Science Private HPC Cloud Case Studies SARA Cloud NodesKVM on 19 HPC nodes (256 GB RAM 608 cores) Dell PowerEdge and 10 “light” nodes (64 GB RAM 80 cores) Supermicro Network4 x Gigabit (10G) with Arista switch StorageNFS on 400 GB NAS for HPC and ssh for “light” AuthNCore password LinuxCentOS InterfaceSunstone and OCCI App ProfileMPI clusters, windows clusters and independent VMs ww.cloud.sara.nl Typical Workloads Ad-hoc clusters with MPI and pilot jobs Windows clusters for Windows-bound software Single VMs, sometimes acting as web servers to disseminate results

15. 15/30 Bringing Private Cloud Computing to HPC and Science Private HPC Cloud Case Studies SZTAKI Cloud NodesKVM on 7 nodes (1.8 TB RAM – 448 cores) DELL PowerEdge Network2 x Gigabit (1G and 10G) StorageiSCSI on DELL storage server 72 TB shared AuthNX509 LinuxCentOS InterfaceSunstone Self-service, EC2 and OCCI App ProfileIndividual VMs and virtualised computing cluster http://cloud.sztaki.hu/. Typical Workloads Run standard and grid services (e.g.:web servers, grid middlewares…) Development and testing of new codes Research on performance and opportunistic computing

16. 16/30 Bringing Private Cloud Computing to HPC and Science Private HPC Cloud Case Studies KTh Cloud NodesKVM on 768 cores (768 GB RAM) HP ProLiant NetworkInfiniband and Gigabit StorageNFS and LVM AuthNX509 and core password LinuxUbuntu InterfaceSunstone self-service, OCCI and EC2 App ProfileIndividual VMs and virtualised computing cluster http://www.pdc.kth.se/ Typical Workloads Mainly BIO Hadoop, Spark, Galaxy, Cloud Bio Linux…

17. 17/30 Bringing Private Cloud Computing to HPC and Science Private Cloud Trends in Industry Experimenting with ARM for the Private Cloud Why? Decrease power consumption, reduce costs, simplify solutions… Mostly managing bare metal and early experiences with virtualization Tiniest Cloud Ever! (by Citrix and Linaro at LCU 2013) Ubuntu on Versatile Express Cortex-A15 Dual core Ubuntu on Arndale Board Cortex-A15 Dual core http://www.youtube.com/watch?v=xZP9YKv3P_E

18. 18/30 Bringing Private Cloud Computing to HPC and Science Private Cloud Trends in Industry Cloud for Mission-critical Applications Availability and redundancy to keep it running in case of failure Cloud services availability => HA Architectures Application availability => Failover Solutions Service Continuity (by European Aeronautic Company) OpenNebula 4.0 Automatic failover and recovery within 1 minute KVM

19. 19/30 Bringing Private Cloud Computing to HPC and Science Private Cloud Trends in Industry Hybrid Cloud Deployments Transparent and automatic access to the public cloud Dev&testing to the public cloud Security and performance sensitive workloads on the private cloud Cloudbursting Deployment (by Telecom Company) Public Cloud 1 Public Cloud 1 Public Cloud 2 Public Cloud 2 Local data center OpenNebula Private Cloud Cloud API is not relevant

20. 20/30 Bringing Private Cloud Computing to HPC and Science About Grid and Cloud What is the Difference between a Grid Site and a Cloud Provider? Definitions of Grid Site “A resource provider is a site which provides services and resources (e.g. data storage) to this VO“ (GridPP) “What makes a grid site a grid site?. A single grid resource (a grid site) offers compute and/ or storage services to remote users via standardized interfaces” (GridKa) “A typical (minimal) grid site provides computing and storage to supported Virtual Organizations (VOs) and runs a few services to make those resources visible on the grid” (StratusLab) Definitions of Cloud Provider “A cloud provider is a service provider that offers storage and compute resources on a private or public network“ (IBM) “(Cloud) Providers offer resources to the customer – either via dedicated APIs (PaaS), virtual machines and / or direct access to the resources (IaaS)” (EC Report on The Future of Cloud Computing)

21. 21/30 Bringing Private Cloud Computing to HPC and Science Virtual CE, WN… Other (web, mail...) Raw machines LRMS (LSF, PBS … ) Grid Middleware IaaS Interface Access Batch Job Processing Custom Execution Environments Grid Service Integration Industry Applications Other WMS (pilots) Complete Services (cluster) Grid SiteExternal Providers Provision Service About Grid and Cloud The OpenNebula Vision for Grid Sites: Extending the Range of Applications

22. 22/30 Bringing Private Cloud Computing to HPC and Science About Grid and Cloud What is the Difference between a Grid and Cloud Federation? Definitions of Grid Ian Foster’s definition lists these primary attributes: “Computing resources are not administered centrally, open standards are used, and nontrivial quality of service is achieved” Plaszczak/Wellner: “The technology that enables resource virtualization, on-demand provisioning, and service (resource) sharing between organizations” IBM: “The ability, using a set of open standards and protocols, to gain access to applications and data, processing power, storage capacity and a vast array of other computing resources over the Internet” CERN: “A service for sharing computer power and data storage capacity over the Internet” Definition of Cloud Federation “Cloud federation is the practice of interconnecting the cloud computing environments of two or more service providers for the purpose of load balancing traffic and accommodating spikes in demand”, Wikipedia

23. 23/30 Bringing Private Cloud Computing to HPC and Science Grid Services Grid APICloud APIGrid APICloud API Appliance Repo MarketPlace Cloud/Grid Site Sharing existing VM images Registry of metadata Image are kept elsewhere Supports trust Federation facilities Security Grid specific services Storage VM images Distributed Multi-protocol About Grid and Cloud The OpenNebula Vision for Grid Infrastructures, October 2008

24. 24/30 Bringing Private Cloud Computing to HPC and Science Clouds Grids Usage  Job Processing  Big Batch System  File Sharing Services Achievements  Federation of Resources  VO Concept But…  User experience  Complexity Usage  Raw infrastructure  Elasticity & Pay-per-use  Simple Web Interface Achievements  Agile Infrastructures  IT is another Utility But…  Interoperability  Federation Customize Environments Uniform Security Resource Management Scientific Applications Resource Sharing Flexibility & Simplicity About Grid and Cloud Grid and Cloud as Complementary Computing Models

25. 25/30 Bringing Private Cloud Computing to HPC and Science About Grid and Cloud EGI Federated Cloud Doing a Pioneering Work in the Field

26. 26/30 Bringing Private Cloud Computing to HPC and Science About Grid and Cloud Different Names for the Same Model? Same Challenges but Different Technologies? Grid Computing Cloud Computing

27. 27/30 Bringing Private Cloud Computing to HPC and Science Try it Out! OpenNebula Sandboxes ●OpenNebula pre-installed in a VM: VirtualBox, KVM, VMware, Amazon

28. 28/30 Bringing Private Cloud Computing to HPC and Science Join Us at OpenNebulaConf 2013!

29. 29/30 Bringing Private Cloud Computing to HPC and Science Visit our Booth! Interoperability Features in OpenNebula Wednesday, 18 12:10 - 12:25 Room: Escudo Building a OCCI-compatible Cloud with OpenNebula Thursday, 19 13:30 - 16:00 Room: Toledo

30. 30/30 Bringing Private Cloud Computing to HPC and Science Thanks to People and Organizations that Provided Info to Prepare this Presentation Questions? OpenNebula.org @OpenNebula

31. 31/30 Bringing Private Cloud Computing to HPC and Science A Single CMP Can Not Be All Things to All People OpenNebula vs. OpenStack? Eucalyptus, CloudStack, OpenStack and OpenNebula: A Tale of Two Cloud Models (http://blog.opennebula.org/?p=4042)

32. 32/30 Bringing Private Cloud Computing to HPC and Science Both Are Apache-licensed, Fully Open-source, Publicly Developed Technologies, but… OpenNebula vs. OpenStack? FeatureOpenStackOpenNebula Aim (Technical)Public cloud (AWS-like deployments)Private cloud & virtual datacenters (vCloud- like) FunctionalityUnique features for data center virtualization management, like VDCs, hybrid, multi- tiered application management… Integration Capabilities Very simple integration thanks to its plug-in based modular architecture by sys admins. OpenStack requires modifications in the code by experts Release ModelDeveloper community comprising different subprojects with different levels of maturity that require integration Enterprise open-source product for computing clouds with a single installing, patching and updating process Development Model Consensus-based approach where vendors try to meet the needs of the project and their monetization goals User-driven development with contributions from the users OpenStack, CloudStack, Eucalyptus and OpenNebula: Which Cloud Platform is the Most Open? (http://blog.opennebula.org/?p=4372)

33. 33/30 Bringing Private Cloud Computing to HPC and Science OpenNebula Represents Simplicity in the Specific Niche of Enterprise Cloud Computing OpenNebula vs. OpenStack? OpenNebula and the Beauty of Simplicity: Space Shuttle vs. Soyud, Alberto Zuin (http://blog.opennebula.org/?p=4738)