GRID AND CLOUD COMPUTING OpenStack Overview and Introduction Courtesy of Professor Ahmed Ezzat, Ph.D http://web.uettaxila.edu.pk/CMS/FALL/2017/teGNCCms/ www.svuca.edu

UNIT 15: OpenStack Overview and Introduction The Story of OpenStack Introduction What is OpenStack Learning OpenStack Summary and Conclusions I can pretty much read this one straight through and provide details in following slides.

The Story of OpenStack

The E-mail That Started It All

NASA Nabula Nebula is an open source cloud computing platform that was developed to provide an easily quantifiable and improved alternative to building additional expensive data centers and to provide an easier way for NASA scientists and researchers to share large, complex data sets with external partners and the public. Nebula's high-density architecture allows for a dramatically reduced data center footprint. Each shipping container data center can hold up to 15,000 CPU cores or 15 petabytes (one petabyte equals one million gigabytes), proving 50 percent more energy efficient than traditional data centers. https://www.nasa.gov/open/nebula.html

The Birth of OpenStack Nebula

135 338 15,672 2130 374 115,206 COMPANIES COUNTRIES TOP 10 COUNTRIES INDIVIDUAL MEMBERS United States, China, India, Great Britain, France, Russia, Australia, Canada, Japan, Germany 15,672 AVERAGE MONTHLY CONTRIBUTORS TOTAL DEVELOPERS CODE CONTRIBUTIONS 2130 374 115,206

What is OpenStack?

OpenStack Mission "To produce the ubiquitous Open Source cloud computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to implement and massively scalable."

OpenStack Founding Principles Apache 2.0 license (OSI), open development process Open design process, 2x year public Design Summits Publicly available open source code repository Open community processes documented and transparent Commitment to drive and adopt open standards Modular design for deployment flexibility via APIs

Who’s Behind Openstack Platinum Members Who’s Behind Openstack Gold Members

Community with Broad Commercial Support

OpenStack Release Schedule Essex: April 5, 2012 Diablo: September 22, 2012 Nova, Swift, Glance, Horizon, Keystone Cactus: April 15, 2011 Nova, Swift, Glance Bexar: February 3, 2011 Followed by conference and design summit in Boston in early October Nova, Swift, Glance Austin: October 21, 2010 OpenStack Compute ready for large service provider scale deployments This is the ‘Rackspace-ready’ release; need to communicate Rackspace support and plans for deployment Nova, Swift, Glance OpenStack Compute ready for enterprise private cloud deployments and mid-size service provider deployments Enhanced documentation Easier to install and deploy Nova, Swift Initial Release of OpenStack Combining the two projects Nova and Swift

Last Few Releases of OpenStack 16th Release of Openstack, Released on August, 2017 15th Release of Openstack, Released on February, 2017 14th Release of Openstack, Released on October, 2016 13th Release of Openstack, Released on April, 2016 12th Release of Openstack, Released on October, 2015 11th Release of Openstack, Released on April, 2015 10th Release of Openstack, Released on October, 2014 9th Release of Openstack, Released on April, 2014 8th Release of Openstack, Released on October, 2013 7h Release of Openstack, Released on April, 2013 6th Release of Openstack, Released on September, 2012

Why Openstack for Cloud It's Open Source : All of the code for OpenStack is freely available under the Apache 2.0 license. Anyone can run it, build on it, or submit changes back to the project. Who it's for: Enterprises, service providers, government and academic institutions with physical hardware that would like to build a public or private cloud. How it's being used today: Organizations like Cisco WebEx, DreamHost, eBay, The Gap, HP, MercadoLibre, NASA, PayPal, Rackspace and University of Melbourne have deployed OpenStack clouds to achieve control, business agility and cost savings without the licensing fees and terms of proprietary software.

Public Clouds vs. Private Clouds Public clouds are cloud systems that are available for everyone’s use. Public clouds services can be both free and subscription based, depending on the user’s needs and provider’s business policy.

Private Clouds A private cloud can offer the same services as a public cloud. Its services are limited to people behind the company's firewall.

Automation and Orchestration of IT Resources A Cloud Computing platform sits above the virtual data center and provides both a control plane over and resource access to the virtualized data center. OpenStack, as a Cloud Computing platform, manages virtualized resources, such as virtual machines exported by a hypervisor, network overlays created by Software-Defined Network devices, and volumes exported by virtual storage arrays. OpenStack takes these data center resources and automates and orchestrates them so they can be accessed on demand and be scaled up and down as needed, turning these resources into consumable services.

In a Loosely Coupled Architecture

By Leveraging Various Open Source Projects The OpenStack platform is actually composed of multiple components, called projects. Each project is managed by a technical committee and the OpenStack Foundation decides which projects are ready to be included in the OpenStack core. These projects work together to provide the services required to deliver the Cloud. Nova – The compute project responsible for on-demand creation and termination of compute instances. Nova leverage a number of hypervisors, including KVM, Xen, Hyper-V, and vSphere. Glance – The OS image management project responsible for storage and management of images used to create compute instances with OSes installed, such as Windows and Linux. Quantum – The network project that provides network access and security services to compute instances. Quantum uses plugins to leverage virtual switches and SDN-enabled devices. Swift – The object storage project that provides a scalable repository for storing large quantities of objects such as files and media content. It can also be used as an repository for Glance images. Cinder – The block storage project that provides a virtual storage array that can export out iSCSI volumes. A Cinder/virtual storage array server can be a server with local storage or a server using an external storage array. Horizon – The interactive dashboard project that provides users and admins provisioning and management access to the OpenStack Cloud via a web GUI. Keystone – The identity management project that provides authorization and access security control for all the other OpenStack projects. New projects are being added with each release and as the OpenStack community calls for them. New projects underway include metering, application orchestration, and database-as-a-service.

OpenStack Architecture

OpenStack Architecture

Horizon is a web-based interface for managing OpenStack services. Horizon (Dashboard) Horizon is a web-based interface for managing OpenStack services. It provides a graphical user interface for operations such as launching instances, managing networking and setting access controls. Its modular design allows interfacing with other products such as billing, monitoring and additional management tools.

Keystone (Identity) Keystone is the centralized identity service that provides authentication and authorization for other services. Keystone also provides a central catalog of services running in a particular OpenStack cloud. It supports multiple forms of authentication including user name and password credentials, token-based systems, and Amazon Web Services style logins.

Neutron (OpenStack Networking) OpenStack Networking provides connectivity between the interfaces of other OpenStack services, such as Nova. OpenStack Networking is a pluggable architecture, users can create their own networks, control traffic, and connect servers to other networks. A software defined networking service that supports many plugins like Open vSwitch, Cisco UCS/Nexus, QoS etc.

Cinder is a service that manages storage volumes for virtual machines. Cinder (Block Storage) Cinder is a service that manages storage volumes for virtual machines. This is persistent block storage for the instances running in Nova. Snapshots can be taken for backing up and data, either for restoring data, or to be used to create new block storage volumes.

Nova (Compute) Compute nodes form the resource core of the OpenStack Compute cloud, providing the processing, memory, network and storage resources to run instances. Nova is a distributed component and interacts with Keystone for authentication, Glance for images and Horizon for web interface. Nova is designed to scale horizontally on standard hardware, downloading images to launch instances as required.

Images can be used as templates when setting up new servers. Glance (Image Service) Glance service that acts as a registry for virtual machine images allowing users to copy server images for immediate storage. Images can be used as templates when setting up new servers. Usually the images are stored in the Swift (Object) service.

Swift service providing object storage which allows users to store Swift (Object Storage) Swift service providing object storage which allows users to store and retrieve files. Swift architecture is distributed to allow for horizontal scaling, and to provide redundancy as failure-proofing. Data replication is manage by software, allowing greater scalability and redundancy than dedicated hardware.

creating open source software to build public and private clouds Software to provision virtual machines on standard hardware at massive scale OpenStack Compute creating open source software to build public and private clouds Software to reliably store billions of objects distributed across standard hardware OpenStack Object Storage

OpenStack Compute Key Features ReST-based API Asynchronous eventually consistent communication  Horizontally and massively scalable Hypervisor agnostic: support for Xen ,XenServer, Hyper-V, KVM, UML and ESX is coming Hardware agnostic:  standard hardware, RAID not required

Example OpenStack Compute Hardware (other models possible) Public Network Server Groups Dual Quad Core RAID 10 Drives 1 GigE Public 1 GigE Private 1 GigE Management Private Network (intra data center) Management

OpenStack Compute User Manager Advanced Technology Attachment over Ethernet / Internet Small Computer Systems Interface Cloud Controllers: Global state of system, talks to LDAP, OpenStack Object Storage, and compute/storage/network workers through a queue ATAoE / iSCSI API: Receives HTTP requests, converts commands to/from API format, and sends requests to cloud controller Host Machines: workers that spawn instances Glance: HTTP + OpenStack Object Storage for server images OpenStack Compute

System Components API Server: Interface module for command and control requests Designed to be modular to support multiple APIs In current release: OpenStack API, EC2 Compatibility Module Approved blueprint: Open Cloud Computing Interface (OCCI) Message Queue: Broker to handle interactions between services Currently based on RabbitMQ Metadata Storage: ORM Layer using SQLAlchemy for datastore abstraction In current release: MySQL In Diablo: PostgreSQL User Manager: Directory service to store user identities In current release: OpenLDAP, FakeLDAP (with Redis), Database Scheduler: Determines the placement of a new resource requested via the API Modular architecture to allow for optimization Base schedulers included in Bexar: Round-robin, Least busy

System Components (Cont.) Compute Worker: Manage compute hosts through commands received on the Message Queue via the API Base features: Run, Terminate, Reboot, Attach/Detach Volume, Get Console Output Network Controller: Manage networking resources on compute hosts through commands received on the Message Queue via the API Support for multiple network models Fixed (Static) IP addresses, VLAN with NAT, DHCP Volume Worker: Interact with iSCSI Targets to manage volumes Base features: Create, Delete, Establish Image Store: Manage and deploy VM images to host machines

New Features in Diablo and Beyond Quantum: Networking as a Service Developed in the open by Cisco, Nicira, others Burrow: HTTP-based message queue Red Dwarf: Database as a Service Keystone: Integrated, pluggable auth for all OpenStack components Lunr: Volumes as a Service Dashboard: Control nova and other OpenStack components via web

Object Storage Summary Distributed, REST-based API, No central database Hardware agnostic - commodity hardware, RAID not required Account/Container/Object structure (not file system, no nesting) Replication (N copies of accounts, containers, objects)  Data distributed evenly throughout system Scalable to multiple petabytes, billions of objects 3838 Object Storage Summary

Hardware agnostic: standard hardware, RAID not required Object Storage Key Features Data distributed evenly throughout system Scalable to multiple petabytes, billions of objects REpresentational State Transfer ReST-based API No central database Hardware agnostic: standard hardware, RAID not required

Example Large Scale Deployment -- Many Configs Possible Example OpenStack Object Storage Hardware  To Load Balancers Proxies 5 Zones 2 Proxies per 25 Storage Nodes 10 GigE to Proxies 1 GigE to 24 x 2TB Drives per Storage Node Example Large Scale Deployment -- Many Configs Possible

Proxy Server: Request routing, exposes the public API Replication: Keep the system consistent, handle failures Updaters: Process failed or queued updates Auditors: Verify integrity of objects, containers, and accounts System Components

System Components (Cont.) Account Server: Handles listing of containers, stores as SQLite DB Container Server: Handles listing of objects, stores as SQLite DB Object Server: Blob storage server, metadata kept in xattrs, data in binary format Object location based on hash of name & timestamp System Components (Cont.)

Evolution of Object Storage Architecture Version 1: Central DB (Rackspace Cloud Files 2008) Version 2: Fully Distributed (OpenStack Object Storage 2010)

Example Small Scale Deployment $0.61/kWh, $499 per “half cabinet” – 1 server per cabinet // CAPEX is servers with DAS and switches from a tier 1 hardware vendor at list price // Electricity is the majority of the cost 4444 Example Small Scale Deployment

Learning OpenStack

Reading About OpenStack The OpenStack Foundation http://www.openstack.org/ Official OpenStack Documentation http://docs.openstack.org/ The OpenStack Cloud Computing Cookbook (Second Edition) http://www.amazon.com/OpenStack-Cloud-Computing-Cookbook-Jackson/dp/1782167587/ref=sr_1_1?s=books&ie=UTF8&qid=1382033707&sr=1-1

Trying Out OpenStack TryStack (OpenStack Sandbox) http://trystack.org/ OpenStack-based Public Clouds DreamHost http://dreamhost.com/cloud/ HP Public Cloud https://www.hpcloud.com/ Rackspace Public Cloud http://www.rackspace.com/cloud/

Deploying OpenStack OpenStack Distributions Red Hat - http://openstack.redhat.com/ SUSE - https://www.suse.com/products/suse-cloud/ Ubuntu - http://www.ubuntu.com/cloud Packaged Deploys For Different Linux Distros Mirantis - https://fuel.mirantis.com/ Piston Cloud Computing - http://www.pistoncloud.com/openstack-cloud-software/ Rackspace - http://www.rackspace.com/cloud/private/openstack_software/ Configuration Management Tools Opscode Chef - https://github.com/opscode/openstack-chef-repo/ Puppet Labs Puppet -http://puppetlabs.com/solutions/cloud-automation/compute/openstack

Join the Community Join The OpenStack Community http://www.openstack.org/community/ The OpenStack community is a growing body that spans the globe. Individuals and companies can join the community and learn from other members via IRC chat, mailing lists, and forums.

Assignment #14 What are the names of Service Projects included in latest version of OpenStack (Pike)?

Questions and Comments? Thank You Questions and Comments? http://web.uettaxila.edu.pk/CMS/FALL/2017/teGNCCms/