1. OpenStack High Availability
Jakub Pavlik

2. About me Jakub Pavlík Cloud Platform Engineer 3 years in Cloud
2 years in OpenStack

3. High Availability ≠ Disaster Recovery!
High Availability vs. Disaster Recovery
High Availability = fault detection & correction procedures to maximize availability of critical services and applications, often in an automated fashion.
Disaster Recovery = process of preparing for recovery or continuation of technology infrastructure critical to an organization after a natural or human-induced disaster.
High Availability ≠ Disaster Recovery!

4. Four types of HA in an OpenStack Cloud
Service Resiliency
QoS Cost
Transparency
Data Integrity
…..
Compute Controller
Network Controller
Database
Message Queue
Storage
....
Physical infrastructure
OpenStack
Control services
VMs
OpenStack Compute
Applications
Physical nodes
Physical network
Physical storage
Hypervisor
Host OS ….
Virtual Machine
Virtual Network
Virtual Storage
VM Mobility …

5. Physical Infrastructure

6. tcp cloud VPC Hardware Switch 1 Switch 2 SAN 1 SAN 2 SAN 1 SAN 2
Passthru 1
Passthru 2
Passthru 1
Passthru 2
168 cores 3,46GHz ,336 threads
agregation ¼ : 1344 vCPU
2688 GB RAM
28 x 10GE ports
168 cores 2,67GHz ,336 threads
agregation ¼ : 1344 vCPU
1792 GB RAM
28 x 10GE ports
SAN 1
SAN 2
SAN 1
SAN 2
Controller 1
Controller 2
Controller 1
Controller 2

7. OpenStack Control services

8. OpenStack modules – TCP VPC

9. OpenStack High Availability Concepts
Stateless services
There is no dependency between requests
For example APIs: Nova, Keystone, Glance, Cinder, etc.
Stateful services
An action typically compromises multiple requests
For example: MySQL, RabbitMQ, etc.
Active/Passive
Redundant instances of stateless services are load balanced
For Stateful services a replacement resource can be brought online
Active/Active
Stateful services are managed in such a way that services are redundant, and that all instances have and identical state.

10. Corosync, Pacemaker and HAProxy
Totem single-ring ordering and membership protocol
UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker
Pacemaker
High availability and load balancing stack for the Linux platform.
Interacts with applications through Resource Agents (RA)
HAProxy
Load Balancing and Proxying for HTTP and TCP Applications
Works over multiple connections
Used to load balance API services

11. MySQL Galera
Synchronous multi-master cluster technology for MySQL/InnoDB
MySQL patched for wsrep (Write Set REPlication)
Active/active multi-master topology
Read and write to any cluster node
True parallel replication, in row level
No slave lag or integrity issues

12. RabbitMQ – RPC messaging
Rabbit cluster

13. Sample OpenStack HA architecture
Stateful
Cinder Volume
Neutron L3, DHCP agents
Ceilometer central agent
RabbitMQ
Stateless
Neutron Server
OpenStack APIs
Apache web server
Nova Scheduler
Cinder Scheduler
Neutron agents
(Active)
Neutron agents
(Hot Standby)

14. VMs – Compute nodes

15. VMs HA – two layers Storage
Shared storage filesystem – file disks (qcow2, vmdk, vhv)
Block storage
Network
Vanilla Neutron L3 agent (OpenVSwitch, Linux Bridge)
Vendor plugins - SDN controller

16. No vSphere Style HA with KVM

17. Non-Shared/Shared Storage filesystem
Live migration – just RAM memory
Hypervisor Evacuation – The instance will be booted from same disk and data will be preserved
CEPH, Gluster, NFS, Samba, GFS
Non-Shared Storage
Block Live Migration – disk and RAM
Hypervisor Evacuation – the instance will be booted from a new disk, but will preserve the configuration, e.g. id, name, uuid
Standard filesystem EXT4, etc.

18. Block Storage - Cinder Instance boots from volume
iSCSI/FC direct mapping to instance
Enable Live Migration
Cinder Backends
LVM Driver
Default linux iSCSI server
Vendor software plugins
Gluster, CEPH, VMware VMDK driver
Vendor storage plugins
EMC VNX, IBM Storwize, Solid Fire, etc.

19. Networking - Vanilla Neutron L3 agent
Problems
Routing on Linux server (max. bandwith approximately 3-4 Gbits)
Limited distribution between more network nodes
East-West and North-South communication through network node
High Availability
Pacemaker&Corosync
Keepalived VRRP
DVR + VRRP – should be in Juno release

20. Networking – Vendor SDN Controller plugins
Examples
Juniper OpenContrail, VMware NSX, SDN PLUMgrid
Advantages against Neutron L3 agent
North-South communication on network devices (iBGP, MLPSoverGRE)
East-West communication directly between compute nodes
Higher bandwidth (9.7 Gbits per 10Gbits port)
High Availability
iBGP peering into two routers
Native HA implemented inside of network devices

21. OpenStack HA TCP VPC VIP HAProxy Pacemaker Corosync
MySQL
RabbitMQ
Openstack Controller
GALERA
Zookeeper
Cassandra
Contrail Database
Contrail Config with Analytics & WebUI
Contrail Control
HAProxy
VIP
Bond Interface
Pacemaker
Corosync

22. TCP Virtual Private Cloud

23. HA methods - vendors Vendor Cluster/Replication Technique
Characteristics
RackSpace
Keepalived, HAProxy, VRRP, DRBD
Automatic - Chef
Red Hat
Pacemaker, Corosync, Galera
Manual installation/Foreman
Cisco
Keepalived, HAProxy, Galera
Manual installation, at least 3 controller
tcp cloud
Pacemaker, Corosync, HAProxy, Galera, Contrail
Automatic Salt-Stack deployment
Mirantis
Pacemaker, Corosync, HAProxy Galera
Automatic - Puppet

24. Thank you for your attention!