1. How Adobe Built An OpenStack Cloud
Jun Park (Ph.D, MBA), Solutions Architect At Adobe
Arghya Banerjee, Sr. Systems Engineer At Adobe
OpenStack Mitaka Summit At Tokyo, Oct 2015

2. Swiss Cheese Model Flaws In Defense layers
If aligned, flaws would allow an accident to occur
From Wikipedia

3. Two More Factors That Complicate Things
SpaceTime Continuum
- Einstein
Interactions,
Higgs Field & Boson
From Wikipedia
From Youtube

4. Our Template To Analyze
Components
Dependencies
In Red: Bugs or Issues
In Green: Fix or Stable
Time

5. OpenStack Survey, May 2015 The most common arch:
Ubuntu + KVM + OVS + Ceph

6. Adobe OpenStack Architecture
Storage: Ceph RBD
VM1
VM2
VM3
eth0
eth1
eth0
eth1
eth0
eth1
Private Networks: VxLAN-based
External Provider Networks: VLAN-based
Adobe Network Firewall
Adobe Corporate Networks

7. What Happened At Networking?
A New Bug: OVS Sporadically Crashes In Adding A Port
( and )
Restarting agents
re-establishes entire flows
Neutron
Fix ready, not added
Security Group
O(N^2) Issue
Enhancement Patch
Not Yet Integrated
(e.g., 270 secs to 3 secs
For 25K rules)
OVS 2.0.1
Released:
Mega Flow
Multiprocessing
OpenvSwitch
(OVS)
This Bug
Introduced with
OVS Mega Flow
OVS 2.3.0
OVS 2.1.3
OVS 2.0.2
Released
Bug Fix
In all OVS 2.x
Ubuntu 14.04
Trusty Released
With OVS 2.0.1
Bug Report
With OVS 2.0.1
In Ubuntu 14.04
Cherry-Pick
On OVS 2.0.2
In Ubuntu
Ubuntu 14.04
Jun ‘13
Dec ‘13
Apr ‘14
Jul ‘14
Aug ‘14
May ‘15

8. What Happened At Networking?
A New Bug: OVS Sporadically Crashes In Adding A Port
( and )
OVS 2.0.1
Released:
Mega Flow
Multiprocessing
OVS
Some companies reverted OVS to LinuxBridge!
Some pundits spread FUD about Neutron!
OpenStack
Summits
Atlanta
IceHouse
Paris
Juno
Vancouver
Kilo
Cherry-Pick
Onto OVS 2.0.2
In Ubuntu 14.04
Ubuntu 14.04
Trusty Released
With OVS 2.0.1
Ubuntu 14.04
Dec ‘13
Apr ‘14
May ‘14
Nov ‘14
May ‘15

9. What Happened At Storage?
Ceph Operational Instability,
Cinder Scalability Issue
Cinder is stuck
when Ceph is stuck
(e.g., use local drive
for copying an image)
Enhancement Solution
Not Yet Integrated
(e.g., APIs Stacked Up ->
Multiprocessing)
Cinder
Ceph
Failover Instability
With FireFly
Hammer?
Ubuntu 14.04
Trusty Released
With Ceph FireFly 0.79
Ubuntu 14.04
Ubuntu Updates
With Ceph FireFly
Apr ‘14
May ‘14
July ‘15

10. What Happened At Data Node?
Kernel Memory Bug,
Security Issue
KVM
Security Issue
Security Patch
XFS
Deadlock
Bug
Kernel
Bug Fix
Ubuntu 14.04
Trusty Released
With Kernel…
Ubuntu 14.04
Trusty Released
With Kernel…
Ubuntu 14.04
Dec‘13
Apr ‘14
May ‘14
Nov‘14
May ‘15
July ‘15

11. Check List Networks Storage Understand OVS and find stable OVS
Cherry-pick for Neutron Scalability: firewall rules
Our own out-of-band rate limiting on networks, e.g., 200 Mbps
Set up right MTU size on OVS structure
Turn off GRO/LRO on hosts
Storage
Decouple Storage system from OpenStack API services
Cinder Scalability
Ceph Stability: Hammer, reconfigure towards optimal

12. How To Test at Scale Emulate future production env
Create hundreds of VMs, inject workloads, and destroy all
Recycle this entire test over and over again
Findings: dead tokens stacked up
Each component scalability
Neutron: OVS
Cinder: Ceph
Nova: KVM

13. Have We Done Enough? 4? 3?

14. It's not that I'm so smart, it's just that I stay with problems longer.
- Albert Einstein

15. New Efforts In OpenStack
OpenStack Product Working Group
Link up between contributors and users
Governance/DefCoreCommittee
Defining OpenStack Core
Large Deployment Team
Operational issues for large delpoyments
Open Virtual Network (OVN)
In-kernel Conntrack, DPDK, etc. Will run atop OVS

17. APPENDIX

18. USE CASE: Mesos Cluster

19. Possible Combinations
Containers
In Containers
Bare Metals
Containers
VMs
VMs

20. Mesos Cluster Via Heat Host1 Host2 Host3 VM2: mesos slave1
VM1: mesos master
Marathon
Zookeeper
http server
http server
-> Ubuntu-mesos image
available via diskimage-builder
-> Post configuration for master
-> starting services
-> Ubuntu-mesos image
-> Post configuration for slave
using mesos master IP.
-> starting services

21. Mesos Cluster with Marathon
Request to run a micro-service
via REST API
Marathon
Mesos Master
With ZooKeeper
Mesos Slave1
http server
Mesos Slave2
http server

22. Master + 2 slaves: Heat Stacks

23. Topology of Slave2

24. Marathon: Two Apps on Slave1

25. App Running On Slave

26. Mesos UI

27. Heat Template
Components
Dependencies
Time

28. Adobe OpenStack Architecture
VM1
eth0
eth1
Linux Bridge
OpenvSwitch
bond0
Physical VLANs
External Provider Networks: VLAN-based
Adobe Network Firewall
Adobe Corporate Networks

29. Volume Management in OpenStack
Glance API Server
Set of Images
Image1: Ubuntu Trusty
Cinder API Server
1. Copy
Volume1 : Ubuntu Trusty
Copy-On-Write (COW)
Ceph Volume
2. Snapshot
Snapshot1: Ubuntu Trusty
Base Volume For
All Three VMs
3. Volumes
New Volume1 for VM1
Individual COW Volumes
New Volume2 for VM2
New Volume3 for VM3