1. Marrying OpenStack and Bare-Metal Cloud
Pete Lumbis & David Iles
May 2018

2. Our Speakers Pete Lumbis David Iles Technical Marketing Engineer
Cumulus Networks
David Iles
Senior Director – Ethernet Switching
Mellanox Technologies

3. Multi-Tenant Networks
Single tenant private cloud is easy
Single ownership
Single security policy
Multi-tenant is more challenging
Possible competitor tenants
(mixing these is real bad)
Shared infrastructure
Multiple security policies

4. Multi-Tenant Networks
VLANs
Isolate traffic with network tags (802.1q) over shared physical links
Only Layer 3 Routers can cross VLANs
ML2 can provision VLANs on compute or physical switches
ML2 VLAN Provisioning

5. VLANs Are Not Your Friend
Pros:
Solve multi tenancy networking

6. VLANs Are Not Your Friend
Pros:
Solve multi tenancy networking
Cons:
Single network path

7. VLANs Are Not Your Friend
Pros:
Solve multi tenancy networking
Cons:
Single network path
Low scalability
Scale-Up networking, not scale-out
Limited VLAN range (4096 at best)

8. VLANs Are Not Your Friend
Pros:
Solve multi tenancy networking
Cons:
Single network path
Low scalability
Scale-Up networking, not scale-out
Limited VLAN range (4096 at best)
Large blast Radius
Impacted
Server NIC Failure

9. Beyond VLANs: VxLAN
Same L2 extension and isolation as VLANs (VxLAN ID)
Operate over L3 network
Small blast radius
Enables scale-out networking
Extremely high bandwidth network
Smaller ML2 Footprint
ML2 VxLAN Provisioning

10. Compute Based VxLAN VxLAN from compute to compute
No VxLAN based switches
BGP to the server for simplicity
Same config, troubleshooting for both network and compute
No Layer 2
BGP advertises compute endpoint (VTEP)
ML2 provisions VxLANs per tenant
ML2 VxLAN Provisioning

11. Compute Based VxLAN – The Ugly
VxLAN capable NICs required
Most servers have them, but not all.
Performance hit without NIC offload
BGP to the server is scary
It’s not really, but not everyone likes the idea
Network doesn’t want server folks touching their network
Server folks don’t want to learn BGP
Complicated solution for smaller deployments
Difficult for Ironic (bare metal)

12. Network Design Recap VLAN Based VxLAN Compute + Network Based
Bad, don’t do it
Even at a few racks
VxLAN Compute + Network Based
Great, but still requires network ML2
Loss of a switch == loss of a lot of Neutron state
Still requires VxLAN NICs
VxLAN Compute Only
Best solution if you are okay with BGP on servers
Not the easiest for Ironic

13. Network Design Needs Stability Simplicity Flexibility Scalability
It isn’t cloud if one host brings down the others
Simplicity
I don’t need the Avengers to run the infrastructure
Flexibility
Works for bare metal and virtual workloads
Scalability
Maybe not 1000s but 100s of tenants can be required
Affordable
VxLAN NICs may not be an option

14. The Answer… VLAN based compute + VxLAN based network
Wait….
+ VxLAN based network
But it’s not Hierarchical Port Binding
HPB manages both VxLAN and VLANs
ML2 only drives compute VLANs
Network is pre-provisioned
Offload VxLAN to the network
No need for special NICs
Localized VLANs easily and safely scales to 100s of tenants
Larger scale requires more complex solutions

15. VLAN + VxLAN VxLANs pre-configured between all switches
Compute facing network ports have VLAN trunks configured
ML2 provisions VLANs on compute nodes as needed
Every VLAN is mapped to an identical VxLAN Tunnel (VNI)
ML2 VLAN Provisioning
VxLANs
VLANs

16. VxLAN Without a Controller?
Switches need VxLAN knowledge
WHO: else is a VxLAN end point (VTEP)
WHAT: VxLAN tunnels exist on each VTEP
WHERE: do MAC addresses live
Option 1: VxLAN Controllers
Openstack ML2
OpenContrail/Tungsten Fabric
Open Daylight
Option 2: EVPN
Switches exchange data without a controller
Relies on extensions to BGP
Multi-vendor (Cumulus, Cisco, Arista, Juniper)
VxLAN Information
Controller
VxLAN Information

17. EVPN: A Closer Look A B

18. EVPN: A Closer Look
Switch build BGP relationships
BGP A B

19. EVPN: A Closer Look Switch build BGP relationships
Switch learns MAC addresses from servers
Just like a normal switch
I know about the MAC for server B!
I know about the MAC for server A! A B

20. EVPN: A Closer Look Switch build BGP relationships
Switch learns MAC addresses from servers
Just like a normal switch
MAC information exchanged via BGP
Come to me to reach MAC A
Come to me to reach MAC B A B

21. EVPN: A Closer Look Switch build BGP relationships
Switch learns MAC addresses from servers
Just like a normal switch
MAC information exchanged via BGP
Data sent via VxLAN based on BGP information
From: Switch A
To: Switch B
(VxLAN)
From: Server A
To: Server B A B

22. EVPN: A Closer Look Switch build BGP relationships
Switch learns MAC addresses from servers
Just like a normal switch
MAC information exchanged via BGP
Data sent via VxLAN based on BGP information
Since BGP is used, no shared failures like L2 A B

23. BaGPipe – A Potential Future
Neutron work with BGP called BaGPipe
Two Goals:
Inter-DC VPNs
Layer 3 to the compute node
Can be done today with Free Range Routing + Neutron Networking
What was described earlier
BaGPipe would have Neutron control BGP + VxLAN
Today’s solution Neutron only controls VxLAN
Nearly identical to EVPN on the server
Extremely early days, but value is clear
Early for BaGPipe as well

24. How do we make it work with next gen workloads?
Machine Learning and NVME Fabrics
Next Generation Storage:
Machine Learning Applications
All Flash
GPU accelerated
PCIe attached NVME drives
PCIe attached GPU’s
RDMA over Ethernet – RoCE
RDMA over Ethernet - RoCE
Both must run over an Overlay network
RoCE + VXLAN
ML2 VxLAN Provisioning

25. EVPN Gotcha’s License-free Features: BGP, VXLAN, ZTP, EVPN
VXLAN Routing in Hardware
No Loopback cables
VTEP Scale
Many switches max out at 128 VTEPs
ROCE over VXLAN
NVME over Fabric
Machine Learning

26. Questions??