Presentation is loading. Please wait.

Presentation is loading. Please wait.

Chad Norgan Systems Engineer BeardyMcBeards in #openvswitch Managing Open vSwitch Across a Large Heterogeneous Fleet.

Similar presentations


Presentation on theme: "Chad Norgan Systems Engineer BeardyMcBeards in #openvswitch Managing Open vSwitch Across a Large Heterogeneous Fleet."— Presentation transcript:

1 Chad Norgan Systems Engineer BeardyMcBeards in #openvswitch Managing Open vSwitch Across a Large Heterogeneous Fleet

2 2 About Rackspace 9 Worldwide Data Centers 5,000+ Rackers 300,000+ Customers ≅ 70 PB Stored Global Footprint Customers in 120+ Countries Portfolio of Hosted Solutions Dedicated - Cloud - Hybrid Annualized Revenue Over $1B 60% 100 OF THE We Serve FORTUNE ® OVER

3 Rackspace’s Public Cloud Large Fleet Heterogenous Several different hardware manufacturers Several XenServer major versions (sometimes on varying kernels) Five networking configurations Six production public clouds Six internal private clouds Various non-production environments Tens of thousands of hypervisors Hundreds of thousands of virtual machines Interfaces

4 Networks Available to Customers

5 Rackspace has used Open vSwitch since the 0.9 version Behind most of First Generation Cloud Servers (Slicehost) Powers 100% of Next Generation Cloud Servers Upgraded OVS nine times since the launch of Next Gen Public Cloud in August 2012 Our History With OVS

6 Service provider features: –Overlay Networks –QoS –VLAN Tagging –Port Security –LACP Software = Flexible Upgrades are easier than hardware Why We Use OVS

7 Our Favorite Improvements Save & restore datapath flows during kmod reload OVS 1.7 Logging removed from main loop, faster flow setups OVS 1.9 Collapsed data path & flow-eviction-threshold raised to 2500 OVS 1.10 Megaflows & wildcarding OVS 1.11 Multi-treading! OVS 2.0 flow-limit replaces flow-eviction-threshold & TCP flags OVS 2.1

8 Example: Busy HV With Syslog Collector

9 Mission Accomplished! We moved the bottleneck! New bottlenecks: ●Guest OS kernel configuration ●Xen Netback/Netfront Driver

10 Matching the OVS kernel module to both the running and staged kernel Hypervisor updates often come with a newer kernel We often don’t immediately reboot Running kernel != Kernel at next reboot Detect both kernels and install both sets of OVS kernel modules Heterogeneous Scale Challenges of Upgrading OVS

11 Playbook-style upgrades Asynchronous plays with parallel limits Extensible Easy to build validations and pre-checks to prevent unwanted impact OVS Upgrade Solution

12 Architectural Basics Patch Port Tunnel Encapsulation VIF PIF VIF PIF Integration Bridge Interface Bridge

13 Ansible + OVS = Flexible Network Rewiring Tunnel Encap VIF PIF VIF Integration Bridge Interface Bridge Patch Port PIF

14 Ansible + OVS = Flexible Network Rewiring Tunnel Encap VIF PIF VIF Integration Bridge Interface Bridge Patch Port PIF Public Net Bridge Public Net Bridge Patch Port

15 Ansible + OVS = Flexible Network Rewiring Tunnel Encap VIF PIF VIF Integration Bridge Interface Bridge Patch Port PIF Public Net Bridge Public Net Bridge Patch Port

16 Ansible + OVS = Flexible Network Rewiring Patch Port Tunnel Encap VIF PIF VIF Integration Bridge Public Net Bridge Public Net Bridge Service Net Bridge Interface Bridge Patch Port

17 Ansible + OVS = Flexible Network Rewiring Patch Port Tunnel Encap VIF PIF VIF Integration Bridge Public Net Bridge Public Net Bridge Service Net Bridge Cloud Net Bridge Interface Bridge Patch Port

18

19 Ansible + OVS = Flexible Network Rewiring Patch Port PIF VIF Public Net Bridge Public Net Bridge Public Net Bridge Interface Bridge Patch Port Public Net Bridge_old

20 Measuring OVS – PavlOVS.py Publishes metrics to StatsD/Graphite Per bridge byte, packet, open flow count Datapath hit, missed, lost, flow counts Open vSwitch CPU utilization Instance count Tunnels configured and in fault state

21 2000 Eviction Threshold

22

23 Hit, Miss, Lost

24 OVS CPU

25 The OVS Of Our Dreams Connection Tracking More (efficient) performance JSON Output from ovs-*ctl commands

26 THANK YOU RACKSPACE® | 1 FANATICAL PLACE, CITY OF WINDCREST | SAN ANTONIO, TX US SALES: | US SUPPORT: | © RACKSPACE LTD. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. |

27 THANK YOU RACKSPACE® | 1 FANATICAL PLACE, CITY OF WINDCREST | SAN ANTONIO, TX US SALES: | US SUPPORT: | © RACKSPACE LTD. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | QUESTIONS?


Download ppt "Chad Norgan Systems Engineer BeardyMcBeards in #openvswitch Managing Open vSwitch Across a Large Heterogeneous Fleet."

Similar presentations


Ads by Google