1. An Experimenter’s Guide to OpenFlow GENI Engineering Workshop June 2010 Rob Sherwood (with help from many others)

2. Talk Overview What is OpenFlow How OpenFlow Works OpenFlow for GENI Experimenters Deployments Next Session: OpenFlow “Office Hours” Overview of available software, hardware Getting started with NOX

3. Specialized Packet Forwarding Hardware Ap p Specialized Packet Forwarding Hardware Ap p Specialized Packet Forwarding Hardware Ap p Specialized Packet Forwarding Hardware Ap p Specialized Packet Forwarding Hardware Operating System Operating System Operating System Operating System Operating System Operating System Operating System Operating System Operating System Operating System Ap p 3 Current Internet Closed to Innovations in the Infrastructure Closed

4. Specialized Packet Forwarding Hardware Ap p Specialized Packet Forwarding Hardware Ap p Specialized Packet Forwarding Hardware Ap p Specialized Packet Forwarding Hardware Ap p Specialized Packet Forwarding Hardware Operating System Operating System Operating System Operating System Operating System Operating System Operating System Operating System Operating System Operating System Ap p Network Operating System App “Software Defined Networking” approach to open it

5. App Simple Packet Forwarding Hardware App Simple Packet Forwarding Hardware Network Operating System 1. Open interface to hardware 3. Well-defined open API 2. At least one good operating system Extensible, possibly open-source The “Software-defined Network”

6. Simple Packet Forwarding Hardware Network Operating System 1 Open interface to hardware Virtualization or “Slicing” Layer Network Operating System 2 Network Operating System 3 Network Operating System 4 Ap p Many operating systems, or Many versions Open interface to hardware Isolated “slices” Simple Packet Forwarding Hardware

7. Open Systems Performan ce Fidelity ScaleReal User Traffic? ComplexityOpen Simulationmedium nomediumyes Emulationmediumlownomediumyes Software Switches poorlowyesmediumyes NetFPGAhighlowyeshighyes Network Processors highmediumyeshighyes Vendor Switches high yeslowno gap in the tool space none have all the desired attributes!

8. What is OpenFlow?

9. Short Story: OpenFlow is an API Control how packets are forwarded Implementable on COTS hardware Make deployed networks programmable –not just configurable Makes innovation easier Goal (experimenter’s perspective): –No more special purpose test-beds –Validate your experiments on deployed hardware with real traffic at full line speed

10. OpenFlow: a pragmatic compromise + Speed, scale, fidelity of vendor hardware + Flexibility and control of software and simulation Vendors don’t need to expose implementation Leverages hardware inside most switches today (ACL tables)

11. How Does OpenFlow Work?

12. Ethernet Switch

13. Data Path (Hardware) Control Path Control Path (Software)

14. Data Path (Hardware) Control Path OpenFlow OpenFlow Controller OpenFlow Protocol (SSL/TCP)

15. Controller PC Hardware Layer Software Layer Flow Table MAC src MAC dst IP Src IP Dst TCP sport TCP dport Action OpenFlow Firmware **5.6.7.8***port 1 port 4port 3 port 2 port 1 1.2.3.45.6.7.8 OpenFlow Flow Table Abstraction

16. OpenFlow Basics Flow Table Entries Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport RuleActionStats 1.Forward packet to port(s) 2.Encapsulate and forward to controller 3.Drop packet 4.Send to normal processing pipeline 5.Modify Fields + mask what fields to match Packet + byte counters

17. Examples Switching * Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * 00:1f:.. *******port6 Flow Switching port3 Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action 00:20..00:1f..0800vlan11.2.3.45.6.7.841726480port6 Firewall * Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Forward ********22drop

18. Examples Routing * Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action *****5.6.7.8***port6 VLAN Switching * Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action ** vlan1 ***** port6, port7, port9 00:1f..

19. OpenFlowSwitch.org Controller OpenFlow Switch PC OpenFlow Usage Dedicated OpenFlow Network OpenFlow Switch OpenFlow Switch OpenFlow Protocol Aaron’s code RuleActionStatisticsRuleActionStatisticsRuleActionStatistics

20. OpenFlow Road Map OF v1.0 (current) – bandwidth slicing – match on Vlan PCP, IP ToS OF v1.1: Extensions for WAN, late 2010 –multiple tables: leverage additional tables –tags, tunnels, interface bonding OF v2+ : 2011? –generalized matching and actions: an “instruction set” for networking

21. What OpenFlow Can’t Do (1) Non-flow-based (per-packet) networking –ex: sample 1% of packets –yes, this is a fundamental limitation –BUT OpenFlow can provide the plumbing to connect these systems Use all tables on switch chips –yes, a major limitation (cross-product issue) –BUT an upcoming OF version will expose these

22. What OpenFlow Can’t Do (2) New forwarding primitives –BUT provides a nice way to integrate them New packet formats/field definitions –BUT plans to generalize in OpenFlow (2.0) Setup new flows quickly –~10ms delay in our deployment –BUT can push down flows proactively to avoid delays –Only a fundamental issue when delays are large or new flow-rate is high

23. OpenFlow for Experimenters Experiment Setup Design considerations OpenFlow GENI architecture Limitations

24. Why Use OpenFlow in GENI? Fine-grained flow-level forwarding control –e.g., between PL, ProtoGENI nodes –Not restricted to IP routes or Spanning tree Control real user traffic with Opt-In –Deploy network services to actual people Realistic validations –by definition: runs on real production network –performance, fan out, topologies

25. Experiment Setup Overview Step 1: Write/Configure/Deploy OpenFlow controller Step 2: Create Slice and register experiment Step 3: Control the traffic of Users that opt-in to Your experiment Each controller implements per-experiment custom forwarding logic Write your own or download pre-existing Configure per-experiment topology, queuing restricted to subset of real topology Specify desired user traffic: e.g., tcp.port=80 Users opt-in via the Opt-In Manager website Reserving a compute node makes the experimenter a user on the network

26. Experiment Design Decisions Forwarding logic (of course) Centralized vs. distributed control Fine vs. coarse grained rules Reactive vs. Proactive rule creation Likely more: open research area

27. Centralized vs Distributed Control Centralized Control OpenFlow Switch OpenFlow Switch OpenFlow Switch Controller Distributed Control OpenFlow Switch OpenFlow Switch OpenFlow Switch Controller

28. Flow Routing vs. Aggregation Both models are possible with OpenFlow Flow-Based Every flow is individually set up by controller Exact-match flow entries Flow table contains one entry per flow Good for fine grain control, e.g. campus networks Aggregated One flow entry covers large groups of flows Wildcard flow entries Flow table contains one entry per category of flows Good for large number of flows, e.g. backbone

29. Reactive vs. Proactive Both models are possible with OpenFlow Reactive First packet of flow triggers controller to insert flow entries Efficient use of flow table Every flow incurs small additional flow setup time If control connection lost, switch has limited utility Proactive Controller pre-populates flow table in switch Zero additional flow setup time Loss of control connection does not disrupt traffic Essentially requires aggregated (wildcard) rules

30. Examples of OpenFlow in Action VM migration across subnets energy-efficient data center network WAN aggregation network slicing default-off network scalable Ethernet scalable data center network load balancing formal model solver verification distributing FPGA processing Summary of demos in next session

32. Opt-In Manager User-facing website + List of experiments User’s login and opt-in to experiments –Use local existing auth, e.g., ldap –Can opt-in to multiple experiments subsets of traffic: Rob & port 80 == Rob’s port 80 –Use priorities to manage conflicts Only after opt-in does experimenter control any traffic

33. Deployments

34. OpenFlow Deployment at Stanford 34 Switches (23) APs (50) WiMax (1)

35. Live Stanford Deployment Statistics http://yuba.stanford.edu/ofhallway/wide-right.html http://yuba.stanford.edu/ofhallway/wide-left.html

36. GENI OpenFlow deployment (2010) 8 Universities and 2 National Research Backbones

37. Three EU Projects similar to GENI: Ophelia, SPARC, CHANGE 37  L2 Packet  Wireless  Routing Pan-European experimental facility  L2 Packet  Optics  Content delivery  L2 Packet  Shadow networks  L2 L3Packet  Optics  Content delivery  L2 Packet  Emulation  Wireless  Content delivery

38. Other OpenFlow deployments Japan - 3-4 Universities interconnected by JGN2plus Interest in Korea, China, Canada, …

39. KOREA OpenFlow Network Seoul Daejeon Deagu Busan Gwangju Suwon Controller VLAN on KOREN OpenFlow Switch (Linux PC) NOX OpenFlow Controller TJB TJB Broadcasting Company Japan OpenFlow Network Sapporo Studio Asahi Broadcasting Cooperation (ABC) at Osaka, Japan Sapporo Japan Server Data Transmission An Experiment of OpenFlow-enabled Network (Feb. 2009 - Sapporo Snow Festival Video Transmission) A video clip of Sapporo snow festival is transmitted to TJB (Daejeon, KOREA) via ABC server (Osaka, JAPAN).

40. Highlights of Deployments Stanford deployment –McKeown group for a year: production and experiments –To scale later this year to entire building (~500 users) Nation-wide trials and deployments –7 other universities and BBN deploying now –GEC9 in Nov, 2010 will showcase nation-wide OF –Internet 2 and NLR to deploy before GEC9 Global trials –Over 60 organizations experimenting 2010 likely to be a big year for OpenFlow

41. Slide Credits Guido Appenzeller Nick McKeown Guru Parulkar Brandon Heller Lots of others –(this slide was also stolen)

42. Conclusion OpenFlow is an API for controlling packet forwarding OpenFlow+GENI allows more realistic evaluation of network experiments Glossed over many technical details –What does the API look like? Stay for the next session

43. An Experimenter’s Guide to OpenFlow: Office Hours GENI Engineering Workshop June 2010 Rob Sherwood (with help from many others)

44. Office Hours Overview Controllers Tools Slicing OpenFlow OpenFlow switches Demo survey Ask questions!

45. Controllers

46. Controller is King Principle job of experimenter: customize a controller for your OpenFlow experiment Many ways to do this: –Download, configure existing controller e.g., if you just need shortest path –Read raw OpenFlow spec: write your own handle ~20 OpenFlow messages –Recommended: extend existing controller Write a module for NOX – www.noxrepo.org

47. Starting with NOX Grab and build –`git clone git://noxrepo.org/nox` –`git checkout -b openflow-1.0 origin/openflow-1.0` –`sh boot.sh;./configure; make` Build nox first: non-trivial dependencies API is documented inline –`cd doc/doxygen; make html` – Still very UTSL

48. Writing a NOX Module Modules live in./src/nox/{core,net,web}apps/* Modules are event based –Register listeners using APIs –C++ and Python bindings –Dynamic dependencies e.g., many modules (transitively) use discovery.py Currently have to update build manually –Automated with./src/scripts/nox-new-c-app.py Most up to date docs are at noxrepo.org

49. Useful NOX Events Datapath_{join,leave} –New switch and switch leaving Packet_in/Flow_in –New Datagram, stream; respectively –Cue to insert a new rule/flow_mod Flow_removed –Expired rule (includes stats) Shutdown –Tear down module; clean up state

50. Tools OpenFlow Wireshark plugin MiniNet oftrace many more…

51. OpenFlow WireShark Plugin Ships with OpenFlow reference controller

52. MiniNet Machine-local virtual network –great dev/testing tool Uses linux virtual network features –Cheaper than VMs Arbitrary topologies, nodes Scriptable –Plans to move FV testing to MiniNet http://www.openflow.org/foswiki/bin/view/OpenFlow/Mininet

53. OFtrace API for analyzing OF Control traffic Calculate: – OF Message distribution – Flow Setup time – % of dropped LLDP messages – … extensible http://www.openflow.org/wk/index.php/Liboftrace

54. Slicing OpenFlow Vlan vs. FlowVisor slicing Use cases

55. Switch Based Virtualization Exists for NEC, HP switches but not flexible enough for GENI Normal L2/L3 Processing Flow Table Production VLANs Research VLAN 1 Controller Research VLAN 2 Flow Table Controller

56. OpenFlow Switch OpenFlow Protocol OpenFlow Protocol OpenFlow FlowVisor & Policy Control Craig’s Controller Heidi’s Controller Aaron’s Controller OpenFlow Protocol OpenFlow Protocol FLOWVISOR BASED VIRTUALIZATION OpenFlow Switch OpenFlow Switch

57. –The individual controllers and the FlowVisor are applications on commodity PCs (not shown) Stanford Infrastructure Uses Both Flows OpenFlow switches WiMax Packet processors WiFi APs

58. Use Case: VLAN Based Partitioning Basic Idea: Partition Flows based on Ports and VLAN Tags –Traffic entering system (e.g. from end hosts) is tagged –VLAN tags consistent throughout substrate Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport ****1,2,3***** ****7,8,9***** ****4,5,6*****

59. OpenFlow Protocol OpenFlow FlowVisor & Policy Control Broadcast Multicast OpenFlow Protocol http Load-balancer FLOWVISOR BASED VIRTUALIZATION Separation not only by VLANs, but any L1-L4 pattern OpenFlow Switch OpenFlow Switch OpenFlow Switch

60. Use Case: New CDN - Turbo Coral ++ Basic Idea: Build a CDN where you control the entire network –All traffic to or from Coral IP space controlled by Experimenter –All other traffic controlled by default routing –Topology is entire network –End hosts are automatically added (no opt-in) Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport ***** 84.65.* **** ****** *** **********

61. Use Case: Aaron’s IP –A new layer 3 protocol –Replaces IP –Defined by a new Ether Type Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport ***AaIP* * **** ***!AaIP******

62. Switches

63. Linux based Software Switch Release concurrently with specification Kernel and User Space implementations Note: no v1.0 kernel-space implementation Limited by host PC, typically 4x 1Gb/s Not targeted for real-world deployments Useful for development, testing Starting point for other implementations Available under the OpenFlow License (BSD Style) at http://www.openflowswitch.org Stanford Reference Implementation

64. Wireless Access Points Two Flavors: –OpenWRT based (Busybox Linux) v0.8.9 only –Vanilla Software (Full Linux) Only runs on PC Engines Hardware Debian disk image Available from Stanford Both implementations are software only.

65. NetFPGA NetFPGA-based implementation –Requires PC and NetFPGA card –Hardware accelerated –4 x 1 Gb/s throughput Maintained by Stanford University $500 for academics $1000 for industry Available at http://www.netfpga.org

66. Linux-based Software Switch Released after specification (v1.0 support 1 week old!) Not just an OpenFlow switch; also supports VLAN trunks, GRE tunnels, etc Kernel and User Space implementations Limited by host PC, typically 4x 1Gb/s Available under the Apache License (BSD Style) at http://www.openvswitch.org Open vSwitch

67. OpenFlow Vendor Hardware more to follow... NEC IP8800 HP ProCurve 5400 and others Juniper MX-series (prototype) Cisco Catalyst 6k (prototype) Core Router Enterprise Campus Data Center Circuit Switch Wireless Pronto Prototype Product Ciena CoreDirector WiMAX (NEC) Cisco Catalyst 3750 (prototype) Arista 7100 series (Q4 2010) 67

68. HP ProCurve 5400 Series (+ others) Praveen Yalagandula Jean Tourrilhes Sujata Banerjee Rick McGeer Charles Clark Chassis switch with up to 288 ports of 1G or 48x10G (+ other interfaces available) Line-rate support for OpenFlow Deployed in 23 wiring closets at Stanford Limited availability for Campus Trials Contact HP for support details

69. NEC IP8800 24x/48x 1GE + 2x 10 GE Line-rate support for OpenFlow Deployed at Stanford Available for Campus Trials Supported as a product Contact NEC for details: Don Clark (don.clark@necam.com)don.clark@necam.com Atsushi Iwata (a-iwata@ah.jp.nec.com)a-iwata@ah.jp.nec.com Hideyuki Shimonishi Jun Suzuki Masanori Takashima Nobuyuki Enomoto Philavong Minaxay Shuichi Saito Tatsuya Yabe Yoshihiko Kanaumi (NEC/NICT) Atsushi Iwata (NEC/NICT)

70. Pronto Switch Broadcom based 48x1Gb/s + 4x10Gb/s Bare switch – you add the software Supports Stanford Indigo and Toroki releases See openflowswitch.org blog post for more details

71. Stanford Indigo Firmware for Pronto Source available under OpenFlow License to parties that have NDA with BRCM in place Targeted for research use and as a baseline for vendor implementations (but not direct deployment) No standard Ethernet switching – OpenFlow only! Hardware accelerated Supports v1.0 Contact Dan Talayco (dtalayco@stanford.edu)

72. Toroki Firmware for Pronto Fastpath-based OpenFlow Implementation Full L2/L3 management capabilities on switch Hardware accelerated Availability TBD

73. Ciena CoreDirector Circuit switch with experimental OpenFlow support Prototype only Demonstrated at Super Computing 2009

74. Umesh Krishnaswamy Michaela Mezo Parag Bajaria James Kelly Bobby Vandalore Juniper MX Series Up to 24-ports 10GE or 240-ports 1GE OpenFlow added via Junos SDK Hardware forwarding Deployed in Internet2 in NY and at Stanford Prototype Availability TBD

75. Flavio Bonomi Sailesh Kumar Pere Monclus Various configurations available Software forwarding only Limited deployment as part of demos Availability TBD Work on other Cisco models in progress Cisco 6500 Series

76. Comes with reference distribution Monolithic C code – not designed for extensibility Ethernet flow switch or hub Stanford Reference Controller

77. Available at http://NOXrepo.orghttp://NOXrepo.org Open Source (GPL) Modular design, programmable in C++ or Python High-performance (usually switches are the limit) Deployed as main controller in Stanford NOX Controller Martin Casado Scott Shenker Teemu Koponen Natasha Gude Justin Pettit

78. Available at http://NOXrepo.orghttp://NOXrepo.org Policy + Nice GUI Branched from NOX long ago Available as a binary Part of Stanford deployment Simple Network Access Control (SNAC)

79. Demo Previews FlowVisor Plug-n-Serve Aggregation OpenPipes OpenFlow Wireless MobileVMs ElasticTree

80. –The individual controllers and the FlowVisor are applications on commodity PCs (not shown) Demo Infrastructure with Slicing Flows OpenFlow switches WiMax Packet processors WiFi APs Be sure to check out the demos during the break!!

81. OpenFlow Demonstration Overview Network Virtualization FlowVisor Hardware Prototyping OpenPipes Load Balancing PlugNServe Energy Savings ElasticTree Mobility MobileVMs Traffic Engineering Aggregation Wireless Video OpenRoads TopicDemo

82. FlowVisor Creates Virtual Networks OpenFlow Switch OpenFlow Switch OpenFlow Switch OpenFlow Protocol FlowVisor OpenPipes Demo OpenRoads Demo OpenFlow Protocol PlugNServe Load-balancer OpenPipes Policy FlowVisor slices OpenFlow networks, creating multiple isolated and programmable logical networks on the same physical topology. Each demo presented here runs in an isolated slice of Stanford’s production network.

83. Plumbing with OpenFlow to build hardware systems OpenPipes Partition hardware designs Test Mix resources

84. Goal: Load-balancing requests in unstructured networks Plug-n-Serve : Load-Balancing Web Traffic using OpenFlow OpenFlow means…  Complete control over traffic within the network  Visibility into network conditions  Ability to use existing commodity hardware What we are showing  OpenFlow-based distributed load-balancer  Smart load-balancing based on network and server load  Allows incremental deployment of additional resources This demo runs on top of the FlowVisor, sharing the same physical network with other experiments and production traffic.

85. Dynamic Flow Aggregation on an OpenFlow Network Scope Different Networks want different flow granularity (ISP, Backbone,…) Switch resources are limited (flow entries, memory) Network management is hard Current Solutions : MPLS, IP aggregation How OpenFlow Helps? Dynamically define flow granularity by wildcarding arbitrary header fields Granularity is on the switch flow entries, no packet rewrite or encapsulation Create meaningful bundles and manage them using your own software (reroute, monitor) Higher Flexibility, Better Control, Easier Management, Experimentation

87. Intercontinental VM Migration Moved a VM from Stanford to Japan without changing its IP. VM hosted a video game server with active network connections.

88. ElasticTree: Reducing Energy in Data Center Networks The demo: Hardware-based 16- node Fat Tree Your choice of traffic pattern, bandwidth, optimization strategy Graph shows live power and latency variation Shuts off links and switches to reduce data center power Choice of optimizers to balance power, fault tolerance, and BW OpenFlow provides network routes and port statistics demo credits: Brandon Heller, Srini Seetharaman, Yiannis Yiakoumis, David Underhill