Presentation on theme: "ONOS Open Network Operating System"— Presentation transcript:
1 ONOS Open Network Operating System An Experimental Open-Source Distributed SDN OSIntroduction:Experimental, Open, Distributed----- Meeting Notes (7/29/13 12:57) -----Read the slideThis is just a glimpsePankaj Berde, Umesh Krishnaswamy, Jonathan Hart, Masayoshi Kobayashi, Pavlin Radoslavov, Pingping Lin, Sean Corcoran, Tim Lindberg, Rachel Sverdlov, Suibin Zhang, William Snow, Guru Parulkar
2 Agenda Overview of ONRC (Open Networking Research Center) ONOS ArchitectureScale-out and high availabilityNetwork graph as north-bound abstractionDEMOConsistency modelsDevelopment and test environmentPerformanceNext steps
3 LeadershipNick MckeownKP, Mayfield, Sequoia Professor, StanfordLarry PetersonBob Kahn Professor PrincetonChief Architect, ON.LABScott ShenkerProfessor, UC BerkeleyChief Scientist, ICSINational Academy of Engineering ACM SIGCOMM Award Winners Fellow of IEEE and ACM Entrepreneurs Impact on practice of networking/cloudLet me begin at the top…We are really fortunate to have a leadership of three very accomplished gentlemen – they provide the vision and direction and that defines who we are and what we do.I thought I would simply list their common accomplishments and recognitions…Each one of them iso o oMoreover they have been also successful entrepreneurs and have had a history of impacting real practice of networking….Now you may also wonder why they came together for this venture?
4 Stanford/Berkeley SDN Activities With Partners VM Migration(Best Demo)Trans-PacificBaby GENINation Wide GENI“The OpenFlow Show”– IT WorldSDN ConceptSIGCOMM08GEC3SIGCOMM09GEC6GEC9Interop2011DemoOther countriesOver 68 countries(Europe, Japan, China, Korea,Brazil, etc.)DeploymentUS R&E CommunityGENI: 8 Universities + Internet2 + NLRMany other campusesStanford University~45 switch/APs ~25userIn McKeown GroupCIS/EE BuildingProduction NetworkOpenFlow Specv0.8.9v1.0v1.1Reference SwitchNetFPGASoftwareNetwork OSNOXSNACBeaconVirtualizationFlowVisorFlowVisor (Java)ToolsTest SuiteoftraceMininetMeasurement toolsGENI software suiteExpedient/Opt-in Manager/FOAMDevelopmentPlatformEthane+Broadcom20072008200920102011
5 Scaling of SDN Innovation Build strong intellectual foundationBring open source SDN tools/platforms to communityStandardize OpenFlow and promote SDN~100 Members from all parts of the industryBring best SDN content; facilitate high quality dialogue3 successive sold out events; participation of ecosysSDN AcademyBring best SDN training to companiesto accelerate SDN development and adoption
6 ONRC Organizational Structure BerkeleyScott ShenkerSylvia RatnasamyStanfordNick McKeownGuru ParulkarSachin KattiOpen Network LabExec Director: Guru ParulkarVP Eng: Bill SnowChief Architect: Larry Peterson16-19 Engineers/Tech Leads(includes PlanetLab team)Tools/Platforms for SDN communityOpenCloud demonstration of XaaS and SDNPhD/PostdocsResearchTo achieve our mission we came up with the following structure.It has three parts:Research at BerkeleyResearch at StanfordAnd an independent nonprofit Open Networking Lab.We have formalized our collaboration between Berkeley and Stanford through this center. This center gives us more reasons to get together with our students and promote more interactions. And that is great.I want to mention our collaborator Sachin Katti who is a professor of EE and CS has been working on OpenRadio and SDN based mobile wireless networks.The ON.Lab is kind of unique to our center.We created ON.Lab as an independent lab with the goal to develop, support, and deploy open source SDN tools and platforms for the benefit of the community.And create SDN deployments and demonstrations for R&E networks.
7 MissionBring innovation and openness to internet and cloud infrastructure with open source tools and platforms
8 TestON with debugging support MININET, Cluster Edition Tools & Platforms3rd partycomponentsSDN-IP PeeringAppsAppsAppsAppsOpen InterfacesONOSNetwork OSNetwork OSNetwork HypervisorFlowVisor, OpenVirteXNetSightTestON with debugging supportOpen InterfacesMININET, Cluster EditionForwarding
9 Open Network OS (ONOS) Architecture Scale-out and high availability Network graph as north-bound abstractionDEMOConsistency modelsDevelopment and test environmentPerformanceNext steps
10 ONOS: Executive Summary Distributed Network OSNetwork Graph Northbound AbstractionHorizontally ScalableHighly AvailableBuilt using open source componentsONOSVersion 0.1- Flow API, Shortest Path computation, Sample application- Build & QA ( Jenkins, Sanity Tests, Perf/Scale Tests, CHO)- Deployment in progress at REANNZ (SDN-IP peering)StatusNext is sharing our experience with community. Address the non-goals we discussed earlier and evaluate more use cases and applications on this experimental Open Distributed NOS.Want to join us in this experiment? Lookout for more information on our website.Exploring performance & reactive computation frameworksExpand graph abstraction for more types of network stateControl functions: intra-domain & inter-domain routingExample use cases: traffic engineering, dynamic virtual networks on demand, …Next
12 Open Network OS Focus (Started in Summer 2012) Global network viewRoutingTEMobilityGlobal Network ViewNetwork OSScale-outDesignOpenflowPacketForwardingTwo of them have to do with logical centralization of the control plane and design of Network OS.Number 1, will the Network OS become a performance bottleneck? Or can we scale the Network OS horizontally as we need more horse power?Number 2, will the Network OS become a single point of failure? Or can we make the Network OS and the control plane fault tolerant?The third question has to do with Northbound API. What is the best abstraction the Network OS can offer to application writers that enables reusable and pluggable network control and management applications?ONOS attempts to address exactly these issues…Fault TolerancePacketForwardingProgrammableBase StationPacketForwarding
13 Community needs an open source distributed SDN OS Prior WorkDistributed control platform for large-scale networksFocus on reliability, scalability, and generalityScale-out NOS focused on network virtualization in data centersState distribution primitives, global network view, ONIX APIONIXOther WorkHelios (NEC), Midonet (Midokura), Hyperflow, Maestro, KandooNOX, POX, Beacon, Floodlight, Trema controllersONIX did attempt to solve these issues. There are few more efforts. To enable more research in this area community needs an open distributed NOS.----- Meeting Notes (7/29/13 12:57) -----ONIX is scale out NOS focused on Network Virtualization in data center context and is closed source.Community needs an open source distributed SDN OS
14 ONOS High Level Architecture Network Graph Eventually consistentTitan Graph DBCassandra In-Memory DHTDistributed Registry Strongly ConsistentZookeeperInstance 1Instance 2Instance 3OpenFlow Controller+OpenFlow Controller+OpenFlow Controller+Built on two distributed data constructs1> Network Graph which is the global network view containing the network state represented as a graph which is eventually consistent2> Distributed Registry is the global cluster management state stored in Zookeeper using transactional consistency.Multiple instances of ONOS control different parts of network and help realize a single global network view by cooperatively using these two distributed data constructs.----- Meeting Notes (5/15/13 14:21) -----Distribruted Registry keeps information on who is in control of the switch objects and has write permissions to update the network graph. In general it stores the resource ownership in a strongly consistent way.----- Meeting Notes (7/29/13 12:57) -----order animationremove floodlightHost+Floodlight Drivers
16 ONOS Scale-Out Network Graph Distributed Network OS Instance 1 Global network viewDistributed Network OSInstance 1Instance 2Instance 3A part of network is solely controlled by a single ONOS instance and the same instance is also solely responsible for maintaining the state of the partition into the network graph.[We also refer this as Control isolation.]This enables simpler scale-out design.As the network grows beyond the control capacity one can add another instance which will be responsible for a new part of network . As this part is realized into Network Graph, applications will get a global network view.----- Meeting Notes (7/29/13 12:57) -----Fix animationData planeAn instance is responsible for maintaining a part of network graphControl capacity can grow with network size or application need
17 ONOS Control Plane Failover MasterSwitch A = ONOS 2Candidates = ONOS 3MasterSwitch A = ONOS 1Candidates = ONOS 2, ONOS 3Candidates = ONOS 2, ONOS 3MasterSwitch A = NONECandidates = ONOS 2, ONOS 3Candidates = ONOS 2, ONOS 3Distributed RegistryDistributed Network OSInstance 1Instance 2Instance 3Switch A is being controlled by Instance 1 and the registry shows it as master for switch A.Instance 1 has a failure and dies.Registry detects that instance 1 is down and release the mastership for Switch A.Remaining candidates join the mastership election within registry. Lets say Instance 2 wins the election and is marked in registry as the master for Switch A.The channel with Instance 2 becomes the active channel and other channel becomes passive.This enables a quick failover of switch when there is a control plane failure.----- Meeting Notes (7/29/13 12:57) -----Mention strong consistency and elegent coordinationHostABCDEF
19 ONOS Network Graph Abstraction Id: 1AId: 101, LabelId: 103, LabelId: 2CId: 3BId: 102, LabelId: 104, LabelId: 106, LabelId: 105, LabelTitan Graph DBNetwork graph is organized as a graph database. Vertices as network objects and connected by edges as relation between the vertices.We use Titan as graph DB with Cassandra as its backend.Cassandra is eventually consistentCassandra In-memory DHT
20 Network Graph Network state is naturally represented as a graph portswitchdeviceonlinkhostNetwork is naturally a graph with switches, ports, devices as objects as vertices. Similarly links and attachment points are modeled as edges.Applications can traverse and write to this graph to program the data plane.How? Lets look at this example applicationNetwork state is naturally represented as a graphGraph has basic network objects like switch, port, device and linksApplication writes to this graph & programs the data plane
21 Example: Path Computation App on Network Graph portswitchdeviceFlow pathFlow entryonlinkinportoutporthostflowPath Computation is an application which is using Network Graph.The application can find a path from source to destination by traversing links and program this path with flow entries to create a flow-path.These flow-entries are translated by ONOS core into flow table rules and pushed onto the topology.Last bullet: Application is made simple and stateless. It does not need to worry about topology maintenance.----- Meeting Notes (5/14/13 14:16) -----start without text. Bring in text at end and make one pointApplication computes path by traversing the links from source to destinationApplication writes each flow entry for the pathThus path computation app does not need to worry about topology maintenance
22 Example: A simpler abstraction on network graph? Virtual network objectsLogical CrossbarportswitchdeviceEdge PortonlinkphysicalhostReal network objectsNetwork graph simplifies applications but can it be used to accelerate innovations of simpler abstractions in control plane?Here is an example of Logical Crossbar. The complexity of network state and topology is hidden.One can build hierarchy of these abstractions further hiding the complexity.Last bullet: We feel network graph will unlock innovations.7 minute MarkerApp or service on top of ONOSMaintains mapping from simpler to complexThus makes applications even simpler and enables new abstractions
23 Network Graph Representation Flow entryFlow pathflowVertex with10 propertiesSwitchVertex with3 propertiesflowVertex with11 propertiesFlow entryProperty(e.g. dpid)(e.g. state)…EdgeVertex represented as Cassandra rowRow indices for fast vertex centric queriesColumnValueLabel id + directionPrimary keyEdge idVertex idSignature propertiesOther propertiesEdge represented as Cassandra column
24 Network Graph and Switches Network Graph: SwitchesSwitch ManagerSwitch ManagerSwitch ManagerLet us see how ONOS builds the network graph. Each ONOS node has a switch manager. When switches connect, switches and ports are get added as switches register with an ONOS node. When switches disconnect, they get marked as inactive in the network graph.OFOFOF
25 Network Graph and Link Discovery Network Graph: LinksLink DiscoveryLink DiscoveryLink DiscoverySMSMSMLLDPLLDPEach node sends out LLDP on the switches connected to it. Links with source and destination port controlled by different ONOS nodes can also be discovered using the network graph.
26 Devices and Network Graph Network Graph: DevicesDevice ManagerDevice ManagerDevice ManagerSMLDSMLDSMLDHost packet Ins are used to learn about devices, their attachment points. The network graph is updated with this information.PKTINHostPKTINPKTIN
27 Path Computation with Network Graph Network Graph: Flow PathsFlow 1Flow entriesFlow 2Flow entriesFlow 3Flow entriesFlow 4Flow entriesFlow 5Flow entriesFlow 6Flow entriesFlow 7Flow 8Flow entriesFlow entriesSMLDDMSMLDDMSMLDDMFlow paths are provisioned in ONOS.The source dpid of a flow is used to partition which node will compute the path. Computed paths and flow entries are also stored in the network graph.Flow entries have relationship to the switches.Host
28 Network Graph and Flow Manager PCPCPCNetwork Graph: FlowsFlow 1Flow entriesFlow 2Flow entriesFlow 3Flow entriesFlow 4Flow entriesFlow 5Flow entriesFlow 6Flow entriesFlow 7Flow 8Flow entriesFlow entriesFlow ManagerFlow ManagerFlow ManagerFlowmodFlowmodEach flow manager programs the switches connected to it using the state in the network graph.When a link fails, PC will recompute a new path and Flow Manager will push new flow entries.FlowmodSMLDDMSMLDDMSMLDDMHost
29 ONOS High Level Architecture Network Graph Eventually consistentTitan Graph DBCassandra In-Memory DHTDistributed Registry Strongly ConsistentZookeeperInstance 1Instance 2Instance 3OpenFlow Controller+OpenFlow Controller+OpenFlow Controller+Built on two distributed data constructs1> Network Graph which is the global network view containing the network state represented as a graph which is eventually consistent2> Distributed Registry is the global cluster management state stored in Zookeeper using transactional consistency.Multiple instances of ONOS control different parts of network and help realize a single global network view by cooperatively using these two distributed data constructs.----- Meeting Notes (5/15/13 14:21) -----Distribruted Registry keeps information on who is in control of the switch objects and has write permissions to update the network graph. In general it stores the resource ownership in a strongly consistent way.----- Meeting Notes (7/29/13 12:57) -----order animationremove floodlightHost+Floodlight Drivers
32 Consistency Definition Strong Consistency: Upon an update to the network state by an instance, all subsequent reads by any instance returns the last updated value.Strong consistency adds complexity and latency to distributed data management.Eventual consistency is slight relaxation – allowing readers to be behind for a short period of time.Consistency is not a binary value but a fraction between 0 and 1 excluding both extremes. So lets see what is our consistency definition.Strong consistency: Every update is instantly available on other instances of ONOS.Next bullet: It cannot be instant in reality, there is always some cost/delay involved.Our eventual consistency is much closer to 1 than many other eventual consistency definitions
33 Strong Consistency using Registry Network GraphDistributed Network OSInstance 1Instance 2Instance 3Switch AMaster = ONOS 1Switch AMaster = NONEA =Switch AMaster = ONOS 1Switch AMaster = NONESwitch AMaster = ONOS 1Switch AMaster = NONEA = ONOS 1RegistryWe use Transactional Consistency for Master election.Lets look at Registry: We see Switch has NO Master when we start.Lets see the point in time where Master is elected and ONOS 1 is marked as master. After the Master is elected and mastership is consistently propagated on all instances.All subsequent reads for master on any instance shows Master as ONOS 1 for Switch A. This guarantees that we have control isolation.All instancesSwitch A Master = NONEInstance 1 Switch A Master = ONOS 1Instance 2 Switch A Master = ONOS 1Instance 3 Switch A Master = ONOS 1All instancesSwitch A Master = NONETimelineMaster elected for switch ADelay of Locking & Consensus
34 Why Strong Consistency is needed for Master Election Weaker consistency might mean Master election on instance 1 will not be available on other instances.That can lead to having multiple masters for a switch.Multiple Masters will break our semantic of control isolation.Strong locking semantic is needed for Master ElectionLocking and latches are distributed primitives which need consistency.
35 Eventual Consistency in Network Graph Distributed Network OSInstance 1Instance 2Instance 3Switch AState = ACTIVESWITCH ASTATE= INACTIVESwitch AState = ACTIVESwitch AState = INACTIVESwitch ASTATE = INACTIVESwitch ASTATE = ACTIVEDHTLets take a concrete example.Before switch is connected: it is marked INACTIVE on our Network Graph.As soon as the switch connects to ONOS 1 it is marked ACTIVE immediately on INSTANCE 1.For a fraction of second, instance 2 and instance 3 might still show the switch as INACTIVE till they are eventually consistent (white line)From that point the switch state is consistent on All instances.TimelineAll instancesSwitch A STATE = INACTIVEInstance 1 Switch A = ACTIVEInstance 2 Switch A = INACTIVEInstance 3 Switch A = INACTIVEAll instancesSwitch A STATE = ACTIVESwitch Connected to ONOSDelay of Eventual Consensus
36 Cost of Eventual Consistency Short delay will mean the switch A state is not ACTIVE on some ONOS instances in previous example.Applications on one instance will compute flow through the switch A while other instances will not use the switch A for path computation.Eventual consistency becomes more visible during control plane network congestion.What does it mean? Switch is not available instantly for path computation on some instances. Some flows can be computed without using switch A.Similarly when flow entries are written to Network graph they become eventually available on all instances and each instance picks its flow entries and pushes onto switches.
37 Why is Eventual Consistency good enough for Network State? Physical network state changes asynchronouslyStrong consistency across data and control plane is too hardControl apps know how to deal with eventual consistencyIn the current distributed control plane, each router makes its own decision based on old info from other parts of the network and it works fineStrong Consistency is more likely to lead to inaccuracy of network state as network congestions are real.Asynchronously changing state is not transactional and does not need strong consistency. Control apps know that they are always working on a stale state and know to deal with it.Today’s routers work in the same fashion.Network state is most often used by current gen control applications with assumed eventual consistency.Time check: 13 minutes
38 Consistency learning One Consistency does not fit all Consequences of delays need to be well understoodMore research needs to be done on various states using different consistency models
40 ONOS Development & Test cycle Source code on githubAgile: 3-4 week sprintsMostly Java and many utility scriptsCI: Maven, Jenkins, JUnit, Coverage, TestONVagrant-based development VMDaily 4 hour of Continuous Hours of Operations (CHO) tests as part of buildSeveral CHO cycles simulating rapid churns in network & failures on ONOS instances
41 ONOS Development Environment Single installation script creates a cluster of Virtual Box VMs
43 ON.LAB ONOS Test implementation ON.LAB team has implemented following aspects of automated testsONOS Unit Tests (70% coverage)ONOS System Tests for Functionality, Scale, Performance and Resiliency test (85% coverage)White Box Network Graph Performance MeasurementsAll tests are executed nightly in Jenkins Continuous Integration environment.
45 Key performance metrics in Network OS Network scale (# switches, # ports) -> Delay and ThroughputLink failure, switch failure, switch port failurePacket_in (request for setting reactive flows)Reading and searching network graphNetwork Graph TraversalsSetup of proactive flowsApplication scale (# operations, # applications)Number of network events propagated to applications (delay & throughput)Number of operations on Network Graph (delay & throughput)Parallelism/threading for applications (parallelism on Network Graph)Parallel path computation performance
46 Performance: Hard Problems Off the shelf open source does not performUltra low-latency requirements are uniqueNeed to apply distributed/parallel programming techniques to scale control applicationsReactive control applications need event-driven framework which scale
47 ONOS: Summary ONOS Status Next Distributed Network OS Version 0.1 Network Graph Northbound AbstractionHorizontally ScalableHighly AvailableBuilt using open source componentsONOSVersion 0.1- Flow API, Shortest Path computation, Sample application- Build & QA ( Jenkins, Sanity Tests, Perf/Scale Tests, CHO)- Deployment in progress at REANNZ (SDN-IP peering)StatusNext is sharing our experience with community. Address the non-goals we discussed earlier and evaluate more use cases and applications on this experimental Open Distributed NOS.Want to join us in this experiment? Lookout for more information on our website.Exploring performance & reactive computation frameworksExpand graph abstraction for more types of network stateControl functions: intra-domain & inter-domain routingExample use cases: traffic engineering, dynamic virtual networks on demand, …Next