2Negative interactions OutlineMotivationProblems with the underlying routing systemSource routing, overlay networks, and hybridsOverlay networksPros: flexibility, limited overhead, & value-addedCons: data-path overhead, probes, & feedbackNegative interactionsWith other overlays: the price of anarchyWith the underlay: influence on traffic engineeringWith itself: bi-stability and trunk reservationFuture directions
3What’s Wrong With Internet Routing? Restrictive path-selection modelDestination-based packet forwardingSingle best BGP path per prefixBGP routing constrained by policiesIgnoring congestion and delayIgnoring application requirementsUnappealing protocol dynamicsPersistent oscillation (due to policy conflicts)Slow convergence (due to path exploration)Lost reachability (due to route-flap damping)Stems from the need for routing to scale to millions of routers
4Putting More Power in End Hosts Source routing (e.g., Nimrod)End host selects the end-to-end pathRouters simply forward packets on the pathRequires the routers to agree to participateOverlay networks (e.g., RON)Conventional computers act as logical routersReal routers deliver packets to intermediate hostsNo need for cooperation from the real routersHybrid schemesSource routing at the AS levelSource routing in the overlay network
6Advantage: Flexible Routing Paths that violate BGP routing policyE.g., A to C goes through AT&T and Sprint… and C to B goes through UUNETBGP would not allow AT&T-Sprint-UUNET pathQuick adaptation to network problemsFast detection of congestion and outages… by probing as aggressively as necessarySelecting paths based on different metricsE.g., overlay selects paths based on latency… whereas the underlay might try to balance load
7Advantage: Fewer Worries About Scalability Small number of nodesJust enough nodes to have diverse pathsA few friends who want better serviceVirtual Private Network of several corporate sitesBalancing the trade-offsHigh probe frequency for maximum adaptivityLow probe frequency for minimum overheadSimple routing protocolLink-state protocol to learn probing resultsSelecting a good intermediate hop when neededDeploy multiple small overlay networks, if necessary
8Advantage: Customizing Packet Delivery Recovering from packet lossPacket retransmissionForward error correctionQuality-of-service differentiationClassify packets based on header bitsSchedule packet transmissions based on resultIncremental deployment of new featuresMulticast communication (e.g., MBone)IPv6 (e.g., 6Bone)Encryption of packet contents
9Disadvantage: Traversing Intermediate Nodes Processing delayPackets going through multiple software nodesNetwork performancePropagation delay on circuitous pathNetwork congestion from extra loadFinancial costBill for traffic going in/out of intermediate nodeABC
10Disadvantage: Limitations of Active Probes Bandwidth overheadProbe traffic between two nodesPropagating probe results to other nodesLimited accuracy of end-to-end probesAvailable bandwidth of logical link?Losses due to congestion vs. failure?Problem on forward vs. reverse path?Limited visibilityLogical links may share underlay routers/linksMay be hard to detect the dependencies
11Disadvantage: Feedback Effects Background trafficOverlay traffic consumes extra resources… at the expense of regular background trafficBut, the overlay traffic does get out of the way!Other overlaysPotential competition between multiple overlaysE.g., one overlay picks a (longer) alternate path… and extra load causes another overlay to adaptUnderlying networkOverlay network changes the traffic matrix… forcing operators to adapt the underlay routingAre these effects significant? Any positive effects?
12Price of Anarchy (Roughgarden & Tardos) Worst-case exampleTwo paths from s to dTotal of one unit of loadLatency as function of loadSelfish source routingAll traffic through bottom linkMean latency of 1Latency-optimal routingMinimize mean latencySet x = [1/(n+1)]1/nMean latency goes to 0SDL0(x) = xnL1(y) = 1Total load: x + y = 1Mean latency: x L0(x) + y L1(y)
13Internet-Like Environments (Qiu et al) Realistic networksBackbone network topologiesLink delay (propagation and queuing delay)Routing set to minimize network congestionRealistic overlaysSmall number of overlay nodes (limited flexibility)Overlay paths chosen to minimize latencyPractice doesn’t match the worst-case theorySome tension between the two different metricsBut, not anywhere near as bad as the worst case
14Interaction With Traffic Engineering (Qiu et al) Underlay network: traffic engineeringInputs: traffic matrix Mij and physical topologyObjective: minimize overall network congestionOutput: routing Rijl: fraction of (i,j) traffic on link lpropagation and queuing delay on virtual links21i3132315j43
15Interaction With Traffic Engineering (Qiu et al) Overlay network: selecting intermediate nodesInputs: measured delay for each virtual linkObjective: minimizing end-to-end latencyOutput: choice of intermediate nodes for traffictraffic matrix on the underlay networkkij
16Interaction With Traffic Engineering (Qiu et al) Underlay network: traffic engineeringInputs: traffic matrix and physical topologyObjective: minimize overall network congestionOutput: selection of paths in underlay networkpropagation and queuing delay on virtual linksOverlay network: selecting intermediate nodesInputs: measured delay for each virtual linkObjective: minimizing end-to-end latencyOutput: choice of intermediate nodes for traffictraffic matrix on the underlay network
17Interaction With TE: OSPF Weight Tweaking OSPF optimizer interacts poorly with selfish overlays because it only has very coarse-grained control.
18Interaction with TE: Multi-Commodity Flow Multi-commodity flow optimizer interactswith selfish overlays much more effectively.
19History: Bistability in Single Overlay Phone network is an overlayLogical link between each pair of switchesPhone call put on one-hop path, when possible… and two-hop alternate path otherwiseProblem: inefficient path assignmentTwo-hop path for one phone call… stops another call from using direct path… forcing the use of a two-hop alternate pathbusybusy
20Preventing Inefficient Routes: Trunk Reservation Two stable states for the systemMostly one-hop calls with low blocking rateMostly two-hop calls with high blocking rateMaking the system stableReserve a portion of each link for direct callsWhen link load exceeds threshold…… disallow two-hop paths from using the linkRejects some two-hop calls… to keep some spare capacity for future one-hop callsStability through trunk reservationSingle efficient, stable state with right threshold
21Should ISPs Fear Overlays, or Favor Them? BillingCon: overlays commoditize the network providersPro: overlay traffic adds traffic subject to billingEngineeringCon: traffic matrix becomes less predictablePro: TE less important because overlays can adaptValue-added servicesCon: overlays become the place for new servicesPro: ISPs can provide overlay nodes in the coreDo the pros outweigh the cons? Beat ‘em, or join ‘em?
22What Could ISPs do to Help Overlays? Better visibilityMeasurements from the underlying networkBetter controlInfluence over underlay path selectionLower costAvoiding need to traverse intermediate hostsBetter joint optimizationUnderlay adaptation accountings for the overlays
23ISP Support: Greater Visibility Via Measurements Underlying topologyDependencies: between paths for virtual linksResource limits: capacity of the underlying linksRouting-protocol update streamsFast adaptation: BGP may provide early warningBetter adaptation: identify location of a problemPerformance measurementStatistics: per-link delay, loss, and throughputEfficiency: consolidating probes for many overlaysAre there enough incentives to share the data?
24ISP Support: Direct Control Influence over paths for virtual linksGuarantees of link/node disjoint paths?Control over packet forwardingOverlays allowed to install forwarding entries?Warnings about planned changesPrior notification about planned maintenanceRerouting of virtual link before the outageAre there effective ways to share control?
25ISP Support: Lower Cost Measurement service to overlaysReduce overhead and cost of active probingDeflection points inside the networkAvoiding packets traversing intermediate hostsEncapsulation to deflect packet through landmarkHosting of servers for running overlaysEnhanced form of a hosting serviceServers running directly in the ISP’s PoPsAre there reasonable business models and technical solutions?
26ISP Support: Joint Optimization Changes in traffic-engineering practicesOverlay-friendly traffic engineering?Ways to ensure stability and optimality?Or, just let the overlays manage the traffic?Cross-domain cooperationManagement of paths and traffic across ASesVirtual links that span two (or more) ISPsService-level agreements spanning ASes?Are there reasonable optimization or game-theory models?
27Are overlay networks a good idea? DiscussionAre overlay networks a good idea?Just a hack to avoid changing the underlay?What if we could “fix” the underlying network?Would we still have a need for overlay networks?Should we have overlay-friendly underlays?Or underlay-friendly overlays, or both?Visibility, control, economics, efficiency, …Or, are the two systems inherently at odds?What about interactions between overlays?Cooperate to reduce measurement cost and prevent suboptimality and instability?Compete because that’s the way life works?
28Overlays Interaction effects Avenues for new work Conclusions Enables innovation in routing and forwarding… without changing the underlying networkInteraction effectsWith background trafficWith other overlaysWith traffic engineeringAvenues for new workPossibility the interaction effects are good?Ensuring stability and efficiency are achieved?Right interplay between underlay and overlay?