1 Internet Routing: BGP Routing Convergence Jennifer Rexford Princeton University

2 Goals of This Section BGP routing changes –Detecting failures –Path exploration Reducing convergence time –Route flap damping and lower timer values –Favoring stability, root-cause tags, extra routes BGP stability –Stable paths problem and policy conflicts –Policy guidelines that ensure stability Active research areas on Internet routing –Location/identifier separation, routing servers, multipath routing, overlays and network virtualization

3 BGP Routing Changes

4 Causes of BGP Routing Changes Topology changes –Equipment going up or down –Deployment of new routers or sessions BGP session failures –Due to equipment failures, maintenance, etc. –Or, due to congestion on the physical path Changes in routing policy –Changes in preferences in the routes –Changes in whether the route is exported Persistent protocol oscillation –Conflicts between policies in different ASes

5 BGP Session Failure BGP runs over TCP –BGP only sends updates when changes occur –TCP doesn’t detect lost connectivity on its own Detecting a failure –Keep-alive: 60 seconds –Hold timer: 180 seconds Reacting to a failure –Discard all routes learned from the neighbor –Send new updates for any routes that change AS1 AS2

6 Routing Change: Before and After (1,0) (2,0) (3,1,0) (2,0) (1,2,0) (3,2,0)

7 Routing Change: Path Exploration AS 1 –Delete the route (1,0) –Switch to next route (1,2,0) –Send route (1,2,0) to AS 3 AS 3 –Sees (1,2,0) replace (1,0) –Compares to route (2,0) –Switches to using AS (2,0) (1,2,0) (3,2,0)

8 Routing Change: Path Exploration Initial situation –Destination 0 is alive –All ASes use direct path When destination dies –All ASes lose direct path –All switch to longer paths –Eventually withdrawn E.g., AS 2 –(2,0)  (2,1,0) –(2,1,0)  (2,3,0) –(2,3,0)  (2,1,3,0) –(2,1,3,0)  null (1,0) (1,2,0) (1,3,0) (2,0) (2,1,0) (2,3,0) (2,1,3,0) (3,0) (3,1,0) (3,2,0)

9 BGP Converges Slowly Path vector avoids count-to-infinity –But, ASes still must explore many alternate paths –… to find the highest-ranked path that is still available Fortunately, in practice –Most popular destinations have very stable BGP routes –And most instability lies in a few unpopular destinations Still, lower BGP convergence delay is a goal –Can be tens of seconds to tens of minutes –High for important interactive applications –… or even conventional application, like Web browsing

10 Reducing BGP Convergence Time

11 Existing Solution: Tune MRAI TImer Minimum route advertisement interval (MRAI) –Minimum spacing between announcements –For a particular (prefix, peer) pair Advantages of large MRAI –Provides a rate limit on BGP updates –Allows grouping of updates within the interval Disadvantages of large MRAI –Adds delay to the convergence process –E.g., 30 seconds for each step Trade-off overhead for convergence time

12 Existing Solution: Route-Flap Damping Identify (prefix, next-hop) that changes often –Suppress route until stable for a period of time Problematic in practice –Path exploration can inadvertently trigger RFD –May suppress all routes, leaving no route left Reuse limit Time Penalty Suppress limit Network Announced Network Re-announced Network Not Announced

13 Proposed: Preferring More Stable Routes Alternative to route-flap damping –Score routes on how stable they are  E.g., time elapsed since the last change –Incorporate into the path-selection decision  Prefer more stable routes over less stable routes Advantages –Always select a route, if one is available –Prevents excessive routing changes –Creates incentives for greater stability Disadvantages –Leads to non-determinism in route selection –Requires state for each route

14 Proposed: Root-Cause Tagging Identify reason for changing the route –E.g., which node or edge failed Allow routers to skip routes with same fate –E.g., routes with same node or edge in AS path Practical challenges –Multiple routers or links per AS –Incremental deployment d s 4 5

15 Proposed: Disseminating Backup Routes Disseminating extra (backup) routes –So a route is available after a failure –To enable faster forwarding convergence 1 d d d 2 1 d 1 d Announce alternate route to neighbor –AS 3 makes “3 4 5 d” available to 2 –AS 2 makes “ d” to AS 1 –So ASes can switch immediately

16 BGP Routing Stability

17 Stable Paths Problem (SPP) Model Model of routing policy –Each AS has a ranking of the permissible paths Model of path selection –Pick the highest-ranked path consistent with neighbors Flexibility is not free –Global system may not converge to a stable assignment –Depending on the way the ASes rank their paths 1 2 d 1 d 2 3 d 2 d 3 1 d 3 d d

18 Permanent Oscillation: “Bad Gadget” Pick the highest-ranked path consistent with your neighbors’ choices. Only choice! Top choice! Only choice! Better choice! Only choice! Better choice!

19 Two Stable Solutions: Disagree Each AS prefers the path through the other Two stable states –AS 2 picks “2 0”, and AS 1 picks “1 2 0” –AS 1 picks “1 0”, and AS 2 picks “2 1 0” Outcome depends on timing/ordering of messages

Ways to Achieve Global Stability Detect conflicting rankings of paths? –Computationally intractable (NP-hard) –Requires global coordination Restrict the policy programming languages? –In what way? How to require this globally? –What if the world should change, and the protocol can’t? Rely on economic incentives? –Policies typically driven by business relationships –E.g., customer-provider and peer-peer relationships –Sufficient conditions to guarantee unique, stable solution

21 Bilateral Business Relationships Provider-Customer –Customer pays provider for access to the Internet Peer-Peer –Peers carry traffic between their respective customers d Provider-Customer Peer-Peer Valid paths: “1 2 d” and “7 d” Invalid path: “5 8 d” Valid paths: “6 4 3 d” and “8 5 d” Invalid paths: “6 5 d” and “1 4 3 d”

22 Act Locally, Prove Globally Global topology –Provider-customer relationship graph is acyclic –Peer-peer relationships between any pairs of ASes Route export –Do not export routes learned from a peer or provider –… to another peer or provider Route selection –Prefer routes through customers –… over routes through peers and providers Guaranteed to converge to unique, stable solution

23 Rough Sketch of the Proof Two phases –Walking up the customer-provider hierarchy –Walking down the provider-customer hierarchy d Provider-Customer Peer-Peer

24 Trade-offs Between Assumptions Three kinds of assumptions –Route export, route selection, global topology –Relax one assumption, need to tighten other two Extensions for other kinds of relationships –Backups, siblings, … But, many questions remain –Complete understanding of the trade-offs –Business practices may change over time –ASes may lie about their paths –Protocol extensions for multi-path routing

25 Research Directions: New Internet Routing Architectures

26 Why Change Routing? Better performance –Scalability, security, convergence, reliability, flexibility, stability, … Simpler management –For network operators –For folks deploying services Greater extensibility –To enable experimentation –To enable new services

27 What to Change, and Where? Add another layer about network routing –Routing functionality in overlay networks Change the routing protocols –To improve scalability, security, convergence, … Change the division of functionality –Data, control, and management planes Change the division of responsibility –End users, third parties, and service providers ???

28 Theme: Location/Identity Separation Scalability problems with BGP –300,000 prefixes and growing –Difficult in handling mobility Idea: separate location and identity –Identity associated with a host or group of hosts –Location is “looked up” when sending packets Examples –Route packets based on destination AS –Route packets based on “label” found in DNS –Establish e2e paths and associate with labels

29 Server Theme: Separating Routing From Routers Today’s routers do many things –Compute routes, forward packets, monitoring Separate service for computing routes –Better scalability, network-wide view, … Several deployment scenarios –Within an AS  Incrementally deployable  Use BGP to instruct the routers –Across multiple ASes  Routing as a Service  Provided by third parties AS 2 Server

30 Theme: Multipath Routing Benefits of multipath routing –Efficiency, performance, reliability, and security –Greater control to users and edge ASes Many ways to construct multiple paths –Multipath extensions to BGP –Overlays on top of BGP –Stitching together sub-paths –Source routing Many new challenges –Scalability –Stable load balancing –Incentives for participation d

31 Theme: Overlays and Virtualization Build end-to-end topologies –Overlays by tunneling from one node to another –Virtual networks by “hosting” overlays on the routers Separation of “interdomain” issues –Instantiate a (virtual) topology over the infrastructure –Run (intradomain) routing protocols on this topology Competing ISPs with different goals must coordinate Single service provider controls end-to-end path

32 Conclusion Internet routing –A competitive cooperation of ~40,000 networks –Policy-based path-vector routing protocol on prefixes –Tension between local autonomy and global properties Many important practical challenges –Scalability, stability, flexibility, performance, reliability, … Many interesting research directions –Understanding today’s BGP –Extensions and enhancements to BGP –Entirely new Internet routing architectures Please, please help us fix interdomain routing!!!