1. CS 268: Lecture 9 Inter-domain Routing Protocol Scott Shenker and Ion Stoica Computer Science Division Department of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, CA 94720-1776 (*slides from Timothy Griffin and Craig Labovitz)

2. 2 Overview  An Introduction to BGP  BGP and the Stable Paths problem  Convergence of BGP in the real world  The End-to-End Effects of Internet Path Selection

3. 3 Internet Routing  Internet organized as a two level hierarchy  First level – autonomous systems (AS’s) -AS – region of network under a single administrative domain  AS’s run an intra-domain routing protocols -Distance Vector, e.g., RIP -Link State, e.g., OSPF  Between AS’s runs inter-domain routing protocols, e.g., Border Gateway Routing (BGP) -De facto standard today, BGP-4

4. 4 Example AS-1 AS-2 AS-3 Interior router BGP router

5. 5 Inter-domain Routing basics  Internet is composed of over 16000 autonomous systems  BGP = Border Gateway Protocol -Is a Policy-Based routing protocol -Is the de facto inter-domain routing protocol of today’s global Internet  Relatively simple protocol, but configuration is complex and the entire world can see, and be impacted by, your mistakes.

6. 6 Inter-domain Routing  Use TCP  Border Gateway Protocol (BGP), based on Bellman-Ford path vector  AS’s exchange reachability information through their BGP routers, only when routes change  BGP routing information – a sequence of AS’s indicating the path traversed by a route; next hop  General operations of a BGP router: -Learns multiple paths -Picks best path according to its AS policies -Install best pick in IP forwarding tables

7. 7 BGP Operations (Simplified) Establish session on TCP port 179 Exchange all active routes Exchange incremental updates AS1 AS2 While connection is ALIVE exchange route UPDATE messages BGP session

8. 8 Customers and Providers Customer pays provider for access to the Internet provider customer IP traffic provider customer

9. 9 The “Peering” Relationship peer customerprovider Peers provide transit between their respective customers Peers do not provide transit between peers Peers (often) do not exchange $$$ traffic allowed traffic NOT allowed

10. 10 Peering Provides Shortcuts Peering also allows connectivity between the customers of “Tier 1” providers. peer customerprovider

11. 11 Peering Wars  Reduces upstream transit costs  Can increase end-to-end performance  May be the only way to connect your customers to some part of the Internet (“Tier 1”)  You would rather have customers  Peers are usually your competition  Peering relationships may require periodic renegotiation Peering struggles are by far the most contentious issues in the ISP world! Peering agreements are often confidential. PeerDon’t Peer

12. 12 Architecture of Dynamic Routing AS 1 AS 2 BGP EGP = Exterior Gateway Protocol IGP = Interior Gateway Protocol Metric based: OSPF, IS-IS, RIP, EIGRP (cisco) Policy based: BGP The Routing Domain of BGP is the entire Internet OSPF EIGRP

13. 13 AS-Path  Sequence of AS’s a route traverses  Used for loop detection and to apply policy 120.10.0.0/16 130.10.0.0/16 110.10.0.0/16 AS-1 AS-2 AS-3 AS-4 AS-5 120.10.0.0/16 AS-2 AS-3 AS-4 130.10.0.0/16 AS-2 AS-3 110.10.0.0/16 AS-2 AS-5

14. 14 Four Types of BGP Messages  Open : Establish a peering session.  Keep Alive : Handshake at regular intervals.  Notification : Shuts down a peering session.  Update : Announcing new routes or withdrawing previously announced routes. announcement = prefix + attributes values

15. 15 Two Types of BGP Neighbor Relationships External Neighbor (eBGP) in a different Autonomous Systems Internal Neighbor (iBGP) in the same Autonomous System AS1 AS2 eBGP iBGP iBGP is routed using Interior Gateway Protocol (IGP)!

16. 16 iBGP Peers Must be Fully Meshed iBGP neighbors do not announce routes received via iBGP to other iBGP neighbors. eBGP update iBGP updates iBGP is needed to avoid routing loops within an AS Injecting external routes into IGP does not scale and causes BGP policy information to be lost BGP does not provide “shortest path” routing

17. 17 Important BGP attributes  LocalPREF -Local preference policy to choose “most” preferred route  Multi-exit Discriminator (MED) -Which peering point to choose?  Import Rules -What route advertisements do I accept?  Export Rules -Which routes do I forward to whom?

18. 18 Route Selection Summary Highest Local Preference Shortest ASPATH Lowest MED i-BGP < e-BGP Lowest IGP cost to BGP egress Lowest router ID traffic engineering Enforce relationships Throw up hands and break ties

19. 19 Implementing Customer/Provider and Peer/Peer relationships  Enforce transit relationships -Outbound route filtering  Enforce order of route preference -provider < peer < customer Two parts:

20. 20 Import Routes From peer From peer From provider From provider From customer From customer provider routecustomer routepeer routeISP route

21. 21 Export Routes To peer To peer To customer To customer To provider From provider provider routecustomer routepeer routeISP route filters block

22. 22 Overview  An Introduction to BGP  BGP and the Stable Paths problem  Convergence of BGP in the real world  The End-to-End Effects of Internet Path Selection

23. 23 This talk What Problem is BGP solving? Underlying problem Shortest Paths Distributed means of computing a solution. X? RIP, OSPF, IS-IS BGP  aid in the design of policy analysis algorithms and heuristics,  aid in the analysis and design of BGP and extensions,  help explain some BGP routing anomalies,  provide a fun way of thinking about the protocol Having an X can

24. 24 1 Q : How simple can X get? A: The Stable Paths Problem (SPP) 2 5 5 2 1 0 0 2 1 0 2 0 1 3 0 1 0 3 0 4 2 0 4 3 0 3 4 2 1  A graph of nodes and edges,  Node 0, called the origin,  For each non-zero node, a set or permitted paths to the origin. This set always contains the “null path”.  A ranking of permitted paths at each node. Null path is always least preferred. (Not shown in diagram) An instance of the SPP : When modeling BGP : nodes represent BGP speaking border routers, and 0 represents a node originating some address block most preferred … least preferred (not null)

25. 25 5 5 2 1 0 1 A Solution to a Stable Paths Problem 2 0 2 1 0 2 0 1 3 0 1 0 3 0 4 2 0 4 3 0 3 4 2 1  node u’s assigned path is either the null path or is a path uwP, where wP is assigned to node w and {u,w} is an edge in the graph,  each node is assigned the highest ranked path among those consistent with the paths assigned to its neighbors. A Solution need not represent a shortest path tree, or a spanning tree. A solution is an assignment of permitted paths to each node such that

26. 26 Example: SHORTEST1 12430 2 0 2 1 0 1 0 1 3 0 4 3 0 4 2 0 3 0

27. 27 Example: SHORTEST1 (Solution) 12430 2 0 2 1 0 1 0 1 3 0 4 3 0 4 2 0 3 0

28. 28 Example: SHORTEST2 12430 2 0 2 1 0 1 0 1 3 0 4 2 0 4 3 0 3 0

29. 29 Example: SHORTEST2 (Solution) 12430 2 0 2 1 0 1 0 1 3 0 4 2 0 4 3 0 3 0

30. Example: GOOD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 0

31. 31 Example: GOOD GADGET (Solution) 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 0

32. 32 A Stable Paths Problem may have multiple solutions First solution 102 1 2 0 1 0 102102 2 1 0 2 0 1 2 0 1 0 2 1 0 2 0 1 2 0 1 0 2 1 0 2 0 Second solution DISAGREE

33. 33 Example: NAUGHTY GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 4 2 0 3 0

34. 34 Example: NAUGHTY GADGET (Solution 1) 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 4 2 0 3 0

35. 35 Example: NAUGHTY GADGET (Solution 2) 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 4 2 0 3 0

36. 36 SPP helps explain possibility of BGP divergence  BGP is not guaranteed to converge to a stable routing. Policy inconsistencies can lead to “livelock” protocol oscillations.  See “Persistent Route Oscillations in Inter-domain Routing” by K. Varadhan, R. Govindan, and D. Estrin. ISI report, 1996 SolvableCan Diverge must converge must diverge The SPP view :

37. 37 Example: NAUGHTY GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 4 2 0 3 0

38. 38 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0

39. 39 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

40. 40 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 2 chooses (2 1 0) …

41. 41 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 3 chooses (3 0) …

42. 42 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 1 chooses (1 3 0) …

43. 43 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 2 chooses (2 0) …

44. 44 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 3 chooses (3 4 2 0) … LOOP !

45. 45 2 0 3 1 2 1 0 2 0 1 3 0 1 0 3 4 2 0 3 0 4 4 2 0 4 3 0 BAD GADGET : No Solution With a BGP-like protocol, each node will do the best it can, so at least one node will always have the opportunity to improve its path. Result : persistent oscillation.

46. 46 SURPRISE : Beware of Backup Policies 2 0 3 1 2 1 0 2 0 1 3 0 1 0 3 4 2 0 3 0 4 4 0 4 2 0 4 3 0 Becomes BAD GADGET if link (4, 0) goes down. BGP is not robust : it is not guaranteed to recover from network failures.

47. 47 PRECARIOUS 1 0 2 1 2 0 1 0 2 1 0 2 0 3 4 5 6 5 3 1 0 5 6 3 1 2 0 5 3 1 2 0 6 3 1 0 6 4 3 1 2 0 6 3 1 2 0 4 3 1 0 4 5 3 1 2 0 4 3 1 2 0 3 1 0 3 1 2 0 As with DISAGREE, this part has two distinct solutions This part has a solution only when node 1 is assigned the direct path (1 0). Has a solution, but can get “trapped”

48. 48 What is to be done? Static Approach Inter-AS coordination Automated Analysis of Routing Policies (This is very hard). Dynamic Approach Extend BGP with a dynamic means of detecting and suppressing policy-based oscillations? These approaches are complementary

49. 49 Theoretical Results  The problem of determining whether an instance of stable paths problem is solvable is NP- complete  Shortest path route selection is provably safe

50. 50 Overview  An Introduction to BGP  BGP and the Stable Paths problem  Convergence of BGP in the real world  The End-to-End Effects of Internet Path Selection

51. 51 Convergence in the real-world?  [Labovitz99] Experimental results from two year study which measured 150,000 BGP faults injected into peering sessions at several IXPs  Found -Internet averages 3 minutes to converge after failover -Some multihomed failovers (short to long ASPath) require 15 minutes

52. 52 BGP Convergence Example R AS0 AS1 AS2 AS3 *B Rvia AS3 B R via AS0,AS3 B R via AS2,AS3 *B Rvia AS3 B R via AS0,AS3 B R via AS2,AS3 *B Rvia AS3 B R via AS1,AS3 B R via AS2,AS3 AS0AS1AS2 *** *B R via 203 *B R via 013 B R via 103

53. 53 Convergence Result  If we assume 1.unbounded delay on BGP processing and propagation 2.Full BGP mesh BGP peers 3.Constrained shortest path first selection algorithm  BGP is O(N!), where N number of default-free BGP speakers There exists possible ordering of messages such that BGP will explore all possible ASPaths of all possible lengths

54. 54 Outline of the Today’s class  An Introduction to BGP  BGP and the Stable Paths problem  Convergence of BGP in the real world  The End-to-End Effects of Internet Path Selection

55. 55 End-to-end effects of Path Selection  Goal of study: Quantify and understand the impact of path selection on end-to-end performance  Basic metric -Let X = performance of default path -Let Y = performance of best path -Y-X = cost of using default path  Technical issues -How to find the best path? -How to measure the best path?

56. 56 Approximating the best path  Key Idea -Use end-to-end measurements to extrapolate potential alternate paths  Rough Approach -Measure paths between pairs of hosts -Generate synthetic topology – full NxN mesh -Conservative approximation of best path  Question: Given a selection of N hosts, how crude is this approximation?

57. 57 Methodology  For each pair of end-hosts, calculate: -Average round-trip time -Average loss rate -Average bandwidth  Generate synthetic alternate paths (based on long-term averages)  For each pair of hosts,graph difference between default path and alternate path

58. 58 Courtesy: Stefan Savage

59. 59 Courtesy: Stefan Savage

60. 60 Courtesy: Stefan Savage

61. 61 Courtesy: Stefan Savage

62. 62 Why Path Selection is imperfect?  Technical Reasons -Single path routing -Non-topological route aggregation -Coarse routing metrics (AS_PATH) -Local policy decisions  Economic Reasons -Disincentive to offer transit -Minimal incentive to optimize transit traffic