1. CS 268: Lecture 11 Inter-domain Routing Protocol Karthik Lakshminarayanan UC Berkeley (substituting for Ion Stoica) (*slides from Timothy Griffin and Craig Labovitz)

2. 2 Overview  An Introduction to BGP  BGP and the Stable Paths problem  Convergence of BGP in the real world  The End-to-End Effects of Internet Path Selection

3. 3 Internet Routing  Internet organized as a two level hierarchy  First level – autonomous systems (AS’s) -AS – region of network under a single administrative domain  AS’s run an intra-domain routing protocols -Distance Vector, e.g., RIP -Link State, e.g., OSPF  Between AS’s runs inter-domain routing protocols, e.g., Border Gateway Routing (BGP) -De facto standard today, BGP-4

4. 4 Example AS-1 AS-2 AS-3 Interior router BGP router

5. 5 Inter-domain Routing basics  Internet is composed of over 20000 autonomous systems  BGP = Border Gateway Protocol -Policy-Based routing protocol -De facto inter-domain routing protocol of Internet  Relatively simple protocol, but… -complex configuration: entire world can see, and be impacted by, configuration mistakes

6. 6 BGP Basics  Use TCP  Border Gateway Protocol (BGP), based on Bellman-Ford path vector  AS’s exchange reachability information through their BGP routers, only when routes change  BGP routing information – a sequence of AS’s indicating the path traversed by a route  General operations of a BGP router: -Learns multiple paths -Picks best path according to its AS policies -Install best pick in IP forwarding tables

7. 7 BGP Operations (Simplified) Establish session on TCP port 179 Exchange all active routes Exchange incremental updates AS1 AS2 While connection is ALIVE exchange route UPDATE messages BGP session

8. 8 Customers and Providers Customer pays provider for access to the Internet provider customer IP traffic provider customer

9. 9 The “Peering” Relationship peer customerprovider Peers provide transit between their respective customers Peers do not provide transit between peers Peers (often) do not exchange $$$ traffic allowed traffic NOT allowed

10. 10 Peering Provides Shortcuts Peering also allows connectivity between the customers of “Tier 1” providers. peer customerprovider

11. 11 Peering Wars  Reduces upstream transit costs  Can increase end-to-end performance  May be the only way to connect your customers to some part of the Internet (“Tier 1”)  You would rather have customers  Peers are usually your competitors  Peering relationships may require periodic renegotiation Peering struggles are by far the most contentious issues in the ISP world! Peering agreements are often confidential. PeerDon’t Peer

12. 12 Architecture of Dynamic Routing AS 1 AS 2 BGP EGP = Exterior Gateway Protocol IGP = Interior Gateway Protocol Metric based: OSPF, IS-IS, RIP, EIGRP (cisco) Policy based: BGP OSPF EIGRP

13. 13 AS-Path  Sequence of AS’s a route traverses  Used for loop detection and to apply policy 120.10.0.0/16 130.10.0.0/16 110.10.0.0/16 AS-1 AS-2 AS-3 AS-4 AS-5 120.10.0.0/16 AS-2 AS-3 AS-4 130.10.0.0/16 AS-2 AS-3 110.10.0.0/16 AS-2 AS-5

14. 14 Four Types of BGP Messages  Open : Establish a peering session.  Keep Alive : Handshake at regular intervals.  Notification : Shuts down a peering session.  Update : Announcing new routes or withdrawing previously announced routes. announcement = prefix + attributes values

15. 15 Two Types of BGP Neighbor Relationships External Neighbor (eBGP) in a different Autonomous Systems Internal Neighbor (iBGP) in the same Autonomous System AS1 AS2 eBGP iBGP iBGP is routed using Interior Gateway Protocol (IGP) such as OSPF/IS-IS

16. 16 iBGP Peers Must be Fully Meshed iBGP neighbors do not announce routes received via iBGP to other iBGP neighbors. eBGP update iBGP updates iBGP is needed to avoid routing loops within an AS Injecting external routes into IGP does not scale and causes BGP policy information to be lost BGP does not provide “shortest path” routing

17. 17 Important BGP attributes  LocalPREF -Local preference policy to choose “most” preferred route  Multi-exit Discriminator (MED) -Which peering point to choose?  Import Rules -What route advertisements do I accept?  Export Rules -Which routes do I forward to whom?

18. 18 Route Selection Summary Highest Local Preference Shortest ASPATH Lowest MED i-BGP < e-BGP Lowest IGP cost to BGP egress Lowest router ID Traffic engineering Enforce relationships Throw up hands and break ties

19. 19 Implementing Customer/Provider and Peer/Peer relationships  Enforce transit relationships -Outbound route filtering  Enforce order of route preference -provider < peer < customer

20. 20 Import Routes From peer From peer From provider From provider From customer From customer provider routecustomer routepeer routeISP route

21. 21 Export Routes To peer To peer To customer To customer To provider From provider provider routecustomer routepeer routeISP route filters block

22. 22 Overview  An Introduction to BGP  BGP and the Stable Paths problem  Convergence of BGP in the real world  The End-to-End Effects of Internet Path Selection

23. 23 Our focus What Problem is BGP solving? Underlying problem Shortest Paths Distributed means of computing a solution X? RIP, OSPF, IS-IS BGP  aid in the design of policy analysis algorithms and heuristics,  aid in the analysis and design of BGP and extensions,  help explain some BGP routing anomalies,  provide a fun way of thinking about the protocol Having an X can

24. 24 1 Q : How simple can X get? A: The Stable Paths Problem (SPP) 25 5 2 1 0 0 2 1 0 2 0 1 3 0 1 0 3 0 4 2 0 4 3 0 3 4 2 1  graph of nodes and edges  node 0, called the origin  for each non-zero node, a set or permitted paths to the origin—this set always contains the “null path”  ranking of permitted paths at each node—null path is always least preferred An instance of the SPP : When modeling BGP : nodes represent BGP speaking border routers, and 0 represents a node originating some address block most preferred … least preferred (not null)

25. 25 5 5 2 1 0 1 A Solution to a Stable Paths Problem 2 0 2 1 0 2 0 1 3 0 1 0 3 0 4 2 0 4 3 0 3 4 2 1  node u’s assigned path is either the null path or is a path uwP, where wP is assigned to node w and {u,w} is an edge in the graph,  each node is assigned the highest ranked path among those consistent with the paths assigned to its neighbors. A Solution need not represent a shortest path tree, or a spanning tree. A solution is an assignment of permitted paths to each node such that

26. 26 Example: SHORTEST1 12430 2 0 2 1 0 1 0 1 3 0 4 3 0 4 2 0 3 0

27. 27 Example: SHORTEST1 (Solution) 12430 2 0 2 1 0 1 0 1 3 0 4 3 0 4 2 0 3 0

28. 28 Example: SHORTEST2 12430 2 0 2 1 0 1 0 1 3 0 4 2 0 4 3 0 3 0

29. 29 Example: SHORTEST2 (Solution) 12430 2 0 2 1 0 1 0 1 3 0 4 2 0 4 3 0 3 0

30. Example: GOOD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 0

31. 31 Example: GOOD GADGET (Solution) 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 0

32. 32 A Stable Paths Problem may have multiple solutions First solution 102 1 2 0 1 0 102102 2 1 0 2 0 1 2 0 1 0 2 1 0 2 0 1 2 0 1 0 2 1 0 2 0 Second solution DISAGREE

33. 33 Example: NAUGHTY GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 4 2 0 3 0

34. 34 Example: NAUGHTY GADGET (Solution 1) 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 4 2 0 3 0

35. 35 Example: NAUGHTY GADGET (Solution 2) 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 4 2 0 3 0

36. 36 SPP explains possibility of BGP divergence  BGP is not guaranteed to converge to a stable routing. Policy inconsistencies can lead to “livelock” protocol oscillations.  See “Persistent Route Oscillations in Inter-domain Routing” by K. Varadhan, R. Govindan, and D. Estrin. ISI report, 1996 SolvableCan Diverge must converge must diverge The SPP view :

37. 37 Example: NAUGHTY GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 3 0 4 2 0 3 4 2 0 3 0

38. 38 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0

39. 39 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

40. 40 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 2 chooses (2 1 0) … 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

41. 41 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 4 chooses ( ) … 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

42. 42 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 3 chooses (3 0) … 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

43. 43 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 4 chooses (4 3 0) … 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

44. 44 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 1 chooses (1 3 0) … 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

45. 45 Example: BAD GADGET 1230 2 1 0 2 0 1 3 0 1 0 3 4 2 0 3 0 … 2 chooses (2 0) … 4 4 2 0 4 3 0 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

46. 46 Example: BAD GADGET 1230 2 1 0 2 0 1 3 0 1 0 3 4 2 0 3 0 … 4 chooses (4 2 0) … 4 4 2 0 4 3 0 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

47. 47 Example: BAD GADGET 1230 2 1 0 2 0 1 3 0 1 0 3 4 2 0 3 0 … 3 chooses (3 4 2 0) … 4 4 2 0 4 3 0 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

48. 48 Example: BAD GADGET 12430 2 1 0 2 0 1 3 0 1 0 4 2 0 4 3 0 3 4 2 0 3 0 … 1 chooses (1 0) … That was one round of oscillation! 2 chooses (2 0) 4 chooses (4 2 0) 3 chooses (3 4 2 0) 1 chooses (1 0)

49. 49 2 0 3 1 2 1 0 2 0 1 3 0 1 0 3 4 2 0 3 0 4 4 2 0 4 3 0 BAD GADGET : No Solution In a BGP-like protocol each node local decisions at least one node can always improve its path Result: persistent oscillation

50. 50 SURPRISE : Beware of Backup Policies 4 0 4 2 0 4 3 0 Becomes BAD GADGET if link (4, 0) goes down. BGP is not robust : it is not guaranteed to recover from network failures. 2 0 3 1 2 1 0 2 0 1 3 0 1 0 3 4 2 0 3 0 4

51. 51 PRECARIOUS 1 0 2 1 2 0 1 0 2 1 0 2 0 3456 5 3 1 0 5 6 3 1 2 0 5 3 1 2 0 6 3 1 0 6 4 3 1 2 0 6 3 1 2 0 4 3 1 0 4 5 3 1 2 0 4 3 1 2 0 3 1 0 3 1 2 0 As with DISAGREE, this part has two distinct solutions This part has a solution only when node 1 is assigned the direct path (1 0). Has a solution, but can get “trapped”

52. 52 Theoretical Results  The problem of determining whether an instance of stable paths problem is solvable is NP- complete  Shortest path route selection is provably safe

53. 53 What is to be done? Static Approach Inter-AS coordination Automated Analysis of Routing Policies (This is very hard). Dynamic Approach Extend BGP with a dynamic means of detecting and suppressing policy-based oscillations? These approaches are complementary

54. 54 Overview  An Introduction to BGP  BGP and the Stable Paths problem  Convergence of BGP in the real world  The End-to-End Effects of Internet Path Selection

55. 55 Convergence in the real-world?  [Labovitz99] Experimental results from two year study which measured 150,000 BGP faults injected into peering sessions at several IXPs  Found: -Internet averages 3 minutes to converge after failover -Some multihomed failovers (short to long ASPath) require 15 minutes

56. 56 BGP Convergence Example R AS0 AS1 AS2 AS3 *B Rvia AS3 B R via AS0,AS3 B R via AS2,AS3 *B Rvia AS3 B R via AS0,AS3 B R via AS2,AS3 *B Rvia AS3 B R via AS1,AS3 B R via AS2,AS3 AS0AS1AS2 *** *B R via 203 *B R via 013 B R via 103

57. 57 Convergence Result  If we assume 1.unbounded delay on BGP processing and propagation 2.Full BGP mesh BGP peers 3.Constrained shortest path first selection algorithm  Convergence time of BGP is O(N!), where N number of BGP speakers There exists possible ordering of messages such that BGP will explore all possible ASPaths of all possible lengths

58. 58 Overview  An Introduction to BGP  BGP and the Stable Paths problem  Convergence of BGP in the real world  The End-to-End Effects of Internet Path Selection

59. 59 End-to-end effects of Path Selection  Goal of study: Quantify and understand the impact of path selection on end-to-end performance  Basic metric -Let X = performance of default path -Let Y = performance of best path -Y-X = cost of using default path  Technical issues -How to find the best path? -How to measure the best path?

60. 60 Approximating the best path  Key Idea -Use end-to-end measurements to extrapolate potential alternate paths  Rough Approach -Measure paths between pairs of hosts -Generate synthetic topology – full NxN mesh -Conservative approximation of best path  Question: Given a selection of N hosts, how crude is this approximation?

61. 61 Methodology  For each pair of end-hosts, calculate: -Average round-trip time -Average loss rate -Average bandwidth  Generate synthetic alternate paths (based on long-term averages)  For each pair of hosts,graph difference between default path and alternate path

62. 62 Courtesy: Stefan Savage

63. 63 Courtesy: Stefan Savage

64. 64 Courtesy: Stefan Savage

65. 65 Courtesy: Stefan Savage

66. 66 Why Path Selection is imperfect?  Technical Reasons -Single path routing -Non-topological route aggregation -Coarse routing metrics (AS_PATH) -Local policy decisions  Economic Reasons -Disincentive to offer transit -Minimal incentive to optimize transit traffic