1. Beyond BGP Dan Massey Colorado State University

2. 24 October 042massey@cs.colostate.edu Internet Routing l Challenges Facing Internet Routing n Internet Has Grown Dramatically –Large number of routing entries –High volumes of updates –Frequent topological changes n Fault-Model Has Changed Dramatically –More malfunctioning components –Intentional attacks l Do we need a fundamentally new routing architecture?

3. 24 October 043massey@cs.colostate.edu Toward a New Architecture l One claim: BGP is nearing the end of its useful lifetime n The Internet will soon collapse unless we act!! l Other claim: BGP is the best engineering solution we are likely to produce n We need incremental patches to new problems l Who is right? n Beyond BGP uses –Measurements to assess where we are –Identification of (new?) routing requirements –Development of changes (incremental or new system) to address the above

4. 24 October 044massey@cs.colostate.edu How Did We Get To BGP l Simple Distance Vector Routing Algorithms n Used in early Internet routing designs n Convey only limited information n Prone to long lasting loops l Expensive Link State Routing Algorithms n Learn the Full Network Topology n Signal every change in every link l Path Vector Routing (BGP) n Middle ground that signals some path data n But does not signal the full topology

5. 24 October 045massey@cs.colostate.edu RIP and DBF RIP Keep shortest path only Distributed Bellman-Ford(DBF) Keep distance info from all neighbors A B C EF D D:1 D:3 D:2 D:3 B’s route to D: Nexthop=A, Dist=4 B’s route to D: Nexthop=A, dist=4 Alternate Nexthop=C, Dist=4 D: infinity 30sec refreshing interval Damping timer to space out two triggered updates: 1~5 seconds Poison reverse: B sends infinity distance to A RIP and DBF: Exchange distance info.

6. 24 October 046massey@cs.colostate.edu Internet: composed of thousands of Autonomous Systems(ASes). BGP Background BGP (Border Gateway Protocol): the de facto inter-AS routing protocol AS A R1 R2 R3 AS B AS C R4 R5 AS E R6 BGP Routers

7. 24 October 047massey@cs.colostate.edu How BGP works l Uses path vector protocol –similar to distance vector protocol. what if no path available? Consider an AS as a node Route via A = Route via C = B’s route to D: route includes entire path(sequence of nodes) D A B C E D:

8. 24 October 048massey@cs.colostate.edu Path Vector Routing Changes l Worms triggered edge instabilty n Routers crashed due to ARP cache overflow. n Links were congested by worm traffic. l BGP Path Exploration Exacerbates Dynamics B’s route to D Route via A= Route via C= D ABC E Obsolete backup path is used and convergence is delayed withdraw

9. 24 October 049massey@cs.colostate.edu Policies and Policy Withdrawal But A could stop advertising to B due to a policy change, path is still valid! ABC E policy withdraw D Attach a Failure Withdrawal Community Attribute Only apply the approach to failure withdrawal B’s route to D Route via A= Route via C= Route via A= A B C E

10. 24 October 0410massey@cs.colostate.edu BGP Traffic Engineering BGP Traffic Engineering: R4 chooses path R5 chooses path We assumed an AS could be modeled as a node with a single best path to the destination But a single AS may advertise more than one path. Divide one AS into Logical ASes such that All routers within a logical AS have the same best path  each logical AS can be modeled as a node.

11. 24 October 0411massey@cs.colostate.edu Number of Updates Number of ASes in Network Number of Updates Original BGP Enhanced BGP Substantial reduction is achieved. E.g. 3766 to 1419 in the 60-AS topology MinRouteAdver timer: within 30 seconds, only one advertisement is allowed. It “packs” consecutive changes into one update.

12. 24 October 0412massey@cs.colostate.edu Convergence time Number of ASes in Network Convergence Time(seconds) Original BGP Enhanced BGP Enhanced BGP reduces the convergence time substantially. E.g. 337.0 seconds to 19.5 seconds in the 60- AS topology Elimination of one advertisement can cut convergence time by 30 seconds

13. 24 October 0413massey@cs.colostate.edu Improving Path Vector Convergence l Infocom 02 [4] uses consistency to detect invalid paths. n Reject path if r1 is adirect neighbor r1’s path is not n Adjusted to account for policy and implement in BGP l Infocom 03 [Afek, et al] quickly flushes invalid paths. n BGP requires updates be separated by a min interval n Send withdraw (to flush route) if blocked by the interval l Our recent work [5] attaches a new attribute: Root Cause Notification (RCN) n Identifies the failed link and includes a sequence number. n Allows any route relying on the failed link to be rejected.

14. 24 October 0414massey@cs.colostate.edu Analyzing Path Vector Convergence l Route fail-over has two stages. l First, nodes inside the blue triangle lose routes and explore backup paths. n All short invalid paths are explored l Second, an edge (a0) eventually selects the valid backup path via Sk. n Valid routes begin to propagate through the blue triangle.

15. 24 October 0415massey@cs.colostate.edu Generic Convergence Results Algorithm Fail-Over Convergence Bounds SPVP (BGP)(N-1) (M + ld) + 3 Pmax(|E|-degree(G,0)) SPVP-AS(N- degree(G,0) ) (M+ld) + 3Pmax(|E| - |E^| + Degree(G^)) SPVP-GF(N-1) ld + 3Pmax(|E| - degree(G,0)) SPVP-RCNDistance(G,0) (ld) + (Pmax) Distance(G,0) Pmax = Node Processing Delay, ld = Link Delay M = Minimum Advertisement Interval

16. 24 October 0416massey@cs.colostate.edu Simulation Results

17. 24 October 0417massey@cs.colostate.edu What About Security? l Convergence Discussion Neglects Security n What if routers send intentionally bad information? l What is the Simplest Possible Attack? n Announce someone elses routes l Example: Suppose Univ. of Colorado announces it is the origin for 129.82.0.0/16 n In other words, CU announces CSU IP Address Space l Can this Happen and/or What Would Prevent It?

18. 24 October 0418massey@cs.colostate.edu Multiple Origin AS (MOAS) Cases l Prefixes originate from Multiple Origin AS (MOAS) n Lower curve likely due to valid operational needs l Spikes are errors that disrupt routing to prefix n Includes loss of routes to top level DNS servers

19. 24 October 0419massey@cs.colostate.edu Infrastructure Faults and Attacks Internet c.gtld-servers.net BGP monitor 192.26.92.30 originates route to 192.26.92/24 l BGP and DNS Provide No Authentication n Faults and attacks can mis-direct traffic. n One (of many) examples observed from BGP logs. n Server could have replied with false DNS data. ISPs announced new path for 20 minutes to 3 hours

20. 24 October 0420massey@cs.colostate.edu BGP-based Solution Example router bgp 59 neighbor 1.2.3.4 remote-as 52 neighbor 1.2.3.4 send-community neighbor 1.2.3.4 route-map setcommunity out route-map setcommunity match ip address 18.0.0.0/8 set community 59:MOAS 58:MOAS additive Example configuration: AS58 18/8, PATH, MOAS{4,58,59} AS59 18.0.0.0/8 18/8, PATH, MOAS{58,59} 18/8, PATH, MOAS{52, 58} AS52

21. 24 October 0421massey@cs.colostate.edu (b) Two Origin AS’s(a) One Origin AS BGP false origin detection Simulation Results

22. 24 October 0422massey@cs.colostate.edu A Simple Filter l Current BGP provides dynamic routes n Explore the opposite extreme... l Select a single static route to each server. n Apply AS path filters to block all other announcements. –Also filter against more specifics. l Route changes on a frequency of months, if at all. n Change in IP address, origin AS, or transit policy. n Adjust route only after off-line verification

23. 24 October 0423massey@cs.colostate.edu Why This Works: Theory l Scale is limited to a small number of routes. n No exponential growth in top level DNS servers. l Loss of a server is tolerable, invalid server is not. n Resolvers detect and time-out unreachable servers. –Provided surviving servers handle load, cost is some delay. l Expect predictable properties and stable routes. n Servers don’t change without non-trivial effort. n Servers located in highly available locations.

24. 24 October 0424massey@cs.colostate.edu Why This Works: Data l Analysis based on BGP updates from RIPE. n Archive of BGP updates sent by each peer. n 9 ISPs from US, Europe, and Japan. n February 2001 - April 2002 l Some data collection notes n Used only peers that exchange full routing tables –Otherwise some route changes are hidden by policies n Adjusted data to discount multi-hop effect. –Multi-hop peering session resets don’t reflect ISP ops.

25. 24 October 0425massey@cs.colostate.edu Impact on Reachability ISP1 (US/Tier 1)

26. 24 October 0426massey@cs.colostate.edu How Static Are The Routes? l 3 changes in route to “A” over 14 months. l 2 (valid) changes in the origin AS n 5/19/01 origin AS changed from 6245 to 11840 n 6/4/01 origin AS changed from 11840 to 19836 l 1 change in transit AS routing policy n 11/8/01 (*,10913, 10913, 10913,*) -> (*,10913, *) n Could have built filter to allow this...

27. 24 October 0427massey@cs.colostate.edu What Routes Are Lost? l Results from 3/1/01 until 5/19/01 AS change. n Reduced reachability to “A” from 99.997% to 99.904% l 18 events when trusted route was withdrawn n 2 resulted in no route available (28 secs, 103 secs) n 8 instances of a back-up route lasting over 3 minutes n Longest lasting back-up advertised for 15 minutes l Similar results for other time periods and servers.

28. 24 October 0428massey@cs.colostate.edu Example of Filtered Routes l With filter no route at 16:06:32 19836 10913 1239 701 * server No route at 16:08:30

29. 24 October 0429massey@cs.colostate.edu Worst Case In Study ISP 3 (Europe) ISP 3 used one main route and a small number of consistent back-up routes.

30. 24 October 0430massey@cs.colostate.edu Toward a More Balanced Approach l Required infrequent updates to the filter. n Especially useful to automate infrequent tasks. –Natural tendency to forget task or forget how to do task l More paths improves robustness n Simple filtered allowed only 1 path. n ISP3’s reachability can be improved if filter allows two routes… l Strike a balance between allowing dynamic changes and restricting to trusted paths.

31. 24 October 0431massey@cs.colostate.edu BGP Adaptive Filters l Slow down the route dynamics and add validation. n Apply hysteresis before accepting new paths n Add options for validating new paths: –Believe route based purely on hysteresis –Probabilistic query/response testing against known data. –Trigger off-line checking (did origin AS really change?)

32. 24 October 0432massey@cs.colostate.edu Impacts on Reachability ISP1 Root servers gTLD servers

33. 24 October 0433massey@cs.colostate.edu Impacts on Reachability ISP3 Root servers gTLD servers

34. 24 October 0434massey@cs.colostate.edu Convergence And Authentication l BGP Suffers From Both Convergence Problems and Authentication Problems n Convergence fixes are good, if no attacks. n Authentication fixes work for redundant sites l Can you improve both convergence and authentication in a realistic environment? n Do you need to replace BGP? –If yes, with what? n Would you pick BGP for your new network? –If no, what would you do instead? l Wide Variety of Other Routing Challenges n Check out CS 580 and BBGP Project if interested

35. 24 October 0435massey@cs.colostate.edu BGP Measurement and Artifacts l BGP peers establish TCP session and send full route table (120K+ routes) n Updates sent only if routes change. l Our results show frequent session resets between ISP routers and the monitoring point. n Monitoring point sessions cross multiple systems in the Internet. n Each reset adds 120K updates. n But very few ISP-ISP session resets. l Our work in [1] presents rules to remove session reset artifacts. Initial Table (120K+ routes) Route Changes Initial Table (120K+ routes)

36. 24 October 0436massey@cs.colostate.edu BGP Updates During Slammer Worm

37. 24 October 0437massey@cs.colostate.edu BGP Updates During Nimda Worm Measurement Artifacts Routing Changes Total Attack

38. 24 October 0438massey@cs.colostate.edu What Our Analysis Shows 40.2% A substantial percentage of the BGP messages during the worm attack were not about route changes 37.6% 8.8% 8.3%

39. 24 October 0439massey@cs.colostate.edu FRTR: Improving Peer Communication l BGP Updates Are Not (Topology) Event Driven n Session resets trigger high volume surges –Govindan shows cascade failures can result. l Lifetime of Invalid Routes is Unbounded n Never recover (until reset) if update is somehow lost. –Despite TCP, we found cases of “lost” withdrawals. n Attacker can poison a route with one update. l Soft-state (periodic re-announce) is too costly… l FRTR Uses Periodic Bloom Filter Digests n Digests quickly confirm state after session reset. n Periodic digests bound lifetime of faults (w/ high prob). n Co-Author Keyur Patel (Cisco) is exploring Cisco development.

40. 24 October 0440massey@cs.colostate.edu FRTR Performance l For each route at receiver, check against the digest. n Bloom filter results in no false negatives. l Compare total digests for missing route detection. n False positive possible with known rate. n Add salts to reduce the chance of repeated false positives. l Overhead is a function of digest size and frequency. l Work with Cisco suggests a 1.3% overhead increase. l Complete Details to appear in [2] (DSN 2004)

41. 24 October 0441massey@cs.colostate.edu Packet Delivery during Routing Convergence l Failures do occur in the Internet –20% of intra-ISP links have a MTTF < 1 day[Diot:IMW02] –40% of Inter-ISP routes have a MTT-Change < 1 day [Labovitz:FTCS-29] l Routing convergence after failure takes time –IS-IS(Intra-ISP protocol): 5+ seconds [Diot:IMW02] –BGP(Inter-ISP protocol): 3+ minutes [Labovitz:Sigcomm00] l Packets can be delivered during convergence ABC E F D G

42. 24 October 0442massey@cs.colostate.edu What Is the Goal of Routing l How to maximize packet delivery during routing convergence? –Topological connectivity’s impact? –Studying: RIP, Distributed Bellman-Ford( DBF ), BGP – Previous work focused on: preventing loops, minimizing convergence time and routing overhead This problem becomes more important with Larger Internet topology [Huston01] --> higher freq. of component failures Richer connectivity[Huston01] --> potentially helps with more alternate paths Higher bandwidth --> more packets sent during convergence

43. 24 October 0443massey@cs.colostate.edu Simulation conducted 7 by 7 mesh topologies similar those in [Baran64] 20 pkts/second l Measure Packet loss, loops, path convergence time, throughput, and e2e delay. Simulated node degree range [3 ~ 16]

44. 24 October 0444massey@cs.colostate.edu Packet Losses (I) : Observation RI P DBF, BGP’ and BGP Packet losses of DBF, BGP’ and BGP decrease to zero at degree 6. Richer connectivity helps RIP little. Node Degree Packet Loss

45. 24 October 0445massey@cs.colostate.edu Packet Loss(II): Lessons Learned l Keeping alternate paths F D A B C E F D A B C E Connectivity Matters no immediate available alternative due to poor connectivity and poison reverse RIP: DBF, BGP: alternative is more likely with richer connectivity

46. 24 October 0446massey@cs.colostate.edu Is an alternate path valid? l Valid Alternate Paths: not using the failed link n Poison reverse and BGP’s path information are not enough! [Pei:Infocom2002] F D A B C E U X V W Richer connectivity --> reduces one single link’s impact better availability of valid(but may be suboptimal) path C2 D:

47. 24 October 0447massey@cs.colostate.edu Transient Loops(I): Observation DB F BGP’ BGP BGP has the most loops! RIP has no loops Richer connectivity reduces the chance of looping. Node Degree Losses due to loops

48. 24 October 0448massey@cs.colostate.edu F D A B C E Transient Loops(II): Msg Propagation Damping timer slows the msg propagation, causing looping U X V W Y D: D:<BAEF>D:<BAEF> Richer connectivity can reduce the chance of looping More details in: “A Study of Transient Loops in BGP” 30 seconds! D:

49. 24 October 0449massey@cs.colostate.edu Instantaneous Throughput RIP DBF BGP’ BGP RIP Time Throughput(pkts/second

50. 24 October 0450massey@cs.colostate.edu Packet Delay During Convergence

51. 24 October 0451massey@cs.colostate.edu Forwarding Path Convergence time BGP: no loss at degree 6 or higher Shall we still tune MRAI timer to minimize convergence time(with the risk of increasing overhead)? Node Degree BGP:70 BGP’:10 Time till there is no routing msg. BGP:13 BGP’:2 Time till the forwarding path from S to D stabilizes.

52. 24 October 0452massey@cs.colostate.edu Packet Delivery After a Failure