1. 1 Internet Routing Instability and it's Origins Ilia Ferdman Lilia Tsvetinovich

2. 2 Abstract zProblems discussed yInternet Routing Instability yOrigins of Internet Routing Instability

3. 3 Internet Routing Instability zDefined as rapid fluctuation of network reachability and topology information zAlso referred as “route flap”

4. 4 Origins of Routing Instability zRouter configuration errors zTransient physical and data link problems zSoftware bugs

5. 5 Primary Effects zInstability can lead to yIncreased packet loss yDelays in the time for network convergence yAdditional resource overhead(memory, CPU) zImminent “death of the Internet”

6. 6 Internet Structure zComprised of interconnected regional and national backbones zLarge public exchange points are the “core” of the Internet

7. 7 Internet Structure (cont.) zBSP – Backbone service provider zEP – Exchange points BSP 1 BSP 1 EP2EP2 EP2EP2 EP4EP4 EP4EP4 EP3EP3 EP3EP3 EP1EP1 EP1EP1 BSP 3 BSP 3 BSP 2 BSP 2 BSP 5 BSP 5 BSP 7 BSP 7 BSP 4 BSP 4 BSP 6 BSP 6

8. 8 Internet Structure (cont.) zBackbone service providers exchange yTraffic yRouting information zBackbones in the core maintain default- free routing table

9. 9 Internet Structure (cont.) zAutonomous systems yDistinct routing policies yConnect to private or public exchange points zPeer border routers in AS exchange reachability information to prefixes zPrefixes – IP address blocks zExchange information through BGP

10. 10 Border Gateway Protocol zBGP yIncremental protocol yUses TCP yLimits distribution of routing information z IGRP, OSPF, etc yInterior protocols yUse datagram service yFlood network with all known routing table entries zBGP vs. IGRP & OSPF

11. 11 BGP (cont.) zAllows configuration for policy (MED) zMED – Multi Exit Descriptor zASPATH - list of AS numbers

12. 12 BGP (cont.) zAllows configuration for policy (MED) zMED – Multi Exit Descriptor zASPATH - list of AS numbers zBGP updates yAnnouncements yWithdrawals

13. 13 BGP updates - Withdrawals zExplicit Withdrawals zImplicit Withdrawals R1R1 R1R1 R2R2 R2R2 R3R3 R3R3 R1R1 R1R1 R2R2 R2R2

14. 14 BGP updates - Withdrawals zExplicit Withdrawals R1R1 R1R1 R2R2 R2R2 R3R3 R3R3

15. 15 BGP updates - Withdrawals zImplicit Withdrawals R1R1 R1R1 R2R2 R2R2

16. 16 BGP (cont.) zAllows configuration for policy (MED) zASPATH zBGP updates yAnnouncements yWithdrawals zStable wide-area networks performance expectations

17. 17 Methodology zSince January 1996, 9 months zRouting Arbiter project zPublic exchange points: AADS, Mae-East, Mae-West, PacBell, Sprint

18. 18 Methodology

19. 19 Methodology zMae-East backbone service providers: ANS, BBN, MCI, Sprint and UUNet zRAP – Routing Arbiter Project zRoute Servers used to collect information z12 gigabytes of compressed data

20. 20 Types of Routing Instability zBGP updates  Instability rate zForwarding instability zRouting Policy Fluctuations zPathological updates zInstability – instance of forwarding instability or policy fluctuations

21. 21 Possible impacts zIncrease in cache misses zCPU & memory problems zRoute “flap storm” zForwarding loops

22. 22 Route Caching Architecture zRouting table cache of destination and next-hop lookups zRouting table is too big to keep it in main memory zInstability causes increase in cache misses zLoad on CPU

23. 23 Route Caching Architecture zPossible solution: yFull routing table in main memory

24. 24 Possible impacts zIncrease in cache misses zCPU & memory problems zRoute “flap storm” zForwarding loops

25. 25 CPU & Memory Problems zNormally could manage the router’s computational needs zInstability places large demands on a router’s CPU zKeep-Alive packets delayed

26. 26 Possible impacts zIncrease in cache misses zCPU & memory problems zRoute “flap storm” zForwarding loops

27. 27 Route “flap storm” Overloaded router marked as unreachable Peer routers choose alternative paths Peers update their peers “Down” router recovers and tries to re- initiate peering sessions Large state dump transmissions are generated Increased load causes more routers to fail

28. 28 Route “flap storm” (cont.) zPossible solution: yHigher priority to Keep-Alive messages

29. 29 Possible impacts zIncrease in cache misses zCPU & memory problems zRoute “flap storm” zForwarding loops

30. 30 Forwarding loops zDefined as steady-state cyclic transmission of user data between a set of peers zLoop verification by checking ASPATH zUnconstrained routing policies

31. 31 BGP Update Types zWA Different – WADiff zAA Different – AADiff zWA Duplicate – WADup zAA Duplicate – AADup zWW Duplicate – WWDup

32. 32 BGP Update Types - WADiff zExplicit withdrawal zUnreachable route is replaced by alternative route zASPATH or next-hop attribute differs zForwarding instability

33. 33 BGP Update Types - AADiff zImplicit withdrawal zRoute is unreachable zAlternative path becomes available zForwarding instability

34. 34 WADiff and AADiff zWADiff yExplicit withdrawal yForwarding instability z AADiff yImplicit withdrawal yForwarding instability zRoute is replaced by alternative one

35. 35 BGP Update Types - WADup zExplicit withdrawal zRoute explicitly withdrawn and then re- announced a reachable zTransient topological problems (link or router) zForwarding instability or Pathological behavior

36. 36 BGP Update Types - AADup zImplicit withdrawal zRoute is implicitly withdrawn and replaced by it’s duplicate zDuplicate route does not differ in ASPATH or next-hop attribute information zPolicy fluctuations and Pathological behavior

37. 37 WADup and AADup zWADup yExplicit withdrawal yPathological behavior Forwarding instability z AADup yImplicit withdrawal yPathological behavior Policy fluctuations

38. 38 BGP Update Types - WWDup zRepeated BGP withdrawals for a prefix that is unreachable zPathological behavior

39. 39 BGP Update Types - Summary Explicit Withdrawal Implicit Withdrawal Forwarding instability Policy Fluctuations Patholo gical Behavi or WADiff V V AADiff VV WADup V V V AADup V VV WWDup – – V

40. 40 BGP Update Types

41. 41 WW Duplicate zTransmitted by routers of AS that never previously announced reachability for the withdrawn prefixes

42. 42 Let’s have a break

43. 43 Internet Routing Instability and it's Origins Ilia Ferdman Lilia Tsvetinovich

44. 44 Instability Origins zHardware configuration problems zSoftware bugs problems zMulti – Homing sites zBGP implementation problems

45. 45 Instability Origins – Hardware configuration zInternet growth -> Traffic growth -> New hardware need zOld Hardware -> Increase in number of updates : yCPU overload yLink failures zSmall Service Providers use old hardware

46. 46 Instability Origins – Hardware configuration zCache architecture yNot all prefix table in memory yIncrease in number of updates -> Increase in number of cache misses

47. 47 Instability Origins – Software bugs zUse of old or not configured software is the reason for Routing Instability zSmall Service Providers use old software

48. 48 Instability Origins – Multi – Homing sites zEnd-sites connect to Internet via multiple Service Providers(SP) zMulti-Homed customer prefixes require global visibility zRouters maintain longer prefixes SP1 SP2 SP3 Site

49. 49 Instability Origins – BGP implementation zStateless BGP yAnnouncements or withdrawals are send without check yO (N*U) additional updates xN – number of routers xU – number of updates zThere are better implementations

50. 50 Instability Origins – BGP implementation zMisconfigured interaction between different gateway protocols R1 BGP R1 BGP R2 OSPF R2 OSPF

51. 51 Possible solutions zRoute Server zRoute Dampening Algorithm zAggregation

52. 52 Possible solutions – Route Servers zR1,R2,R3,R4,R5 – routers zR.S. – rout server zR.S. collects BGP information from routers R3R3 R3R3 R2R2 R2R2 R5R5 R5R5 R4R4 R4R4 R1R1 R1R1

53. 53 Possible solutions – Route Servers zR1,R2,R3,R4,R5 – routers zR.S. – route server zR.S. collects BGP information from routers R3R3 R3R3 R2R2 R2R2 R5R5 R5R5 R4R4 R4R4 R1R1 R1R1 R. S.

54. 54 Possible solutions – Route Servers zDo not forward network traffic zPeer with service providers zProvide aggregate BGP information zUnique platform for statistic collection and monitoring

55. 55 Possible solutions – Route Dampening Algorithm z“Hold-down” frequent updates zAnnouncements about new networks delayed zDraconian version of enforcing stability

56. 56 Possible solutions – Aggregation zAggregation also calls supernetting zConcept of Aggregation: Several networks to supernetwork

57. 57 Possible solutions – Aggregation Network 1 Network 1 Network 3 Network 3 Network 2 Network 2 Network 4 Network 4

58. 58 Possible solutions – Aggregation Super network

59. 59 Possible solutions – Aggregation (cont.) zAdvantage: ydecrease in number of global visible addresses ydecrease in number of updates zProblems: yNo correlation between Service Providers yMulti – Homing sites prevent aggregation

60. 60 Statistic zDefault – free table size : 45.000 prefixes zNumber of updates per day : between 3 to 6 millions z99 percent of routing information is pathologic zOnly 10 percent of routers send one or less WADiff per day zOnly 20 percent of routers send one or less AADiff per day

61. 61 Statistic – updates per period from 04/96 to 10/96

62. 62 Statistic – updates per week

63. 63 Results zToo many BGP updates exchanged zPathological updates are dominated zDaily and weekly cyclic trends zInstability happens to everybody zForwarding instability the main contributor

64. 64 Statistic – updates from 96 to 98

65. 65 Things that you don’t need to know about Routing Instability zRouting Instability is rapid fluctuation of network reachability and topology information zThere are three types of Instability: forwarding instability, policy fluctuation, pathological updates

66. 66 Things that you don’t need to know about Routing Instability (cont.) zInstability can lead to many unpleasant things such as yIncreased packet loss yDelays in the time for network convergence yAdditional resource overhead(memory, CPU)

67. 67 Things that you don’t need to know about Routing Instability (cont.) zThe possible origins of routing instability are yRouter configuration errors yTransient physical and data link problems ySoftware bugs

68. 68 Things that you don’t need to know about Routing Instability (cont.) zThere are several solutions: yRoute Server yRoute Dampening Algorithm yAggregation

69. 69 The END