1. Interdomain Issues for IP networks Henning Schulzrinne (with lots of borrowed slides...)

2. 19 June 20152 Overview  Architecture review  Interdomain routing  Multicast  VPNs  Interdomain QoS –signaling –charging and settlements  Interdomain application signaling  Carrier selection and multihoming

3. 19 June 20153 Architecture review  Classical view of ISP food chain  Tier-1, tier-2

4. Inter-regional Internet backbone 357 Mbit/s 19’716 Mbit/s Asia- Pacific Latin America & Caribbean 2’638 Mbit/ s 127 Mbit/s Arab States, Africa 468 Mbit/s 171 Mbit/s Europe 56’241 Mbit/s USA & Canada Source: TeleGeography Inc., Global Backbone Database. Data valid for Sept. 2000.

5. 19 June 20155 Examples of carriers  Tier 1: UUNet, Cable & Wireless (C&W), Sprint, Qwest, Genuity, AT&T  Tier 2: America Online, Broadwing, @home  Tier 3: RCN, Verizon, Log On America

6. 19 June 20156 Definitions  Peering: exchange of data between ISPs on a sender-keeps-all basis  Access provider (IAP): provide dial-up and leased line access, buy Internet access from tier-1/2 providers  Transit: Using ISP A to reach customers of ISP B, C,...  Hot potato routing: find earliest exit point to destination network  asymmetric routes

7. 19 June 20157 NAP or IXPs Internet eXchange Points  "An Internet Exchange (IX) acts as a junction between multiple points of Internet presence. Here, peers are able to directly connect to each other to exchange local Internet traffic. Typically, the IX owns and operates the switching platforms used to interconnect the various users/subscribers."  Also known as Metropolitan Area Exchanges (MAEs)  see http://www.telegeography.com/ix/http://www.telegeography.com/ix/  governed by Multi-Lateral Peering Agreements (MLPA)

8. 19 June 20158 Some European IXPs Austria - The Vienna Internet eXchange (VIX) Belgium - Belnet (BNIX) Cyprus - The Cyprus Internet eXchange (CyIX) Denmark - Danish Internet eXchange (DIX) Lyngby Finland - Finnish Commercial Internet eXchange (FCIX) Helsinki) France - Paris Internet eXchange (PARIX) France - French Global Internet eXchange (SFINX) Germany - The Deutsche Central Internet eXchange (DE-CIX) Frankfurt Greece - The Athens Internet eXchange (AIX) Ireland - The Internet Neutral eXchange (INEX) Italy - The Milan Internet eXchange (MIX) Italy - NAP Nautilus (CASPUR) Luxembourg - The Luxembourg Internet eXchange (LIX) Netherlands - The Amsterdam Internat eXchange (AMS-IX) Norway - Norwegian Internet eXchange (NIX) Portugal - The Portuguese Internet eXchange (PIX) Scotland - Scottish Internet Exchange (ScotIX) Spain - El Punto Neutral Espanol (ESPANIX) Sweden - The Netnod Internet eXchange (D-GIX) Switzerland - The Swiss Internet eXchange (SIX) Switzerland - Geneva Cern (CIXP) Switzerland - Zürich Telehouse Internet Exchange (TIX) United Kingdom - The London INternet eXchange (LINX) United Kingdom - Manchester Network Access Point (MaNAP) United Kingdom - London Network Access Point (LoNAP) Bulgaria - The Sofia Internet eXchange (SIX - GoCIS) Czech Rep. - Neutral Internet eXchange (NIX) Prague Latvia - The Global Internet eXchange (GIX) LatNet Romania - The Bucharest Internet eXchange (BUHIX) Slovakia - The Slovak Internet eXchange (SIX) Ukraine - The Central Ukrainian Internet eXchange

9. 19 June 20159 The view from elsewhere  Looking glass sites show BGP routes:

10. 19 June 201510 RADB $ whois -h whois.radb.net AS14 aut-num: AS14 as-name: COLUMBIA descr: Columbia University in the City of New York Network Operations Academic Information Systems 612 West 115th Street New York, NY 10025 admin-c: CU-NOC tech-c: CU239-ORG import: from AS1785 action med=100; # ApTh commodity accept ANY import: from AS701 action med=200; # UUnet commodity accept ANY import: from AS14:AS-ISPPEERS action pref=10; # private ISP peers accept import: from AS14:AS-NNPEERS action pref=10; # private NN peers accept import: from AS145 action med=75; # vBNS I2 accept ANY AND NOT {0.0.0.0/0} import: from AS11537 action med=50; # Abilene I2 accept ANY AND NOT {0.0.0.0/0} import: from AS3754 action med=100; # NYSERNet I2 accept AND NOT {0.0.0.0/0}

11. 19 June 201511 QoS  Interdomain SLAs are rare (or non- existent)  Large difference between inter- and intradomain performance?

12. 19 June 201512 Interdomain QoS Issues  Request authentication  Uniform service levels – my "gold" is your "bronze"...  Payment –NJ Turnpike? –Gardenstate Parkway?

13. 19 June 201513 Interdomain QoS metric

14. 19 June 201514 Carrier selection  Allow selection of carrier  Easy for multi-homed sites  but everything else requires loose source route – but what IP address?  will work in both directions

15. 19 June 201515 Interdomain multicast  Any-source multicast (ASM) has many operational problems: –PIM-SM/DM are only intradomain –PIM-SM complex –RP has scaling and reliability problems –interdomain never got off the ground –no deployed multicast address allocation mechanism –spam problem – anybody can send to group

16. 19 June 201516 Interdomain multicast  Single-source multicast (SSM) –source-filtered  IGMPv3 –{S,G} as group –avoid address allocation –match many applications: Internet radio/TV conferences with single active source

17. 19 June 201517 Distributed Denial of Service (DDOS)  Need packet tracing (in progress)  Need push-back to filter DOS stream –at source –or close to source  Authentication of filter request to prevent malicious blackouts

18. 19 June 201518 Settlements  = payments between providers  long history in telephone network  e.g., 4.6b US$ in 2000 net settlements

19. 19 June 201519 Total Accounting Rate (TAR)  Traditional conceptual cost of connecting a call from country A to country B  Each end contribute the building cost of half circuit to the midpoint  Based on “cost” of early tiny capacity submarine cable  Settlement A to B at 1/2 TAR S. Cheng/ITU

20. 19 June 201520 Total Accounting Rate (TAR) - cont’d  Same Rate for the opposite direction  Apply to all PSTN services  When the accounting rate change in one direction the other direction must follow S. Cheng/ITU

21. 19 June 201521 Termination Rate  Usually based on cost of terminating call by destination carrier  Accounting rate may not be the same for the other direction  Accounting rate in each direction can change independent of each other  May deliver traffic at mid-point or FOB on either end of circuit. S. Cheng/ITU

22. 19 June 201522 Sender keeps all (Peering)  Sender keeps all revenue from calling party  No settlement between carriers  Applicable if average cost and traffic volume are virtually identical in each direction  Usually based on half circuit ownership S. Cheng/ITU

23. 19 June 201523 US domestic telephony settlements  Doesn't quite fit SKA  Long-distance company collects  Pays fixed charge/minute to originating and terminating local exchange carrier (LEC)

24. Where does the money go? Typical US ISP cash-flow $19.95 per month subscription $7.50-$10.50 Wholesale PoP Access $2.00 - $3.00 Customer Care $3.00 amortised customer marketing $3.50-$7.50 margin per customer Source: Adapted from Paul Stapleton, ISP$ Market Report, Boardwatch Magazine.

25. Settlements-based traffic PTO A Collects revenues Collects traffic PTO B Retains revenues Terminates traffic Delivers traffic Pays settlement fees User 1User 2User 3 User 1 User 2User 3 For accounting rate traffic, a direct bilateral relationship is established between the origin and termination operators. Intermediate transit operators are compensated from the accounting rate which is usually split 50:50. PTO B retains net settlement. ……... PTO = Public Telecommunications Operator PTOs A & B split the cost of the int’l circuit

26. Internet Peering traffic (Web) ISP A Exchanges traffic ISP B Collects revenues Requests and terminates traffic One-way (thick pipe) User 1 User 2User 3 For Internet Peering traffic, ISP B pays for both halves of the International circuit(s) which are used for peering with ISP A. ISP B also pays for traffic exchange. ISP B may pay for the circuit directly, or in conjunction with one or more PTOs. ISP = Internet Services Provider PTO B pays the full cost of the int’l circuit Two-way (thin pipe) Web 1

27. 19 June 2015 Settlements and Peering: What’s the difference?  Settlement-payment traffic –Substantial revenue transfers, from core to periphery of network –Promotes “organic” network growth –So, Operators generating less traffic than they receive have an incentive to keep prices high  Peering traffic –Some revenue transfers, from periphery to core of network –Promotes “spontaneous” network growth –So, ISPs generating less traffic than they receive have an incentive to force prices down

28. Internet traffic flows are highly asymmetric Public switched telephone service Traffic flows are bilateral and broadly match value flow in that caller, who initiates the call, also pays for it Call-back reverses the direction of the call, from a statistical viewpoint, but caller still pays & benefits Traffic flows unbalanced between developed and developing countries Public Internet service Traffic flows are multi- lateral: A single session may poll many countries Web-browsing is dominant form of traffic: traffic flow is dominantly towards user who initiates the call. Web traffic highly asymmetric Newer forms of Internet traffic (telephony, push media, streaming video etc) reverses traffic flow to be from user which initiates the call

29. 19 June 201529 Interdomain AAA  Roaming user identified by NAI (RFC 2486), e.g., alice@example.com  Find DIAMETER AAA via NAPTR + SRV  Generic problem different: –User Alice@A from ISP A visits ISP B –ISP B needs to determine whether Alice is a valid customer of A –Alice needs to authorize B to query A –Needs to get authorization for maximum € amount –  very similar to credit card authorization!

30. 19 June 201530 Clearinghouse models  e.g., iPass or GRIC for roaming dial-up and wireless users –member company charges subscribers –gets access to other dial-up ports via clearinghouse –gets reimbursed for "visitors"  GSM roaming is not a good model –no price transparency –inefficient routing

31. 19 June 201531 SIP interdomain  Designed to find proxies by request URI  Authentication and anonymity are issues: –how can callee ascertain identity of random caller? –how can caller know that she's talking to the right person? –trust provider to remove privacy- compromising information

32. 19 June 201532 BGP problems  Trust  Need route filtering: In April 1997, a small ISP in Florida made a mistake in configuring the router that joined its small network to Sprint. This ISP, known as AS number 7007, allowed all the routes it learned from Sprint using BGP to be exported back to Sprint as its own routes. This is easy to do, because BGP implementations can take routes from IGP and convert them into EGP routes. In this case, the IGP converted CIDR routes into classful routes. The Sprint BGP speaker wasn't filtering properly either and began sending out updates that added AS7007 as the correct route for a portion of every CIDR block (essentially, the first class C, 24-bit-long network prefix). This misinformation first spread through Sprint's network, then to neighboring NSPs, including ANS, MCI, UUNet, and others. Many routers crashed because their routing tables suddenly doubled in size (an additional route was added for each CIDR block), and the routing instability spread throughout the Internet. Remember that, when a router crashes, it drops its BGP connection with its peer, which then sends out an update withdrawing all the routes announced previously by the crashed router. (Network Magazine, March 2002)

33. 19 June 201533 Alternatives to improving routing  "Resilient Overlay Networks" (Andersen/Balakrishnan/Kashoek/Morris 2001) –application-layer routing with one hop  Multihoming: –treat networks like cheap PCs –99.5% reliability²  99.9975% reliability

34. 19 June 201534 Multihoming problems  Need either an ASN or two IP address ranges  Only for larger networks  don't allow advertisements for /24  Network impact: two /22 entries for each subnet  Alternative: NAT –doesn't help reachability of servers advertised in DNS

35. BGP Issues Geoff Huston

36. 19 June 201536 Why measure BGP?  BGP describes the structure of the Internet, and an analysis of the BGP routing table can provide information to help answer the following questions: –What is changing in the deployment environment? –Are these changes sustainable? –How do address allocation policies, BGP and the Internet inter-relate? –Are current address allocation policies still relevant? –What are sensible objectives for address allocation policies?

37. 19 June 201537 Techniques  Passive Measurement –Takes measurements from a default-free router at the edge of the local network –Easily configured –Single (Filtered) view of the larger Internet What you see is a collection of best paths from your immediate neighbours Local AS eBGP Measurement Point

38. 19 June 201538 Techniques  Multiple Passive measurement points –Measure a number of locations simultaneously –Can be used to infer policy AS3 Measurement Points AS2 AS1

39. 19 June 201539 Techniques  Single passive measurement point with multiple route feeds –Best example: Route-views.oregon-ix.net Operating since 1995 42 peers Uses eBGP multihop to pull in route views

40. 19 June 201540 Techniques  Active Measurement Tests –Convergence Announcement and withdrawal Monitoring Unit AS2 AS1 Reporting Points Route Injection Point Internet

41. 19 June 201541 Interpretation  BGP is not a link state protocol  There is no synchronized overview of the entire connectivity and policy state  Every BGP viewing point contains a filtered view of the network –Just because you can’t see it does not mean that it does not exist  BGP metrics are sample metrics

42. BGP Table Growth BGP Table Growth – 12 year history

43. BGP Table Growth – 2 year history

44. BGP Table Growth – 2 year & 6 month trends

45. BGP Table Growth – Projections

46. Prefix distribution in the BGP table

47. /24 is the fastest growing prefix length

48. /25 and smaller are the fastest growing prefixes in relative terms

49. 19 June 201549 Prefixes by AS  Distribution of originating address sizes per AS  Address advertisements are getting smaller Prefix Length Number of AS’s Non-Hierarchical Advertisements

50. 19 June 201550 Multi-homing on the rise?  Track rate of CIDR “holes” – currently 41% of all route advertisements are routing ‘holes” This graph tracks the number of address prefix advertisements which are part of an advertised larger address prefix

51. Proportion of BGP advertisements which are more specific advertisements of existing aggregates

52. 19 June 201552 OOPS  Program bug! The number is larger than that.  More specific advertisement of existing aggregates account for 54% of the BGP selected route table from the perspective of AS1221 –56,799 entries from a total of 103,561  Older (mid Jan) data from AS286 has the number at 53,644 from a total of 95,036 (56%)

53. 19 June 201553 Routed Address Space Large fluctuation is due to announcement / withdrawals of /8 prefixes 12 months of data does not provide clear longer growth characteristic

54. 19 June 201554 Routed Address Space (/8 Corrected) Annual compound growth rate is 7% p.a. Most address consumption today appears to be ocurring behind NATs /8 Corrected Data

55. 19 June 201555 AS Number Growth

56. 19 June 201556 AS Number Use - Extrapolation Continued exponential growth implies AS number exhaustion in 2005

57. 19 June 201557 Average size of a routing table entry The BGP routing tale is growing at a faster rate than the rate of growth of announced address space /18.1 /18.5

58. 19 June 201558 Denser Internet Structure Dec-2000 Feb-2001 AS Hops Reachable Addresses

59. 19 June 201559 Denser Internet Structure AS Hops Address Span Feb-2001 Dec-2000 90% point

60. 19 June 201560 Internet ‘Shape’ Distance Span Distance Span  The network is becoming less ‘stringy’ and more densely interconnected –i.e. Transit depth is getting smaller

61. 19 June 201561 Aggregation and Specifics  Is the prevalence of fine-grained advertisements the result of deliberate configuration or inadvertent leakage of advertisements?

62. 19 June 201562 Publicity helps ?  Efforts to illustrate the common problem of unconstrained table growth appear to have had an impact on growth of the table, as seen on the edge of AS1221 since Dec 2000

63. 19 June 201563 But - the view from KPNQwest Data from James Aldridge, KPNQwest - http://www.mcvax.org/~jhma/routing/

64. 19 June 201564 Different Views

65. 19 June 201565 Different Views  Route views in prefix-length-filtered parts of the net do not show the same recent reduction in the size of the routing table.  It is likely that the reduction in routes seen by AS1221 appears to be in the prefix-length filtered ranges –Either more transit networks are prefix length filtering or origin AS’s are filtering at the edge, or both  The underlying growth trend in BGP table size remains strong

66. 19 June 201566 Aggregation possibilities  What if all advertisements were maximally aggregated* ? –27% reduction (103126 -> 74427) using AS Path aggregation –33% reduction (103126 -> 68504) using AS Origin aggregation This assumes that the specific advertisements are not matched by other specific advertisements which have been masked out closer to the origin AS – this is not a terribly good assumption, so these numbers are optimistic to some extent

67. 19 June 201567 Aggregation Potential from AS1221 AS Origin AS Path

68. 19 June 201568 The aggregation potential view from KPNQwest Data from James Aldridge, KPNQwest - http://www.mcvax.org/~jhma/routing/ AS Origin AS Path

69. 19 June 201569 A Longer Term View from AS286

70. 19 June 201570 Different Views  Similar AS Origin, but different AS Path aggregation outcomes  Prevalence of the use of specifics for local inter-domain traffic engineering

71. 19 June 201571 Aggregatability?  A remote view of aggregation has two potential interpretations: –Propose aggregation to the origin AS –Propose a self-imposed proxy aggregation ruleset  Any aggregation reduces the information content in the routing table. Any such reduction implies a potential change in inter- domain traffic patterns.  Aggregation with preserved integrity of traffic flows is different from aggregation with potential changes in traffic flow patters

72. 19 June 201572 Aggregatability  Origin AS aggregation is easier to perform at the origin, but harder to determine remotely IF traffic flows are to be preserved  Proxy Aggregation is only possible IF you know what your neighbors know Yes this is a recursive statement –If an AS proxy aggregates will it learn new specifics in response?

73. 19 June 201573 BGP as a Routing Protocol  How quickly can the routing system converge to a consistent state following dynamic change?  Is this time interval changing over time?

74. 19 June 201574 Increased convergence time intervals for BGP  Measured time to withdraw route: –Up to 2 minutes  Measured time to advertise new route: –Up to 30 minutes

75. 19 June 201575 What is happening here? How long until routes return? (From A Study of Internet Failures)

76. 19 June 201576 Withdraw Convergence AS1 AS2 AS3 AS4

77. 19 June 201577 Withdraw Convergence  Probability distribution  Providers exhibit different, but related convergence behaviors  80% of withdraws from all ISPs take more than a minute  For ISP4, 20% withdraws took more than three minutes to converge

78. 19 June 201578 Failures, Fail-overs and Repairs

79. 19 June 201579 Failures, Fail-overs and Repairs  Bad news does not travel fast…  Repairs (Tup) exhibit similar convergence properties as long-short ASPath fail-over  Failures (Tdown) and short-long fail-overs (e.g. primary to secondary path) also similar –Slower than Tup (e.g. a repair) –60% take longer than two minutes –Fail-over times degrade the greater the degree of multi-homing!

80. 19 June 201580 Conjectures…. BGP table size will continue to rise exponentially  Multi-homing at the edge of the Internet is on the increase  The interconnectivity mesh is getting denser –The number of AS paths is increasing faster than the number of AS’s –Average AS path length remains constant  AS number deployment growth will exhaust 64K AS number space in August 2005 if current growth trends continue

81. 19 June 201581 More conjecturing….  Inter-AS Traffic Engineering is being undertaken through routing discrete prefixes along different paths -- globally (the routing mallet!) –AS Origin aggregation < AS Path aggregation  RIR allocation policy (/19, /20) is driving one area of per-prefix length growth in the aggregated prefix area of the table  BUT - NAT is a very common deployment tool –NAT, multihoming and Traffic Engineering is driving even larger growth in the /24 prefix area

82. 19 June 201582 And while we are having such a good time conjecturing…  Over 12 months average prefix length in the table has shifted from /18.1 to /18.5  More noise (/25 and greater) in the table, but the absolute level of noise is low (so far)  Most routing table flux is in the /24 to /32 prefix space – as this space gets relatively larger so will total routing table flux levels –“Flux” here is used to describe the cumulative result of the withdrawals and announcements –This space appears to be susceptible to social pressure – at present

83. 19 June 201583 This is fun – lets have even more conjectures…  CIDR worked effectively for four years, but its effective leverage to support long term dampened route table growth and improved table stability has now finished  Provider-based service aggregation hierarchies as a model of Internet deployment structure is more theoretic than real these days i.e. provider based route aggregation is leaking like a sieve!

84. 19 June 201584 Commentary  draft-iab-bgparch-00.txt –Exponential growth of BGP tables has resumed –AS number space exhaustion –Convergence issues –Traffic Engineering in a denser mesh –What are the inter-domain routing protocol evolutionary requirements?

85. 19 June 201585 Objectives and Requirements  Supporting a larger and denser interconnection topology  Scale by x100 over current levels in number of discrete policy entities  Fast Convergence  Security  Integration of Policy and Traffic Engineering as an overlay on basic connectivity  Control entropy / noise inputs

86. 19 June 201586 Available Options  Social Pressure on aggregation  Economic Pressure on route advertisements  Tweak BGP4 behavior  Revise BGP4 community attributes  BGPng  New IDR protocol(s)  New IP routing architecture

87. 19 June 201587 Social Pressure  Social pressure can reduce BGP noise  Social pressure cannot reduce pressures caused by –Denser interconnection meshing –Increased use of multi-homing –Traffic engineering of multiple connections  Limited utility and does not address longer term routing scaling

88. 19 June 201588 Economic Pressure on Routing  Charge for route advertisements –Upstream charges a downstream per route advertisements –Peers charge each other  This topic is outside an agenda based on technology scope  Raises a whole set of thorny secondary issues: –Commercial –National Regulatory –International  Such measures would attempt to make multi-homing less attractive economically. It does not address why multi-homing is attractive from a perspective of enhanced service resilience.

89. 19 June 201589 Tweaking BGP4  Potential tweak to BGP-4 –Auto-Proxy-Aggregation Automatically proxy aggregate bitwise aligned route advertisements Cleans up noise – but reduces information Cannot merge multi-homed environments unless the proxy aggregation process makes sweeping assumptions, or unless there is an overlay aggregation protocol to control proxy aggregation (this is then no longer a tweak)

90. 19 June 201590 Extend BGP4 Communities  We already need to extend community attributes to take on the 2 / 4 octet AS number transition.  Can we add further community attribute semantics to allow proxy aggregation and proxy sublimation under specified conditions?  Extend commonly defined transitive community attributes to allow further information to be attached to a routing advertisement –Limit of ‘locality’ of propagation –Aggregation conditions or constraints  If we could do this, will this be enough? Can this improve –Scaling properties –convergence properties

91. 19 June 201591 BGPng  Preserve: AS concept, prefix + AS advertisements, distance vector operation, AS policy “opaqueness”  Alter: convergence algorithm (DUAL?), advertisement syntax (AS + prefix set + specifics + constraints), BGP processing algorithm  Issues: –Development time –Potential to reach closure on specification –Testing of critical properties –Deployment considerations –Transition mechanisms

92. 19 June 201592 IDR  A different IDR protocol? –Can we separate connectivity maintenance, application of policy constraints and sender- and/or receiver- managed traffic engineering? SPF topology maintenance Inter-Domain Policy Protocol to communicate policy preferences between policy “islands” Multi-domain path maintenance to support traffic engineering requirements –Eliminate the need to advertise specifics to undertake traffic engineering –Multi-homing may still be an issue – is multi-homing a policy issue within an aggregate or a new distinct routing “entity”? –Can SPF scale? Will SPF routing hierarchies impose policy on the hierarchy elements?

93. 19 June 201593 New IP Routing Architecture  Separate Identity, Location and Path at an architectural level?  Identity –How do you structure an entirely new unique identity label space? How do you construct the “identity lookup” mechanism?  Location –How can location be specified independent of network topology?  Path: –Is multi-homing an internal attribute within the network driven by inter-domain policies, or is multi-homing an end-host switching function

94. 19 June 201594 New IP Routing Architecture  Other approaches? –Realms and RSIP –Inter-Domain CRLDP approaches where policy is the constraint

95. 19 June 201595 Slide credits  Geoff Huston  Tim Kelly, ITU  http://www.itu.int/ITU-D/ict/papers/