1. Amogh Dhamdhere Provider and Peer Selection in the Evolving Internet Ecosystem Committee: Dr. Constantine Dovrolis (advisor) Dr. Mostafa Ammar Dr. Nick Feamster Dr. Ellen Zegura Dr. Walter Willinger (AT&T Labs-Research)

2. The Internet Ecosystem 27,000 autonomous networks independently operated and managed The “Internet Ecosystem” Different types of networks Interact with each other and with “environment” Network interactions Localized, in the form of interdomain links Competitive (customer-provider), symbiotic (peering) Distributed optimizations by each network Select providers and peers to optimize utility function The Internet ecosystem evolves 8/19/2015 2

3. High Level Questions How does the Internet ecosystem evolve? What is the Internet heading towards? Topology Economics Performance How do the strategies used by networks affect their utility (profits/costs/performance)? How do these individual strategies affect the global Internet? 8/19/2015 3

4. Previous Work Static graph properties No focus on how the graph evolves “Descriptive” modeling Match graph properties e.g. degree distribution Homogeneity Nodes and links all the same Game theoretic, computational Restrictive assumptions Dynamics of the evolving graph Birth/death Rewiring “Bottom-up” Model the actions of individual networks Heterogeneity Networks with different incentives Semantics of interdomain links 8/19/2015 4

5. Our Approach “Measure – Model – Predict” 8/19/2015 5 Measure the evolution of the Internet Ecosystem Topological, but focus on business types and rewiring Model strategies and incentives of different network types Predict the effects of provider and peer selection strategies

6. Outline Measuring the Evolution of the Internet Ecosystem The Core of the Internet: Provider and Peer Selection for Transit Providers The Edge of the Internet: ISP and Egress Path Selection for Stub Networks ISP Profitability and Network Neutrality 8/19/2015 6 [IMC ’08] [to be submitted] [Netecon ’08] [Infocom ‘06]

7. Motivation How did the Internet ecosystem evolve during the last decade? Is growth more important than rewiring? Is the population of transit providers increasing or decreasing? Diversification or consolidation of transit market? Given that the Internet grows in size, does the average path length also increase? Where is the Internet heading? 8/19/2015 7

8. Approach Focus on Autonomous Systems (ASes) As opposed to networks without AS numbers Start from BGP routes from RouteViews and RIPE monitors during 1997-2007 Focus on primary links Classify ASes based on their business function Enterprise ASes, small transit providers, large transit providers, access providers, content providers Classify inter-AS relations as “transit” and “peering” Transit link – Customer pays provider Peering link – No money exchanged 8/19/2015 8 Visibility Issue

9. Internet growth Number of CP links and ASes showed initial exponential growth until mid-2001 Followed by linear growth until today Change in trajectory followed stock market crash in North America in mid-2001 8/19/2015 9

10. Path lengths stay constant Number of ASes has grown from 5000 in 1998 to 27000 in 2007 Average path length remains almost constant at 4 hops 8/19/2015 10

11. Rewiring more important than growth Most new links due to internal rewiring and not birth (75%) Most dead links are due to internal rewiring and not death (almost 90%) 8/19/2015 11

12. Classification of ASes based on business function Four AS types: Enterprise customers (EC) Small Transit Providers (STP) Large Transit Providers (LTP) Content, Access and Hosting Providers (CAHP) Based on customer and peer degrees Classification based on decision-trees 80-85% accurate 8/19/2015 12

13. Evolution of AS types LTPs: constant population (top-30 ASes in terms of customers) Slow growth of STPs (30% increase since 2001) EC and CAHP populations produce most growth Since 2001: EC growth factor 2.5, CAHP growth factor 1.5 8/19/2015 13

14. Multihoming by AS types CAHPs have increased their multihoming degree significantly On the average, 8 providers for CAHPs today Multihoming degree of ECs almost constant (average < 2) Densification of the Internet occurs at the core 8/19/2015 14

15. Conjectures on the Evolution of peering Peering by CAHPs has increased significantly CAHPs try to get close to sources/destinations of content 8/19/2015 15

16. Conclusions Where is the Internet heading? Initial exponential growth up to mid-2001, followed by linear growth phase Average path length practically constant Rewiring more important than growth Need to classify ASes according to business type ECs contribute most of the overall growth Increasing multihoming degree for STPs, LTPs and CAHPs Densification at the core CAHPs are most active in terms of rewiring, while ECs are least active 16 8/19/2015

17. Outline Measuring the Evolution of the Internet Ecosystem The Core of the Internet: Provider and Peer Selection for Transit Providers The Edge of the Internet: ISP and Egress Path Selection for Stub Networks ISP Profitability and Network Neutrality 8/19/2015 17

18. Modeling the Internet Ecosystem From measurements: Significant rewiring activity Especially by transit providers Networks rewire connectivity to optimize a certain objective function Distributed Localized spatially and temporally Rewiring by changing the set of providers and peers What are the global, long-term effects of these distributed optimizations? Topology and traffic flow Economics Performance (path lengths) 8/19/2015 18

19. The Feedback Loop 8/19/2015 19 When does it converge? When no network has the incentive to change its connectivity – “steady-state” Interdomain TM Traffic flow Interdomain topology Per-AS profit Cost/price parameters Routing Provider selection Peer selection

20. Impact of provider/peer selection strategies 8/19/2015 20 CC S SS C C P1 P2 P3 S No peering P2 and P3 peer P1 open to peering with CPs

21. Our Approach What is the outcome when networks use certain provider and peer selection strategies? Model the feedback loop in the Internet ecosystem Real-world economics of transit, peering, operational costs Realistic routing policies Geographical constraints Provider and peer selection strategies Computationally find a “steady-state” No network has further incentive to change connectivity Measure properties of the steady-state Topology, traffic flow, economics 8/19/2015 21

22. Network Types Enterprise Customers (EC) Stub networks at the edge (e.g. Georgia Tech) Either sources or sinks Small Transit Providers (STP) Provide Internet transit Mostly regional in presence (e.g. France Telecom) Large Transit Providers (LTP) Transit providers with global presence (e.g. AT&T) Content Providers (CP) Major sources of content (e.g. Google) 8/19/2015 22 Provider and peer selection for STPs and LTPs

23. What would happen if..? The traffic matrix consists of mostly P2P traffic? P2P traffic benefits STPs, can make LTPs unprofitable LTPs peer with content providers? LTPs could harm STP profitability, at the expense of longer end-to-end paths Edge networks choose providers using path lengths? LTPs would be profitable and end-to-end paths shorter 8/19/2015 23

24. Provider and Peer Selection Provider selection strategies Minimize monetary cost (PR) Minimize AS path lengths weighted by traffic (PF) Avoid selecting competitors as providers (SEL) Peer selection strategies Peer only if necessary to maintain reachability (NC) Peer if traffic ratios are balanced (TR) Peer by cost-benefit analysis (CB) Peer and provider selection are related 8/19/2015 24

25. Provider and Peer Selection are Related 8/19/2015 25 A A C C B B U U B B U U A A C C Peering by necessity Level3-Cogent peering dispute Restrictive peering A A C C B B ?X

26. Economics, Routing and Traffic Matrix Realistic transit, peering and operational costs Transit prices based on data from Norton Economies of scale BGP-like routing policies No-valley, prefer customer, prefer peer routing policy Traffic matrix Heavy-tailed content popularity and consumption by sinks Predominantly client-server: Traffic from CPs to ECs Predominantly peer-to-peer: Traffic between ECs 8/19/2015 26

27. Algorithm for network actions Networks perform their actions sequentially Can observe the actions of previous networks And the effects of those actions Network actions in each move Pick set of preferred providers Attempt to convert provider links to peering links “due to necessity” Evaluate each existing peering link Evaluate new peering links Networks make at most one change to their set of peers in a single move 8/19/2015 27

28. Solving the Model Determine the outcome as each network selects providers and peers according to its strategy Too complex to solve analytically: Solve computationally Typical computation Proceeds iteratively, networks act in a predefined sequence Pick next node n to “play” its possible moves Compute routing, traffic flow, AS fitness Repeat until no player has incentive to move “steady-state” or equilibrium 8/19/2015 28

29. Properties of the steady-state Is steady-state always reached? Yes, in most cases Is steady-state unique? No, can depend on playing sequence Different steady-states have qualitatively similar properties Multiple runs with different playing sequence Average over different runs Confidence intervals are narrow 8/19/2015 29

30. Canonical Model Parameterization of the model that resembles real world Traffic matrix is predominantly client-server (80%) Impact of streaming video, centralized file sharing services 20% of ECs are content sources, 80% sinks Heavy tailed popularity of traffic sources Edge networks choose providers based on price 5 geographical regions STPs cheaper than LTPs 8/19/2015 30

31. Model Validation Reproduces almost constant average path length Activity frequency: How often do networks change their connectivity? ECs less active than providers – Qualitatively similar to measurement results 8/19/2015 31

32. Results – Canonical Model 8/19/2015 32 EC LTP S1 S2 CP EC Hierarchy of STPs Traffic can bypass LTPs – LTPs unprofitable STPs should not peer with CPs Resist the temptation!

33. Results – Canonical Model 8/19/2015 33 LTP S1 S2 CP EC What-if: LTPs peer with CPs Generate revenue from downstream traffic Can harm STP fitness Long paths

34. Deviation 1: P2P Traffic matrix 8/19/2015 34 LTP S1 S2 CP EC P2P traffic helps STPs Smaller traffic volume from CPs to Ecs More EC-EC traffic => balanced traffic ratios More opportunities for STPs to peer Peering by “traffic ratios” makes sense LTP S1 S2 EC S3 EC

35. Conclusions A model that captures the feedback loop between topology, traffic and fitness in the Internet Considers effects of Economics Geography Heterogeneity in network types Predict the effects of provider and peer selection strategies Topology, traffic flow, economics, and performance 8/19/2015 35

36. Outline Measuring the Evolution of the Internet Ecosystem The Core of the Internet: Peer and Provider Selection for Transit Providers The Edge of the Internet: ISP and Egress Path Selection for Stub Networks ISP Profitability and Network Neutrality 8/19/2015 36

37. The Edge of the Internet Sources and sinks of content Content Providers (CP): sources Enterprise Customers (EC): sinks From measurements: ECs connect increasingly to STPs Cost conscious ? CPs connect increasingly to LTPs Performance ? Increasing multihoming (about 60% of stubs) Redundancy, load balancing, cost effectiveness How should stub networks choose their providers? 8/19/2015 37

38. Major Questions How to select the set of upstream ISPs ? Low monetary cost Short AS paths to major destinations Path diversity to major traffic destinations – robustness to network failures How to allocate egress traffic to the set of selected ISPs ? Objective: Avoid congestion on the upstream paths Also maintain low cost 8/19/2015 38

39. ISP Selection Select k ISPs out of N Let C be a subset of k ISPs out of N Total cost of a selection of ISPs C: Weighted sum of monetary, path length and path diversity costs Select combination C with minimum total cost Feasible to enumerate all combinations 8/19/2015 39

40. Monetary and Path Length Cost For set of ISPs C, what is the monetary and path length cost of routing egress flows? Find the minimum cost mapping G* of flows to ISPs (Bin Packing) Flows = items ISPs = bins NP hard ! Use First Fit Decreasing (FFD) heuristic Mapping G* very close to optimal Monetary and path length costs of C are calculated using the mapping G* 8/19/2015 40

41. Path Diversity selection C gives K paths to each destination d K-shared link to d: link shared by all K paths to d If a K-shared link fails, destination d is unreachable Minimize the number of K- shared links Path diversity metric: The number of k-shared links to destination d averaged over all destinations Gives the best resiliency to single-link failures 8/19/2015 41

42. Summary Algorithms for ISP selection Choosing best set of upstream ISPs Objectives are minimum monetary cost, short AS paths and high path diversity ISP selection for monetary and performance constraints Formulated as a bin-packing problem Heuristic gives solution very close to optimal ISP selection for path diversity Returns set of ISPs with best path diversity to the set of major destinations 8/19/2015 42

43. Outline Measuring the Evolution of the Internet Ecosystem The Edge of the Internet: ISP and Egress Path Selection for Stub Networks The Core of the Internet: Peer and Provider Selection for Transit Providers ISP Profitability and Network Neutrality 8/19/2015 43

44. The debate Recent evolution trend: Large amounts of video and peer-to-peer traffic Content providers (CP) generate the content Provide content and services “over the top” of the basic connectivity provided by ISPs Profitable (think Google) Access Providers (AP) deliver content to users Recent trend: Not profitable Commoditization of basic Internet access Want a share of the pie Tension between AP and CPs: “Network neutrality” 8/19/2015 44

45. A Technical View Previous work Mostly non-technical Highly emotional debates in the press Legislation/policy aspects: Do we need network neutrality legislation? But what about the underlying problem: Non- profitability of Access Providers? Our approach: A quantitative look at AP profitability Investigate reasons for non-profitability Evaluate strategies for remaining profitable 8/19/2015 45

46. Modeling AP Profitability Three AS types: AP, CP and transit provider (TP) Focus on the AP AS links customer-provider (customer pays provider) peering (no payments) AP and CPs can transfer traffic either through customer-provider or peering links 8/19/2015 46

47. AP Profitability Reasons why APs can be unprofitable AP users The impact of video traffic AP strategies: Pricing Heavy hitter charging Heavy hitter blocking Non-neutral charging AP strategies: Connection Caching CP content Peering selectively with CPs 8/19/2015 47

48. Major Findings Variability in AP users can cause large variability in costs Video traffic: Increases costs for AP AP strategies based on differential/non-neutral pricing may not succeed Have to account for user departure due to competition AP strategies based on connection are promising Caching content from CPs Peering selectively with large CPs 8/19/2015 48

49. Contributions of this Thesis A measurement study of the evolution of the Internet ecosystem Modeling the evolution of the Internet ecosystem “what-if” questions about possible evolution paths Optimizations at the edge of the Internet Algorithms for provider selection and egress routing A technical view of the network neutrality debate Strategies for ISP profitability 8/19/2015 49

50. Future Directions Measurements: Investigate the evolution of the connectivity for monitor ASes We can observe all links for such ASes Focus on transitions between peering and customer-provider links Measurements: What does the interdomain traffic matrix really look like? Can we use measurements from a large Tier-1 provider? Can we augment that data with information about the interdomain topology? 8/19/2015 50

51. Future Directions What is the best strategy for different types of providers? Strategies for classes of providers Strategies for individual providers Do the distributed optimizations by networks solve a centralized problem? E.g., minimizing path lengths 8/19/2015 51

52. Other things I’ve been up to Router buffer sizing “Buffer Sizing For Congested Internet Links” “Open Issues in Router Buffer Sizing” Network troubleshooting “NetDiagnoser: Troubleshooting network disruptions using end-to-end probes and routing data” Network monitoring “Route monitoring from passive data plane measurements” Measurement “Poisson vs. Periodic Path Probing” “Bootstrapping in Gnutella” 8/19/2015 52 [Infocom ‘05] [CCR ‘06] [CoNext ‘07] [In progress] [IMC ‘05] [PAM ‘04]

53. Thank You ! 8/19/2015 53

54. 8/19/2015 Issue-1: remove backup/transient links Each snapshot of the Internet topology captures 3 months 40 snapshots – 10 years Perform “majority filtering” to remove backup and transient links from topology For each snapshot, collect several “topology samples” interspersed over a period of 3 weeks Consider an AS-path only if it appears in the majority of the topology samples Otherwise, the AS-path includes links that were active for less than 11 days (probably backup or transient links) Snapshot Samples 8/19/2015 54

55. Issue-2: variable set of BGP monitors Some observed link births may be links revealed due to increased monitor set Similarly for observed link deaths We calculated error bounds for link births and deaths Relative error < 10% for CP links See paper for details 8/19/2015 55

56. Issue-3: visibility of ASes, Customer-Provider (CP) and Peering (PP) links Number of ASes and CP links is robust to number of monitors But we cannot reliably estimate the number of PP links 8/19/2015 56

57. Global Internet trends 8/19/2015 57

58. Transit (CP) vs Peering (PP) relations The fraction of peering links has been increasing steadily But remember: this is just a lower bound At least 20% of inter-AS links are of PP type today 8/19/2015 58

59. The Internet gets larger but not longer Average path length remains almost constant at 4 hops Average multihoming degree of providers increases faster than that of stubs Densification at core much more important than at edges 8/19/2015 59

60. 8/19/2015 60

61. Regional distribution of AS types Europe is catching up with North America w.r.t the population of ECs and LTPs CAHPs have always been more in Europe More STPS in Europe since 2002 8/19/2015 61

62. Evolution of Internet transit: the customer’s perspective 8/19/2015 62

63. Customer activity by region Initially most active customers were in North America After 2004-05, customers in Europe have been more active Due to increased availability of providers? More competitive market? 8/19/2015 63

64. How common is multihoming among AS species? CAHPs have increased their multihoming degree significantly On the average, 8 providers for CAHPs today Multihoming degree of ECs has been almost constant (average < 2) Densification of the Internet occurs at the core 8/19/2015 64

65. Who prefers large vs small transit providers? After 2004, ECs prefer STPs than LTPs Mainly driven by lower prices or regional constraints? CAHPs connect to LTPs and STPs with same probability 8/19/2015 65

66. Customer activity by region Initially most active customers were in North America After 2004-05, customers in Europe have been more active Due to increased availability of providers? More competitive market? 8/19/2015 66

67. Evolution of Internet transit: the provider’s perspective 8/19/2015 67

68. Attractiveness (repulsiveness) of transit providers Attractiveness of provider X: fraction of new CP links that connect to X Repulsiveness, defined similarly Both metrics some positive correlation with customer degree Preferential attachment and preferential detachment of rewired links 8/19/2015 68

69. Evolution of attractors and repellers A few providers (50-60) account for 50% of total attractiveness (attractors) The total number of attractors and repellers increases The Internet is NOT heading towards oligopoly of few large players LTPs dominate set of attractors and repellers CAHPs are increasingly present however 8/19/2015 69

70. Correlation of attractiveness and repulsiveness Timeseries of attractiveness and repulsiveness for each provider Calculate cross-correlation at different lags Most significant correlation values at lags 1,2 and 3 Attractiveness precedes repulsiveness by 3-9 months 8/19/2015 70

71. Evolution of Internet peering (conjectures) 8/19/2015 71

72. Evolution of Internet Peering ECs and STPs have low peering frequency Aggressive peering by CAHPs after 2003 Open peering policies to reduce transit costs 8/19/2015 72

73. Which AS pairs like to peer? Peering by CAHPs has increased significantly CAHPs try to get close to sources/destinations of content Peering by LTPs has remained almost constant (or declined) “Restrictive” peering by LTPs 8/19/2015 73

74. 8/19/2015 Conclusions Where is the Internet heading towards? Initial exponential growth up to mid-2001, followed by linear growth phase Average path length practically constant Rewiring more important than growth Need to classify ASes according to business type ECs contribute most of the overall growth Increasing multihoming degree for STPs, LTPs and CAHPs Densification at core CAHPs are most active in terms of rewiring, while ECs are least active 8/19/2015 74

75. Conclusions Where does the Internet head toward? Positive correlations between attractiveness & repulsiveness of provider and its customer degree Strong attractiveness precedes strong repulsiveness by period of 3-9 months Number of attractors and repellers between shows increasing trend The Internet market will soon be larger in Europe than in North America In terms of number of transit providers and CAHPs Providers from Europe increasingly feature in the set of attractors and repellers 8/19/2015 75

76. Multihoming Multihoming: Connection of a stub network to multiple ISPs x% of stub networks are multihomed Redundancy primary/backup relationships Load Balancing Distribute outgoing traffic among ISPs Cost Effectiveness Lower cost ISP for bulk traffic, higher cost ISP for performance- sensitive traffic Performance Intelligent Route Control 8/19/2015 76

77. ISP selection Should consider both monetary cost and performance Minimum monetary cost Estimate the cost that “would be” incurred if a set of ISPs was selected Minimum AS path lengths Longer paths: delays, interdomain routing failures Measure AS path length offline using Looking Glass Servers Maximum Path diversity AS-level paths to destinations should be as “different” as possible 8/19/2015 77

78. Problem Definition Two phases Phase I – ISP Selection: Select K upstream ISPs K depends on monetary and performance constraints “Static” operation Change only when major changes in the traffic destinations or ISP pricing Phase II – Egress Path Selection Allocate egress traffic to selected ISPs Avoid long term congestion and minimize cost “Semi-static” operation, performed every few hours or days 8/19/2015 78

79. Evaluation – Path Diversity AS-level paths and traffic rates are input to simulator 9 ISPs, 250 destinations Given K, find the selection C* with the minimum path diversity cost For each selection C, find u(C) = total traffic lost due to the failure of each link in topology Calculate Δu(C) = u(C) – u(C*) for each selection C 8/19/2015 79 Single link failures: C* is the optimal selection

80. Egress Path Selection After Phase-I, S has K upstream ISPs Problem: How to map outgoing traffic to the ISPs M flows: K M mappings of flows to ISPs Some mappings may cause congestion to flows ! Flows can be congested at access links or further upstream Objective: Find the loss-free mapping with the minimum cost Challenges: Upstream topology and capacities are unknown Iterative routing approaches required Propose an iterative routing based on simulated annealing 8/19/2015 80

81. Evaluation – Path Diversity AS-level paths and traffic rates are input to simulator 9 ISPs, 250 destinations Given K, find the selection C* with the minimum path diversity cost For each selection C, find u(C) = total traffic lost due to the failure of each link in topology Calculate Δu(C) = u(C) – u(C*) for each selection C 8/19/2015 81 Single link failures: C* is the optimal selection 2,3 link failures: C* is close to the optimal selection

82. Provider and Peer Selection Detailed model for provider and peer selection Complex real-world decisions Provider selection objectives Monetary cost AS path lengths Peer selection Minimize transit costs Maintain reachability Constraints Only local knowledge Geographical constraints 8/19/2015 82

83. Peering Federation Traditional peering links: Not transitive Peering federation of A, B, C: Allows mutual transit Longer chain of “free” traffic Incentives to join peering federation? What happens to tier-1 providers if smaller providers form federations? 8/19/2015 83 A A B B C C

84. Why can the AP be unprofitable? Variability of users => high variability in the costs incurred by AP Variability increases with the access speed Increase in video traffic: higher transit payment by AP 8/19/2015 84

85. Baseline model AP and CP connect to the TP as customers N users of AP, charged a flat rate R ($/month) Transit pricing: 95 th percentile of traffic volume, concave transit pricing functions 95th / mean = 2:1 for normal traffic, 4:1 for video 1 More video means higher transit payment by AP AP users: Heavy tailed distribution of content downloaded per month High variability in AP costs 8/19/2015 85 1 Norton’06: Internet Video: The Next Wave of Massive Disruption to the U.S. Peering Ecosystem

86. AP Strategies Charging strategies AP charges “heavy hitters” according to volume downloaded AP caps heavy hitters AP charges CP (non-network neutral) Charging strategies are disruptive AP cannot control customer departure probability 8/19/2015 86

87. AP Strategies Charging heavy hitters download amount D, threshold T, flat rate R c(D) = D*R/T AP’s profit is sensitive to customer departure prob Capping heavy hitters and non-neutral charging would not work for the same reason 8/19/2015 87

88. AP Strategies Connection Strategies AP caches content from CPs AP peers with CPs Non-disruptive Caching can reduce transit costs of AP But depends on the amount of content cacheable Selective peering with CPs can improve profitability Peering cost depends on CP Cost/benefit analysis for each CP CP with large network: low cost of peering 8/19/2015 88

89. AP Strategies Connection Strategies AP caches content from CPs AP peers with CPs Non-disruptive Cost-benefit analysis for peering Peering cost depends on CP (easy/medium/hard) r = saving/cost (both estimated) Peer if r > R AP controls the factor R 8/19/2015 89

90. AP Strategies Charging heavy hitters download amount D, threshold T, flat rate R c(D) = D*R/T AP’s profit is sensitive to customer departure prob Non-neutral charging Customer departure prob “How discriminatory is my AP?” AP’s profit is sensitive to customer departure prob 8/19/2015 90

91. Why Study Internet Evolution? “Bottom-up” models (more later) Understand how local actions lead to emerging properties Performance of protocols over time “How would BGP perform 10 years from now?” Clean slate vs. evolutionary design After initial design, both must evolve ! Understanding evolution of the current Internet can help design 8/19/2015 91