What Lies Beneath: Understanding Internet Congestion Leiwen Deng Aleksandar Kuzmanovic Northwestern University Bruce Davie, Cisco Systems

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 2 Common Wisdom and Our Key Results No congestion in the Internet core –Links are over-provisioned, hence no congestion No correlation among congestion events in the Internet –Diversity of traffic and links make large and long- lasting link congestion dependence unlikely Our key results –There is a subset of links (both inter-AS and intra- AS) that exhibit strong congestion intensity –Congestion events in the core can be highly correlated (up to 3 ASes)

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 3 Why Do We Care? Congestion in the core –Can depend on upon internal network policies or complex inter-AS relationships –Variable queuing delay can lead to jitter, affecting VoIP or streaming applications Correlation –Guidelines for re-routing systems –Most tomography models assume link congestion independence

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 4 Challenges Scalability –How to concurrently monitor a large number of Internet links? Need a light monitoring tool Need a triggered monitoring system Our approach –Pong: a light monitoring tool Per-path overhead 18 kbps –TPong: a triggered monitoring system Capable of monitoring up to 8,000 links concurrently

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 5 Congestion Events Congestion Intensity –How frequently does queue build-ups happen over 30 seconds time scales? We focus on persistent congestion events: –Intensity > 5%; duration > 2 minutes

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 6 Coordinated Probing SD Probe f s d b 4-p probing: a symmetric path scenario Combines e2e and router-targeted probing f probeb probes probed probe,,,

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 7 Pong: Coordinated Probing SD f s d b ΔfsΔfs ΔfdΔfd Half-path queuing delay Locating Congestion Points Tracing Congestion Status Probe ΔdΔd ΔbΔb ΔfΔf ΔsΔs

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 8 Pong: Methodology Highlights Coordinated probing –Send 4, 3, or 2 packets from two endpoints Quality of Measurability (QoM) –Able to deterministically detect its own inaccuracy Self-adaptivity –Switch among different probing schemes based on QoM and path properties

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 9 Vantage Point Selection Problem How to select vantage points to accurately measure congestion at a given link? Link measurability score –How well are we able to measure a specific link from a specific pair of endpoints; a function of: Quality of measurability (QoM) for a given node Queuing-delay threshold quality Observability score –Avoid paths that “see” multiple congested links concurrently

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 10 Triggered Monitoring System Paths usedPath selection algorithm Probing method Probing rateObjective All pathsNo selection, full mesh Low-rate probing Once every 5 minutesTrack topology and path reachability TMon paths – a subset of all paths Greedy TMon path selection Fast-rate probing 5 probes/secMonitor end-to-end congestion Pong paths – a subset of TMon paths upon triggering Priority-based Pong path allocation Coordinated probing 10 probes/sec for e2e probing, 2 probes/sec for router-targeted probing Locate and monitor link-level congestion Greedy algorithm to determine a subset of links Covered 65% (7,800) links with 4.9% (1,750) paths Limit the per-node measurement overhead Priority-based Pong path allocation Maximize quality of measurability

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 11 Coverage & Overhead Statistics We observe ~ 36,000 paths –N^2, N = 191 nodes –Expose ~ 12,100 links at a time Due to routing changes, we are able to observe ~ 29,000 links in total TMon paths: –Up to 2,000 paths running fast-rate probing concurrently –Cover up to 8,000 links concurrently 4.9% paths cover 65% of total links Pong paths –Up to 30 Pong paths; cover up to 350 links concurrently Overhead per node: –Average: 30 kbps, Peak: 68 kbps

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 12 Measurement Quality How good is our vantage-point selection algorithm? –Link Measurability Score: % of measurement samples have non-zero score 80% of measurements is better than fair 60% of measurements is better than good –The key point is that we know how good or bad we are doing

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 13 Key Findings Time-invariant hot spots Strong spatial correlation among congested links Root-cause analysis

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 14 Time-invariant Hot Spots Time-of-day effects for the number of congestion events Small number of links show strong time- invariant congestion intensity

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 15 Time-invariant Hot Spots Most of the links are not inter-continental links as we initially hypothesized Inter-AS links between large backbone networks as well as intra-AS links within these networks AS #Description 174Cogent Communications, a large Tier-2 ISP. 1299TeliaNet Global Network, a large Tier-2 ISP GEANT, a main European multi-gigabit computer network for research and education purposes, Tier Time Warner Telecom, a Tier-2 ISP in US. 3356Level 3 Communications, a Tier-1 ISPs. 237Merit, a Tier-2 network in US. 6461Abovenet Communications, a large Tier-2 ISP RedCLARA, a backbone connects the Latin-American National Research and Education Networks to Europe. 6453Teleglobe, a Tier-2 ISP. 2914NTT America, a Tier-1 ISPs. 3549Global Crossing, a Tier-1 ISPs Abilene, an Internet2 backbone network in US. 4538China Education and Research Network.

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 16 Pair-wise correlation –Percent of time 2 links are concurrently congested –Pair-wise correlation can be quite extensive E.g., 20% of pairs has correlation greater than 0.7 –Correlation: weekend > weekdays Overall congestion level smaller during weekends –Distance between correlated link pairs up to 3 ASes Congestion Correlation

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 17 Hypothesis: –When upstream traffic converges to a relatively thin aggregation point, then traffic surges in an upstream link are likely to create congestion at a thin downstream aggregation link Insights: –Aggregation points correspond to time-invariant hot spots –Interaction between an aggregation point and an upstream link causes link-level correlation Aggregation Effect Hypothesis Aggregation link

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 18 Root-cause Analysis: Example 10Gbps 622Mbps

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 19 Final Statistics RankNetworkPeers 1UUNET2,346 2AT&T WorldNet2,092 3Level 3 Comm.1,742 5Cogent Comm.1,642 7Global Crossing1,041 8Time Warner918 9Abovenet798 RankISP 1Level 3 Comm. 2UUNET 3AT&T WorldNet 6Cogent Comm. 9Global Crossing RankISP 1Level 3 Comm. 2TeliaNet Global Network 4Global Crossing 8Teleglobe RankISP 2NTT America 6UUNET 8AT&T WorldNet 9Level 3 Comm. 10Teleglobe Table 1: Matched locations in the top ten networks defined by the number of peers Table 2: Matched locations in the top ten ISPs that most aggressively promote customer access North America Europe Asia

Aleksandar Kuzmanovic What Lies Beneath: Understanding Internet Congestion 20 Conclusions Triggered monitoring system –Measuring congestion in a scalable way –Key feature: Select vantage points to measure congestion as a function of the measurement quality Key findings –A subset of links experience time-invariant high congestion intensity –There is strong correlation among congestion events at different links (up to 3 ASes) –Root cause: aggregation effect some links thinner than others