The Evolution of the U.S. Internet Peering Ecosystem William B. Norton Co-Founder and Chief Technical Liaison Equinix, Inc. NANOG 31 – San Francisco, CA

Peer Trek I have a question – “Internet Peering” ( and maybe doorbells )– the network frontier . Like Jane Goodall studying the gorillas in the jungle, you have studied the Peering Coordinators in their strange habitat. Will you share what they do, how they think, how they justify peering in a $50 per Meg transit alien planet. Will you document, strange Peering Ecosystems, and show what no Internet book has shown before.

Research Internet Operations Research 5 yr Mission w/Peering Coordinators 90% Externally Focused Correct Lack of Operations Documentation on Peering Practices Document Internet Operations findings in publicly available White Papers…

Process Network Operations Document Find Operations Area Experts-White Paper v0.1 Community Walk Through Revise White Paper Present White Paper at conferences Solicit comments over lunches and dinners

Documents Public Network Operations Documents “”Interconnection Strategies for ISPs “Internet Service Providers and Peering “A Business Case for Peering “The Art of Peering: The Peering Playbook “The Peering Simulation Game “Do ATM based Internet Exchanges Make Sense? “Evolution of the U.S. Peering Ecosystem “Asia Pacific Peering Guidebook Email to wbn@equinix.com for any of these

Question The Evolution of the U.S. Peering Ecosystem “”150 Peering Coordinators: How has Peering changed since Telecom Crash? Foreign Tier 1 ISP enters U.S. and expects peering with all the Tier 1 ISPs – -> explain / document motivations of “NO Explain Peering Motivations and Behavior, particularly across Internet Regions

The Internet Peering Ecosystem The U.S. Peering Ecosystem Internet Regions The Internet Peering Ecosystem Definition: Global Internet Peering Ecosystem consists of many Autonomous but interconnected Internet Regions (Countries) Each Internet Region has roughly three classes of “organism” each with their own position, motivationspredict peering behavior: Tier 1 ISPs Tier 2 ISPs Content Players NOTE: reclaiming definitions HERE

Tier 1 ISP Tier 1 ISP 3 Ge Tier 1 ISP Model Peering Links 1998 Def: An ISP that has access to the entire Internet Region routing table , solely through Peering Relationships. Tier 1 ISP Tier 2 ISP Content Tier 1 ISP Transit (Downstream) Links A G U Q L S C Peering Behavior Motivation: “We don’t need any more Peering.” Inclination: Restrictive SEA SJO LAX CHI DAL NYC ASH ATL 8 Interconnection Regions

Tier 2 ISP Upstream Links Tier 2 ISP Model 9 Peering Links Ge Tier 2 ISP Tier 2 ISP Model Peering Links 1998 Def: All non-Tier 1 ISPs (must buy transit) Tier 1 ISP Tier 2 ISP Content Transit (Downstream) Links Peering Behavior Motivation: Peer to reduce transit fees, performance Inclination: OpenSelective Sparse Peering

Content Provider Model Upstream Links Ge Content Provider Content Provider Model 1998 Def: ProvideS content, Does not peer, Does not sell transit, Mostly Outbound Traffic Tier 1 ISP Tier 2 ISP Content Content Provider Peering Behavior Motivation: Focus on End-User Experience: generally few netstaff Inclination: No Peering

The Internet Peering Ecosystem Pre-Crash: 1998 The Internet Peering Ecosystem Pre-Crash : circa 2000

U.S. Internet Region Tier 1 Full Mesh Tier 2 Sparse Mesh Tier 1 ISPs 1998 Transit Tier 1 ISP Tier 2 ISP Content In-group Peering Tier 2 ISPs Content Tier 1 Full Mesh Tier 2 Sparse Mesh Transit $388/Mbps Booming Economy Peering Growth And then, the crash…

BSOD?

No, not that type of Crash…

Internet Peering Ecosystem The Telecom Industry Crash Crash: ’99-’01 Internet Peering Ecosystem Pre-Crash : circa 2000

Internet Peering Ecosystem Five key events led to a drastic disruption in the U.S. Peering Ecosystem: Crash: ’99-’01 1) Tier 1 ISP Bankruptcies 2) The growth of the Used Equipment Market 3) The Upstream Provider for the Cable Companies (@Home) went bankrupt 4) Peer-to-Peer file sharing systems (like Kazaa) grow in popularity, traffic grows exponentially between Access Providers (1.5MB MP3  700MB AVI files) 5) Transit Prices Drop, Transport Prices Drop Internet Peering Ecosystem Led to 3 major evolutions……  A new Organism

Cable Company Ge Transit (Upstream) Links Cable Organism @HOME Handled Internet 30-60 days notice Tier 1 Transit 40% Traffic Kazaa 3-5 Gbps of transit  Up to 2Gbps of Peering Traffic! 2002 Tier 1 ISP Tier 2 ISP Content Cable Co. Peering Links Cable Co Peering Behavior Motivation: Peer to save money, improve performance Inclination: Open-Selective Sparse (Regional) Peering

U.S. Cable Companies Eyeballs 12,665,600

AOL is the 800 Pound Gorilla

Significant Evolution… T1 ISPs T2 ISPs Content Evolution #1 Cable Companies Peer Evolution #1 Significant Evolution… Volume of traffic is huge Cable Cos Open Peering “Kazaa Effect” amplifies peering benefits

Large Scale Network Savvy Content Player Organism 393 Large Scale Network Savvy Content Providers Ge Transit (Upstream) Links Transit (Upstream) Links Large Scale Network Savvy Content Player Organism 2002 Large Scale Network Savvy Content Player Tier 1 ISP Tier 2 ISP Content Cable Co. LSNSCP Peering Links Peering Behavior Motivation: Peer to save money, improve performance Inclination: Open Sparse (Regional) Peering

Significant Evolution… Large Scale Content Players Peer T1 ISPs Significant Evolution… Volume of traffic is huge Content is Open Peering Improves End-User Experience 4) Leading Players are paving the way …need to move out of Bankrupt colo anyway… T2 ISPs T2 ISPs Content Content

Significant Evolution… Cable Cos Peer w/Large Scale Content Players T1 ISPs Significant Evolution… Volume of traffic pulled away from T1s is huge 2) Reduces perceived need for T1s (for local delivery anyway) 3) T1s still needed for distance  Content Literally right on the Cable Company Network T2 ISPs T2 ISPs T2 ISPs Content Content Content

Peering Ecosystem Evolution Summary NoPeering Open Selective Restrictive Lg.Content Tier2 ISPs CableCos RBOCs Tier 1 ISPs Foreign ISPs Excite@Home

Questions?