The Evolution of the U.S. Internet Peering Ecosystem William B. Norton Co-Founder and Chief Technical Liaison Equinix, Inc.

Internet Researcher 90% with Internet Companies Engineers EQIX: Massive Carrier Neutral Colocation Lots of Blinking lights… Observe: documentation on HW&Protocols Lack of Operations documents Research How does Peering work? White paper process..

Community Operations Research Ground Truth w/dozens of experts Write White Paper v0.1 Walk community through WP for comments Revise White Paper into new version Present White Paper at conferences Solicit comments over lunches and dinners White papers so far…

Internet Operations White Papers 1)“Interconnection Strategies for ISPs” 2)“Internet Service Providers and Peering” 3)“A Business Case for Peering” 4)“The Art of Peering: The Peering Playbook” 5)“The Peering Simulation Game” 6)“Do ATM-based Internet Exchanges Make Sense Anymore?” Freely available. Send to New white Paper: The Evolution of the U.S. Peering Ecosystem, Agenda…

Agenda Definitions to level set –Peering, Transit, ISPs –Synch Point here. Introduction to notion:“Peering Ecosystem” –Internet=Many Internet Regions each with –Tier 1 ISPs, Tier 2 ISPs, Content Providers Evolution of the U.S. Peering Ecosystem –3 major evolutions

Def: Internet Service Provider Def: Internet Service Provider is an entity that sells access to the entire Internet. This service is often called “Internet Transit” Def: Transit is a business relationship whereby one entity sells access to the entire Internet.

Definition of Transit 1) ISP A buys Transit Service Upstream Transit Provider ISP A Upstream Transit Provider Upstream Transit Provider Upstream Transit Providers Def: Transit is the business relationship whereby one ISP announces (sells) reachability to the *entire* Internet to a customer. $$$ ? Typically usage-based Pricing ’04: $18-$200/Mbps Volume based on 95 th Percentile measure Transit is Simple, Convenient: Upstream handles the delivery of packets to the Internet by some means Usage: “I’m purchasing transit from Level 3.” 3) Traffic Flows INTERNETNETWORKSINTERNETNETWORKS 2) Reachability Announcement

Cost of Transit Traffic Exchange 2001 Pricing Sampling Range: $100-$1200/Mbps My Financial Models used $388/Mbps Source: 2004 Survey Avg. Range: $20-$400/Mbps 95 th Percentile Why Peer? 1/15/04 field spot quotes: GX: $65/Mbps, HE: $25/Mbps Cogent $20/Mbps UU: $45/Mbps, etc. varied commits Mbps

Scaling at the Edge Content Heavy ISP Access Heavy ISP UPSTREAMSUPSTREAMS V IG M B T Streaming Media 56k 384k 1.5m $$$ Broadcast.com: 100,000 concurrent unicast streams 15 million streaming hrs/mo, 1300 Live events/day Access AOL+ DSL: 1,000/day Roadrunner Cable Modems: 1M subscribers Negotiate Peering with Upstream ISPs Or focus on peering with like-minded ISPs… 1 T1 consumer streaming 24/7 cost $388/mo in transit Streaming: Broadcast Telephony Video Multimedia Interactive Gaming SPOT Events Tier1 Equipment: $13B/’00 -> $42B/’04 (Source: Infonetics) The Zone: +1M gaming user/mo!

Seek transport Interconnection $ x Why Peer? Motivations for Peering Financial : Reduce load on expensive Transit service Traffic src/dest Measure vs Intuit Usage-based Billing Engineering : Lower latency 1 st Stage of Peering:  Top 10 destination ISP list Transit $$$ Transit $$$ ISP A ISP B Transit ISP Top 10, Def: Peering…

Sample Top 10 Destination List Def: Peering

Definition of Peering Def: Peering is the business relationship whereby ISPs reciprocally announce reachability to each others’ transit customers. Peering USNetEastNetWestNet Peering Transit Routing Tables Peering is *not* a transitive relationship Peering *does not* provide access to the entire Internet Cost Peering vs. Transit Usage: “I buy transit from UUNet and Peer with EastNet, WestNet.”

The Cost of Peering 1) Monthly Circuit Fee into IX 2) Monthly IX Port Fees

When does Peering Make Sense Financially?

Number of Mbps exchanged Generalization: ISP Peering Breakeven Analysis Graphs $/Mbps Exchanged Cost of Transit Cost of Traffic Exchange in Peering Relationship Breakeven Point (ISPs Indifferent between Peering and Transit traffic exchange) Prefer Peering Peering Risk

Synch Point You should have –a working lexicon at this point –A sense of the motivations for Peering & Transit Any questions so far?

The Internet Peering Ecosystem Pre-Crash : circa 2000

Peering Ecosystem System of interconnected (Peering and Transit) players Each group with homogenous peering motivations and observed behaviors –Tier 1 ISPs –Tier 2 ISPs –Content Providers

Def: Tier 1 ISP Definition: A Tier 1 ISP is an ISP that has access to the entire Internet Region solely through peering relationships. (i.e. doesn’t buy transit from anyone to see regional routes) Motivation: Doesn’t need to peer with anyone else, so Peering only with other Tier 1 ISPs

Def: Tier 2 ISP Definition: A Tier 2 ISPs is any ISP that has to purchase transit to reach destinations in an Internet Region Motivation: Peer to decrease costs, improve performance Peering: with Tier 1’s is preferable but hard to get, peering with Tier 2’s as appropriate Must purchase Transit

Def: Content Providers Def: A Content Provider is an entity that produces content for delivery over the Internet but does not sell transit. Motivation: Best Customer Experience (engineered performance) Peering: Not typically done, prefer SLAs. Transit from ISPs

Content Tier 2 ISPs Tier 1 ISPs Transit $ Thickness represents In-Group Peering

The Evolution of the U.S. Peering Ecosystem Act II:

Evolution of the U.S. Peering Ecosystem Four key events leading to drastic disruption in the Peering Ecosystem: 1)The 1999/2000 Telecom Collapse 2)The growth of the Used Equipment Market 3)The Upstream Provider for the Cable Companies went bankrupt 4)Peer-to-Peer file sharing systems (like Kazaa) grow in popularity, traffic grows exponentially between Access Providers (1.5MB MP3  700MB AVI files) 5)Transit Prices Drop, Transport Prices Drop Leads to 3 Major Evolutions

Evolution #1 – Cable Companies are Peering 40% of traffic Kazaa 3-5 Gbps of transit traffic-> Gbps of peering potential ! Significant because 1)Volume 2)Open Peering 3)Kazaa Effect

U.S. Cable Companies Eyeballs

AOL

Content Tier 2 ISPs Tier 1 ISPs Transit $ Thickness represents In-Group Peering

Content Evolution #1 – Cable Companies Peering Tier 2 ISPs CableCos Tier 1 ISPs Peering

Evolution #2 – Network Savvy Large Scale Content Companies get into Peering To reduce transit costs To improve the end user experience They need to move out of bankrupt colo anyway Significant because 1)Volume of traffic is huge 2)Content Providers have Open Peering 3)Leading players pave the way

Content Tier 2 ISPs Tier 1 ISPs Transit $ Thickness represents In-Group Peering

Content Evolution #2 – Large Scale Network Savvy Content Peers Tier 2 ISPs LSNSC Tier 1 ISPs

Other Broadband Players Significant, but… Not Open Peers so disincentive to peer So less volume and less disruptive

Evolution #3 – Cable Companies Freely Peer with Content Companies Content literally on the Cable Company Network Lowest possible latency from content to eyeballs Internet Gaming, Broadband Streaming, etc.

Content Tier 2 ISPs Tier 1 ISPs Transit $ Thickness represents In-Group Peering

Content Evolution #1 – Cable Companies Peering Tier 2 ISPs CableCos Tier 1 ISPs

Content Evolution #2 – Large Scale Network Savvy Content Peers Tier 2 ISPs LSNSC Tier 1 ISPs

Content Evolution #3 – Cable Peers with Large Scale Network Savvy Content Tier 2 ISPs LSNSC Tier 1 ISPs CableCos

International Dynamics – Separate White Paper JPU.S. U.S. Tier 1 ISPs are Tier 2 ISPs in Japan Internet Region Japan Tier 1 ISPs are Tier 2 ISPs in the U.S. Internet Region

Summary 2003 Peering Ecosystem Evolving NoPeeringOpenSelectiveRestrictive Lg.Content Tier2 ISPs CableCos RBOCs Tier 1 ISPs Foreign ISPs

Q&A