The Next Wave of Massive Disruptions to the Peering Ecosystem Asia Pacific Peering Forum Singapore, October 5, 2006 William B. Norton Co-Founder & Chief Technical Liaison Equinix, Inc. Slide set v0.9

Internet Operations White Papers Name: William B. Norton Internet Researcher What is Peering? When does it make sense?

On the Internet Everyone is a Publisher

Internet Operations White Papers 1)“Interconnection Strategies for ISPs” 2)“Internet Service Providers and Peering” 3)“A Business Case for Peering” 4)“The Art of Peering: The Peering Playbook” 5)“The Peering Simulation Game” 6)“Do ATM-based Internet Exchanges Make Sense Anymore?” 7)“Evolution of the U.S. Peering Ecosystem” 8)“The Asia Pacific Internet Peering Guidebook” 9)“The Folly of Peering Traffic Ratios?” Freely available. See Web site or send to Internet makes anyone a publisher, similar effect now emerging for video

Massive Disruption in U.S. Peering Ecosystem  Short Videos YouTube – founded 2005 –Short video clips – 50 million view per day! –20Gbps of peering traffic Feb 2006 –$1M/month in Sept 2006! –Entering Peering Ecosystem –30 Other competitors  600Gbps peerable? DoveTail Video may dwarf current peered traffic –2010 – 80-90% Internet is Video –Inculcate video guys into peering ecosystem On the Internet Everyone is a Broadcaster Short video clips…Full TV shows… Source: Source:

Massive Disruption in U.S. Peering Ecosystem  Full Episodes “Desperate Housewives” – 210MB/hour –For 320x240 H.264 Video iTunes image 10,000,000 households 2,100,000,000 MB = 2.1 peta-Bytes How long will that take to download? Source: Historical Perspective…review 5yr disruptions… 3 64Gbps non-stop ! Just one show Try 250M*180 Channels*HDTV

Evolution #1 1)Volume of traffic is huge 2)Cable Cos Open Peering 3)“Kazaa Effect” amplifies peering benefits Scale: O(20Gbps) peered Significant Evolution… 2002: Evolution #1 Cable Companies Peer T1 ISPs T2 ISPs Content

1)Volume of traffic is huge 2)Content is Open Peering 3)Improves End-User Experience 4) Leading Players are paving the way Scale: O(100Gbps) peered Significant Evolution… T2 ISPs Content T1 ISPs T2 ISPs Content 2002: Evolution #2 Large Scale Content Players Peer

1)Volume of traffic pulled away from T1s is huge 2) Reduces perceived need for T1s (for local delivery anyway) 3) T1s still needed for distance  Content Literally directly on The Cable Company Network Scale: O(100Gbps) Significant Evolution… 2002: Evolution #3 Cable Cos Peer w/Large Scale Content Players T2 ISPs Content T2 ISPs Content T1 ISPs Content T2 ISPs

1)Volume of traffic is huge 2) Most Traffic is Regional 3) Massive Growth 4) Many Emerging players 5) Video size growth Scale: > O(600Gbps) Significant Evolution… 2006: Evolution #4 VideoPeering T2 ISPs Content T2 ISPs Content T1 ISPs Content T2 ISPs Video Service Providers Notes: Questionable if aggregate capacity exists TBD Impact of CDN/P2P/Satellite/caching/etc. Net Neutrality Issues not considered here

Research Topic Massive Wave of Internet Traffic –90% of all Internet bits by 2010 How will Video Service Providers distribute this massive amount of Video Traffic over the Internet?

Modeling the Video Service Provider Distribution Networks Four Models 1.Commodity Transit 2.CDN 3.Transit/Peering/DIY CDN 4.Peer2Peer Four Load Models A: Small Load B: Medium Load C: Large Load

Load Model A Load Model A – Light Load: Every 5 minutes, 10 customers each start to download a 1.5 GB movie, resulting in an average 15GB five minute load. Adjust load to sinusoidal customer traffic demand curve Jeff Turner: 6.6:1 peak-to-mean

Load Model B Load Model B – Medium Load: Every 5 minutes, 100 customers each start to download a 1.5 GB movie, resulting in an average 150GB five minute load. Adjust load to sinusoidal customer traffic demand curve Jeff Turner: 6.6:1 peak-to-mean

Load Model C Load Model C – Large Load: Every 5 minutes, 1000 customers each start to download a 1.5 GB movie, resulting in an average 1500GB five minute load. Adjust load to sinusoidal customer traffic demand curve Jeff Turner: 6.6:1 peak-to-mean

Assumptions 1000 full length 1.5GB videos Equipment Costs Transit Costs Colo Costs Staff Costs Software=LAMP Multi-homed

Model 1: Commodity Transit Business Premise: VSP focuses on core competence Transit Providers handle traffic better and cheaper –Economies of scale, Aggregation, Expertise, Billing, Peering, etc.

Model 1A: Transit Light Load

Model 1B: Transit Medium Load

Model 1C Server1 GigE Switch1 Upstream ISPs : Server24 8 * 10GE to upstreams each :::: Distribution GigE Switch 48 port GigE for servers 2 10GE for upstream $10,000 Add another every 24 servers Routers Cisco 6509Sup720-3bxl w/4*4-port 10GE, $150,000 80Gbps from switches, 80Gbps to upstreams Router1 10G Server262 Server263 Server264 : : GigE Switch14 : Router2 Router4 10G

Model 2: Content Delivery Networks (CDNs) for the Distribution of Video Content Business Premise: Single-site transit traffic traverses potentially many network devices, increasing latency and the potential of packet loss: –By spreading web objects closer to the eyeball networks latency is reduced –Fewer network elements are traversed so reliability is improved –Congestion points in the core of the Internet are avoided –CDNs have the expertise, deployed infrastructure, economies of scale from aggregation efficiencies.

Assumption CDN Price Points

Model 2A: CDN Light Load

Model 2B: CDN Medium Load

Model 2C: CDN Large Load

Model 3: Transit/Peering/DIY CDN Business Model Premise: Operation of the Internet distribution is seen as strategic to the VSP: –End-user experience is mission-critical so outsourcing the end user experience to a transit provider or CDN puts the VSP at risk. –The VSP has visibility into what video are being released, which ones are likely to be hot and which ones don’t require special infrastructure adjustments. –Internet Video distribution is so new that the VSP prefers control. This is a strategic focus of the VSP: ensuring reliability, scalability, through the constant monitoring and evolution of the infrastructure to ensure the end user experiences during these early phases of Internet Video Distribution. –The traditional CDN may be ill-suited to distribute very large video object, therefore we have to do it yourself.

Model 3: Transit/Peering Light Load

Model 3A: Peering/Transit Light Load

Model 3B: Transit/Peering Medium Load

Model 3C: Transit/Peering Heavy Load

Model 4: Peer2Peer Business Model Premise: The current Internet Service Providers and CDNs at the core can not handle the load across single or even multiple locations: –Backbone, peering interconnects, and the hundreds of thousands of routers deployed can not handle the load of today and tomorrows video. –the leaf nodes (i.e. Grandma’s PCs left on) in aggregate have the cycles and network capacity, if shared, to distribute popular content today. –Popular content can be chopped up into small chunks such that many downloaders become sources, and topologically close downloaders will prefer the topologically close sources. This ‘swarmcasting’ requires only a source ‘seed’, and a lookup mechanism for the first downloaders to find the seed, and then to direct future downloaders to topologically closer sources.

Model 4A: P2P Light Load

Model 4B: P2P Medium Load

Model 4C: P2P Large Load

Summary Per Video Cost Of delivery

Acknowledgements Vish Yelsangikar (NetFlix), Peter Harrison (NetFlix), Aaron Weintraub (Cogent), Jon Nistor (TorIX), Barrett Lyon (BitGravity), Dave Knight (ISC), Aaron Hughes (Caridien), David Filo (Yahoo!), Jim Goetz (Sequoia Capital), Jason Holloway (DoveTail), Matt Peterson, Richard Steenbergen (nLayer), Lane Patterson (Equinix), Eric Schwartz (Equinix), Pete Ferris (Equinix), David Cheriton (Sun), Andy Bechtolsheim (Sun), Jeffrey Papen (Peak Web Consulting), KC Broberg (Rackable), Henk Goosen (Sun), Geoffrey Noer (Rackable), Jeff Turner (InterStream/nuMetra), Vab Goel (NorWest Venture Partners), Phil Thomas (Quad)