1. The Evolution of the U.S. Internet Peering Ecosystem
William B. Norton
Co-Founder and Chief Technical Liaison
Equinix, Inc.
NANOG 31 – San Francisco, CA

2. Peer Trek
I have a question – “Internet Peering” ( and maybe doorbells )– the network frontier . Like Jane Goodall studying the gorillas in the jungle, you have studied the Peering Coordinators in their strange habitat. Will you share what they do, how they think, how they justify peering in a $50 per Meg transit alien planet. Will you document, strange Peering Ecosystems, and show what no Internet book has shown before.

3. Research Internet Operations Research 5 yr Mission
w/Peering Coordinators
90% Externally Focused
Correct Lack of Operations
Documentation on Peering Practices
Document Internet Operations findings in
publicly available White Papers…

4. Process Network Operations Document Find Operations Area
Experts-White Paper v0.1
Community Walk Through
Revise White Paper
Present White Paper at conferences
Solicit comments over lunches and dinners

5. Documents Public Network Operations Documents
“”Interconnection Strategies for ISPs
“Internet Service Providers and Peering
“A Business Case for Peering
“The Art of Peering: The Peering Playbook
“The Peering Simulation Game
“Do ATM based Internet Exchanges Make Sense?
“Evolution of the U.S. Peering Ecosystem
“Asia Pacific Peering Guidebook
to for any of these

6. Question The Evolution of the U.S. Peering Ecosystem
“”150 Peering Coordinators:
How has Peering changed since Telecom Crash?
Foreign Tier 1 ISP enters U.S. and expects
peering with all the Tier 1 ISPs –
-> explain / document motivations of “NO
Explain Peering Motivations and Behavior,
particularly across Internet Regions

7. The Internet Peering Ecosystem
The U.S. Peering Ecosystem
Internet Regions
The Internet Peering Ecosystem
Definition: Global Internet Peering Ecosystem consists of many
Autonomous but interconnected Internet Regions (Countries)
Each Internet Region has roughly three classes of “organism”
each with their own position, motivationspredict peering behavior:
Tier 1 ISPs
Tier 2 ISPs
Content Players
NOTE: reclaiming definitions HERE

8. Tier 1 ISP Tier 1 ISP 3 Ge Tier 1 ISP Model Peering Links
1998
Def: An ISP that has access to the entire Internet Region routing table , solely through Peering Relationships.
Tier 1 ISP
Tier 2 ISP
Content
Tier 1 ISP
Transit (Downstream) Links A G U Q L S C
Peering Behavior
Motivation: “We don’t need any more Peering.”
Inclination: Restrictive
SEA
SJO
LAX
CHI
DAL
NYC
ASH
ATL
8 Interconnection Regions

9. Tier 2 ISP Upstream Links Tier 2 ISP Model 9 Peering LinksGe
Tier 2 ISP
Tier 2 ISP Model
Peering Links
1998
Def: All non-Tier 1 ISPs (must buy transit)
Tier 1 ISP
Tier 2 ISP
Content
Transit (Downstream) Links
Peering Behavior
Motivation: Peer to reduce transit fees, performance
Inclination: OpenSelective
Sparse Peering

10. Content Provider Model
Upstream Links Ge
Content Provider
Content Provider Model
1998
Def: ProvideS content, Does not peer,
Does not sell transit,
Mostly Outbound Traffic
Tier 1 ISP
Tier 2 ISP
Content
Content
Provider
Peering Behavior
Motivation: Focus on End-User Experience: generally few netstaff
Inclination: No Peering

11. The Internet Peering Ecosystem
Pre-Crash: 1998
The Internet Peering Ecosystem
Pre-Crash : circa 2000

12. U.S. Internet Region Tier 1 Full Mesh Tier 2 Sparse Mesh
Tier 1 ISPs
1998
Transit
Tier 1 ISP
Tier 2 ISP
Content
In-group
Peering
Tier 2 ISPs
Content
Tier 1 Full Mesh
Tier 2 Sparse Mesh
Transit $388/Mbps
Booming Economy
Peering Growth
And then, the crash…

13. BSOD?

14. No, not that type of Crash…

15. Internet Peering Ecosystem
The Telecom Industry Crash
Crash: ’99-’01
Internet Peering Ecosystem
Pre-Crash : circa 2000

16. Internet Peering Ecosystem
Five key events led to a drastic disruption
in the U.S. Peering Ecosystem:
Crash: ’99-’01
1) Tier 1 ISP Bankruptcies
2) The growth of the Used Equipment Market
3) The Upstream Provider for the Cable Companies went bankrupt
4) Peer-to-Peer file sharing systems (like Kazaa) grow in popularity, traffic grows exponentially between Access Providers (1.5MB MP3  700MB AVI files)
5) Transit Prices Drop, Transport Prices Drop
Internet Peering Ecosystem
Led to 3 major evolutions……  A new Organism

17. Cable Company Ge Transit (Upstream) Links Cable Organism
@HOME Handled Internet
30-60 days notice
Tier 1 Transit
40% Traffic Kazaa
3-5 Gbps of transit
 Up to 2Gbps of Peering Traffic!
2002
Tier 1 ISP
Tier 2 ISP
Content
Cable Co.
Peering
Links
Cable Co
Peering Behavior
Motivation: Peer to save money, improve performance
Inclination: Open-Selective
Sparse (Regional) Peering

18. U.S. Cable Companies Eyeballs
12,665,600

19. AOL is the 800 Pound Gorilla

20. Significant Evolution…
T1 ISPs
T2 ISPs
Content
Evolution #1
Cable Companies Peer
Evolution #1
Significant Evolution…
Volume of traffic is huge
Cable Cos Open Peering
“Kazaa Effect” amplifies
peering benefits

21. Large Scale Network Savvy Content Player Organism
393
Large Scale
Network Savvy
Content Providers Ge
Transit (Upstream) Links
Transit (Upstream) Links
Large Scale Network Savvy Content Player Organism
2002
Large Scale
Network Savvy
Content Player
Tier 1 ISP
Tier 2 ISP
Content
Cable Co.
LSNSCP
Peering
Links
Peering Behavior
Motivation: Peer to save money, improve performance
Inclination: Open
Sparse (Regional) Peering

22. Significant Evolution…
Large Scale
Content Players Peer
T1 ISPs
Significant Evolution…
Volume of traffic is huge
Content is Open Peering
Improves End-User
Experience
4) Leading Players are
paving the way
…need to move out of
Bankrupt colo anyway…
T2 ISPs
T2 ISPs
Content
Content

23. Significant Evolution…
Cable Cos Peer w/Large
Scale Content Players
T1 ISPs
Significant Evolution…
Volume of traffic pulled away
from T1s is huge
2) Reduces perceived need for
T1s (for local delivery anyway)
3) T1s still needed for distance
 Content Literally right on the
Cable Company Network
T2 ISPs
T2 ISPs
T2 ISPs
Content
Content
Content

24. Peering Ecosystem Evolution
Summary
NoPeering
Open
Selective
Restrictive
Lg.Content
Tier2 ISPs
CableCos
RBOCs
Tier 1 ISPs
Foreign ISPs

25. Questions?