1. Peering 101 Introduction to Internet Peering
Hello
My name is Bill Norton and I will be presenting this Peering Introduction to Internet Peering presentation today.
We have assembled a wealth of information at the DrPeering web site and will share a little bit here today.
William B. Norton (DrPeering)

2. Our Promise to you
You will understand (roughly) how the Internet is interconnected
Specifically, you will understand and apply the following terms:
1) Internet Transit
2) Internet Peering
3) Internet Peering Ecosystem, Tier 1 ISPs, Tier 2 ISPs; their position and motivations, and the role of Internet Exchange Points
So this is the promise we will make to you regarding the next 30 minutes.
You will have a sense of how the Internet is interconnected, and have a working knowledge of the terms Internet Peering, Internet Transit, and how this whole thing can be viewed within the paradigm of a global ecosystem.
We will do this in three section....
4 sections

3. Quizzes scattered throughout
Building Blocks
Part I - Definitions of Transit and Peering
Part II - Application of definitions: The Internet Peering Ecosystem
We will first introduce the definitions and roles of Internet Transit and Peering,
then we will apply these definitions to give them a test drive and have a quiz/discussion to make sure we have a working knowledge of these terms
Kevin will step in and speak a little bit about the practical side of peering, ease a novice peering coordinator into a seasoned peering coordinator by sharing the top lessons from the trenches.
We will finish up with a conceptual bit about the theoretical framework behind some operations infrastructure that supports peering, an Internet Exchange Point. We will discuss why it makes sense to ISPs, to the IX itself, and present a theoretical model for valuing an IX.
Quizzes scattered throughout

4. Who am i? William B. Norton (Bill Norton)
(former) Co-Founder and Chief Technical Liaison for Equinix, Inc.
North American Network Operators Group (NANOG) Chair
…Internet Operations Researcher: Authored Industry White Papers…
(…Now Retired)
So, who is Bill Norton?
I had the title of co-founder and chief technical liaison for Equinix, a carrier neutral internet exchange point. My primary responsibility was facilitating the interaction between Internet Service Providers, Carriers and content providers that colocated within the building.
Prior to Equinix, I worked for Merit network during the operation of the National Science Foundation Network: the precursor to the modern commercial Internet. During these eleven years at Merit, I developed software, led the consturction of the Network Operations Center, and chaired the North American Network Operators Group: the operations forum for the North American Internet.
But I am probably most well known for authoring a series of 12 white papers documenting Internet Operations activities surrounding internet peering and business.
So why did I write these white papers?

5. Internet White Papers 1998: Protocols, HW, algorithms all documented.
No Docs on Internet OPERATIONS Activities like PEERING
In 1998 I noticed that there were many books on protocols, hardware, algorithms, but there was No Documents on Peering
So I spent all of my time working with the Internet Operations friends, asking questions like
What is Peering?, why peer? and why not peer?, when do IXes make sense?
I documented the answers in the form of white papers. I would then walk other people through these answers and refined the papers
through about 100 walkthroughs.
I made the research papers freely available and these white papers have become reflections of the community mindset on the particular topic.

6. Part I: Definitions of Transit and Peering
Let’s now get into the lexicon of Peering.

7. Def: The Internet is a network of networks.
Def: ISP sells access to the Internet, so...
An ISP must itself get attached to an ISP already attached to the Internet.
Def: ‘Transit’ is service whereby one ISP sells access to the Internet.
“A port in the wall that says ‘Internet this way’”
Upstream
ISP
At the most basic level, understand that the Internet is a network of networks.
An Internet Service Provider by definition sells access to the Global Internet.
So, an Internet Service Provider must itself get attached to an entity that is already attached to the global Internet.
There are two dominant methods for Internet interconnections.
Transit is defined as a service whereby one ISP sells access to the global Internet. Think of it as a port in the wall that says “Internet this way”.
To illustrate, let’s assume you are starting an ISP service called “Blue ISP”. You purchase transit from an “Upstream ISP” who is already connected to the global Internet.
Your “Upstream ISP” will handle everything for you. The Upstream ISP makes sure that your packets get to the right place on the global Internet, and make sure that the the rest of the Internet knows how to get to you and your customers.
Transit is typically a metered service; every 5 minutes a measure is taken, and at the end of the month, the samples are stacked lowest to highest. The 95th percentile measure is the volume upon which your transit service is billed.
Blue
ISP
Transit Service
metered
95th percentile...
Q?Transit Pricing and Billing...

8. Internet Transit Pricing
1999 $1200/Mbps
2004 $120/Mbps
2008 $12/Mbps
2010 $5/Mbps
Source: “Transit Pricing Historical and Projected”

9. Billing Internet Transit: 95th Percentile
1 month of 5 min Samples=vn - vn-1
Lowest
Monthly Sample
Highest Monthly
Sample
}~36 hrs/month to burst for free : : :
134Mbps
95th
Percentile measure (aka 95P) : :
Let’s examine transit service more closely.
Every 5 minutes a byte count sample across a transit interface is taken, and the delta is calculated and stored.
With 12 samples per hour, 24 hours in a day, and 28 to 31 days per month; there are roughly minute samples in a 30 day month.
The 95th percentile means that, for 5 percent of the time, which is about 36 hours, you can burst your traffic up to capacity of the transit pipe, and it won’t cost you anything more. It is only that 95th percentile volume measurement that impacts your transit bill at the end of the month.
So, how could one game this system?
Blue
ISP
Transit Service
1Mbps
Challenge: How can you GAME this system?

10. Free Transit!! 35hrs burst to 26 ISPs
. . .
900Mbps
900Mbps
900Mbps
900Mbps
0Mbps
0Mbps
0Mbps
0Mbps
95th
One approach to game this transit system is to blast traffic to each of 26 ISPs for only 35 hours, making all measures at or below the 95th percentile measures essentially 0.
By round robin blasting of traffic, the upstream ISPs must provide the capacity to handle all the peak traffic, but collect no revenue for the 95th percentile measure.
This is a ricky strategy though because if you don’t account preceisely, the peak traffic could hit the 95th percentile across multiple ISPs, effectively doubling, tripling, or worse your transit bill.
As a result of this type of thing, transit providers implement “minimum commits”, a minimum traffic volume that you will pay for no matter what.
. . .
0Mbps
0Mbps
0Mbps
0Mbps
Upstream ISP A B C Z
Enter minimum commits...

11. Minimum Commits: Tiered Transit Pricing
$/Mbps
Business Knobs:
ISP(s) selection
Minimum Commits
Negotiated Price 12 11 10 9 8 7 6 5 4 3
So, Transit is Easy, Cheap
Why do we need this Peering thing?
100Gbps*$4/Mbps=$400K/mo
So here is what the pricing curves tend to look like for transit.
The more you commit to, the lower the metered unit price for transit.
It becomes a cost minimization function to accurately predict the volume and select the proper commits.
The metered rate today for transit is as high as $20 per megabit-per-second, and as low as free. Free is available under certain rare and very specific conditions.
When I started working for Equinix back in 1998, the metered price of transit was $1200 per megabit per second!
Now it is sub-$12 per megabit per second.
With prices this low, why do we need to even talk about peering?
This is a question that comes up between peering coordinators all the time, and is a worth while one to explore next.
Gbps commits
Source: 2008 NANOG Discussions: $4/Mbps!

12. Def: ‘Peering’ is a reciprocal exchange of access to each others customers.
Motivations to peer
1) Cost Savings
2) Performance Benefits
3) Additional revenue
3a) ABOV
3b) AMZN
All traffic except red traffic goes this way
All traffic except blue traffic goes this way
Upstream
ISP(s)
Blue
ISP
Transit Service
Let’s start out with the definition of peering.
Peering is defined as a reciprocal exchange of access to each others customers.
Think of peering as a local routing optimization.
In the diagram below, the blue ISP sends all of its traffic to its upstream provider, and the red ISP sends all of its traffic to its upstream ISP. Both ISPs realize that they are both sending a healthy chunk of traffic to each other through their upstream Internet Service Providers. They realize that they could directly interconnect their networks, and send traffic destined to or from each others customers directly to each other, bypassing the upstream ISP and its meter.
It may save money to ‘peer‘ with each other, depending on the cost of peering, and the volume of traffic they can peer with each other for free. This is the peering math that you, as a peering coordinator must master.
So, why peer?
Saving this money is a key reason cited, but there a performance benefits to peering; the shortest path between two points is a straight line, and that logic applies here also. Some argue that the performance of a better, faster interconneected Internet will make more money for those that charge based on traffic volume. Therefore, peering and managing the interconnects very well, will actually make you more money.
These are all topics of heated debate in the peering coordinator community.
Red
ISP
Transit Service

13. Two key points about peering
Transit Service
Blue
ISP
Upstream
ISP(s)
Red
Green
all but red
all but blue
& green
1) Peering is not Transitive
2) Peering is not a perfect substitute for Transit
There are two key points you need to know about peering.
First, peering relationships are not transitive. That is, if the Blue ISP peers with the Red ISP, and the Red ISP peers with the Green ISP, this does NOT imply that the Blue ISP can reach the Green ISP through its peering relationship with the Red ISP. Remember that peering is a reciprocal relationship whereby two ISPs exchange access to each others customers, and this does not necessarily include other peers customers.
In that way, Peering is therefore NOT a perfect substitute for Transit.

14. Part II - The Internet Peering Ecosystem
From 30,000 feet - applying the definitions
Now we are ready to apply the definitions we have just learned by looking holistically at the Internet Peering Ecosystem.
Armed with these definitions...

15. Global Internet Peering Ecosystem
Def: Global Internet Peering Ecosystem consists of a set of interconnected internet regions (countries).
Global Internet Peering Ecosystem
JP Internet Region
AU Internet Region
Now to take a giant step back we see that the big “I” Internet consists of a set of ecosystems that I call the Global Peering Ecosystem. The Global Internet Peering Ecosystem has many interconnected Internet regions. You can think of and Internet Region as a country and you’d be mostly correct.
From my research, I have found that each Internet Region contains three distinct types of species of ISPs, and I will reclaim the following names for each species.
There are Tier 1 ISPs that have access to the entire Internet Region solely through free peering relationships.
There are Tier 2 ISPs that have to buy transit from someone in order to reach some destinations in the Internet Region.
And there are Content Providers who do not sell Internet Transit.
Let’s examine each species in turn.
US Internet Region
Tier 1 ISPs
Characteristics of these
Ecosystem Organisms?
Tier 2 ISPs
Content Providers

16. Ecosystem Member: Tier 1 ISP
Def: A Tier 1 ISP is an ISP
that has access to the
ENTIRE Internet Region Routing Table
Solely via Peering Relationships
Tier 1 ISP P $ $ … T T
(Doesn’t buy transit from anyone
to reach any destination in the
Internet Region.)
First, let’s look at the Tier 1 ISP species.
I have drawn a picture to illustrate its essential characteristics so you can identify them in the wild.
The Tier 1 ISP sells transit to downstream ISPs and Content Providers, as shown by the lines coming out the bottom and the dollar signs towards the top of these transit lines. He also Peers with other Tier 1 ISPs in order to reach destinations that aren’t on his network but exist within the Internet Region.
Since the Tier 1 ISP by definition doesn’t buy transit from anyone to reach any desitnation within the Internet Region, there is no motivation top peer with anyone else. In fact, there are disincentives to peer with anyone else, since they or their peers would lose revenue by peering with others.
As a result, the Tier 1 ISP tends to have a “Restrictive Peering Inclination”, as articulated in a “Restrictive Peering Policy”. The Tier 1 ISPs of course do not call their peering policy a restrictive peering policy; they would use words like ‘We will peer with networks of similar size and scale’, but research has shown that very few true Tier 1 ISPs are openly interested in peering with any other ISPs within the Internet Region in which they are Tier 1 ISP.
Motivation: Is NOT motivated to
Peer in region to reduce transit fees,
Is NOT motivated to peer with
anybody else.
Behavior: “Restrictive” Peering
*def: Policy

17. Ecosystem Member: Tier 2 ISP$ $
Ecosystem Member: Tier 2 ISP
Tier 2 ISP T … T P
Def: A Tier 2 ISP is an ISP
that has to purchase Transit to access
some part of the Internet Region. P $ $ … T T
A Tier 2 ISP is just like a Tier 1 ISP except that he has to purchas transit from someone in order to reach some destinations within the Internet Region. These upstream transit connections are shown as the lines going up off the top, to the dollar signs indicating that there is money or something of value exchanged to an upstream.
The Tier 2 ISP peers openly or selectively with other Tier 2 ISPs and even Content Providers, for the reasons explained earlier. If peering can reduce its transit costs, improve performance for its customers, increase revenue, or a fitting combination of these, the Tier 2 ISP is generally open to peering.
For these reasons, you will be able to find Tier 2 ISPs at peering forums, operations conferences, and other forums where they can meet up with each other and try and negotiate peering.
Motivation: Is motivated to
Peer in region to reduce transit fees.
Behavior: “Open” Peering or
“Selective” Peering Policy
Active in Peering Forums

18. Content Providers $ $ T … T Def: A Content Provider focuses on
content development and does not
Sell access to the Internet.
Content Providers don’t sell transit, so they are shown with no lines out the bottom, and only lines up to their upstream transit provider.
It is important to note that the vast majority of Content Providers are focused on content creation, and may have few or no networking people to talk about peering. They are often quite happy to buy transit at today’s low prices and let someone else deal with thte delivery of their traffic to the global Internet.
When we discuss the evolution of the Peering Ecosystem in the U.S. we will see that some large content providers have found value in growing peering tentacles out the side.
Motivation: SLAs w/well known ISP
Behavior: “No Peering” Policy

19. Generic Peering Ecosystem
Tier 1 ISPs
Tier 2 ISPs
Content Providers
Traffic and
$ flow up T T T T T T T T T
So when we put this architecture together we have something that looks like this.
Note that all traffic not peered at a lower level generates traffic and dollars to the Tier 1 ISPs. This is one reason it is good to be a Tier 1 ISP. T T T
Active Peering Groups
Peering Forums
IX Meetings
Test: Apply defs…

20. Quiz 2) Definition of Peering: ________________ Tier 1 ISP X
Tier 1 ISP Y P
1) Definition of Transit:
________________ T T $ $
Tier 2 ISP B
Tier 2 ISP A
So now we have a quiz to see if you have absorbed the material.
Consider the Internet Region above, in which there are two Tier 1 ISPs; ISP X and ISP Y.
ISP X has a downstream ISP A, and ISP Y has a downstream ISP B who in turn has a content provider customer Content Provider C
Question 1: What is the definition of Internet Transit?
Question 2: What is the definition of Peering?
Question 3: What is the definition of an “Open” peering policy?
Question 4: What is the definition of a “Selective” peering policy?
Question 5: What is the definition of a “Restrictive” peering Policy? T $
Content
Provider C
3) Definition of an “Open” Peering Policy: _____________________
4) Definition of a “Selective” Peering Policy: _____________________
5) Definition of a “Restrictive” Peering Policy: ____________________

21. Apply Defs: Peering Dynamics & Motivations
No, like $,
& I Like customer B
Tier 1 ISP X
Tier 1 ISP Y P
No, I already hear your routes
for FREE! P? P?
No, like $. T P? T $ $
Tier 2 ISP B
Tier 2 ISP A T $
Here we will test your understanding of the peering ecosystem and definitions with the typical interactions seen in the wild.
Question: What will ISP Y most likely respond when Tier 2 ISP B requests peering?
Answer: ISP Y will respond negatively to the downstreams request. After all, it prefers the revenue generated from handling the downstream ISPs traffic, and decreasing the transit traffic decreases its revenue.
Question: What will ISP Y most likely respond when Content Provider C requests peering?
Answer: ISP Y will respond negatively to the downstream Content Provider request. After all, it prefers the revenue generated indirectly from handling the downstream ISPs transit traffic, and decreasing the transit traffic decreases its revenue. Further, by decreasing its downstream ISP’s traffic and revenue, he harms his customer. There is little or no motivation to support this peering request.
Question: What will ISP Y most likely respond when Tier 2 ISP A requests peering (remembering that ISP A is the downstream of ISP X)?
Answer: ISP Y will respond negatively to its Tier 1 peers downstreams ISP request. While he doesn’t lose revenue by peering with his peers downtreams, he doesn’t gain any benefits, and may irritate his peer, who will lose some revenue.
Hopefully this provided you with a chance to exercise and test your understanding of the key concepts presented here.
Content
Provider
Synch Point:
You have all the defs needed to predict behavior in the Peering Ecosystem.
You should be able to answer the question at hand.

22. Part III - Evolution of the U.S. Peering Ecosystem
Illustrative of dynamics
Applies definitions
So let’s explore the U.S. Peering Ecosystem. Specifically how it has evolved over the past 10 years.
Based on “The Evolution of the U.S. Peering Ecosystem”

23. Evolution #1 U.S. Evolution #1 Cable Companies Peer
T1 ISPs
T2 ISPs
Content
Evolution #1
U.S. Evolution #1
Cable Companies Peer
Cable
Significant Evolution…
Volume of traffic is huge
Cable Cos Open Peering
“Kazaa Effect” amplifies
peering benefits
went bankrupt the cable companies were forced to fend for themselves.
They needed to set up, operate and maintain their own internet network, with services and support on a months notice.
This was a major change in the U.S. Peering Ecosystem as a dozen new Internet players with millions of eyeballs attached were now separate entities, and the volume of traffic that they pulled down for their eyeballs was huge.
More than that, Kazaa, a popular peer to peer application at that time, represented % of their transit costs. As the cable companies increased the size of the transit pipe, the peer to peer traffic increased to fill it again.
I facilitated the interconnection of the cable companies so that they didn’t have to pay transit fees on the peer to peer traffic that was cable company to cable company traffic. This helped them save 40% on their transit bills, but more importantly, it created a new and powerful class of peering species in the ecosystem.
The volume of peerable traffic was measured in the gigabits per second and they started with an open peering policy. This move alone provided an adrenaline shot to peering since the attractiveness of peering points included the attractiveness of peering large volumes of traffic with the cable companies.

24. U.S. Evolution #2 Large Scale Content Players Peer
T1 ISPs
Significant Evolution…
Cable
Volume of traffic is huge
Content is Open Peering
Improves End-User
Experience
4) Leading Players are
paving the way
…need to move out of
Bankrupt colo anyway…
T2 ISPs
T2 ISPs
LSCP
Content
Around the same time, the largest and most network savvy content companies in the world started pursuing peering as a way to improve the end user experience while also bypassing the metered transit providers. Example of this include Yahoo, Microsoft, Google, Electronic Arts, Sony Online, etc.
Content

25. U.S. Evolution #3 Cable Cos Peer w/Large Scale Content Players
T1 ISPs
Significant Evolution…
T2 ISPs
T2 ISPs
Volume of traffic pulled away
from T1s is huge
2) Reduces perceived need for
T1s (for local delivery anyway)
3) T1s still needed for distance
 Content Literally right on the
Cable Company Network
T2 ISPs
Content
Content
And all of this led to the cable companies openly peering with the largest content companies in the world.
This is significant because the volume of traffic is large and represents a significant decrease in traffic that traverses the traditional tier 1 ISPs.
Content

26. Conversations @GPF Summary... Does peering make sense anymore?
To do Transit, CDN, T&Peering, P2P?
What do you pay for transit from ____? (I pay less)
What ridiculous prices IXes charge. (I negotiated down)
How much traffic are you pushing? (I do more)

27. IX Network Externality
Value
Value derived
from IX participation
V=n(n-1)/2
So here we see the network externality effect as it applies to Internet Exchange points.
The value of the Internet Exchange is plotted on the Y axis, and the number of peering participants and/or the value of the routes is plotted on the X axis.
We also plot on this graph the cost of building into an Internet Exchange, shown as a horizontal line.
So, for the first few ISPs that populate the Internet Exchange, the value of participation is negative; the cost of building in exceeds the value derived from peering. They simply can’t peer enough traffic to make it worth their while. They are building in on faith that the value will be there.
The point where the value of the Internet Exchange equals the cost of building into the Internet Exchange is called the Peering break even point, or the point where the IX reaches critical mass.
The value of the IX grows exponentially from the critical mass point up to the point where the IX can not accept any more participants. This is because the value of the IX for n participants is their ability to peer with (n-1) others.
The value of the IX levels off as the IX reaches capacity and can not add any more participants. Since no one else
can peer, the value levels off.
IX Critical Mass
(Value=Cost)
Cost of IX Participation
f(#participants,uniqueRoutes)
The Startup Hump
Discussion Here

28. Source: The Art of Peering: the IX Playbook
Asked IX Operators
How did you get to critical mass?
Europe: ISP consortium starts it
Commercial Company targets key ISPs
Drop price of Participation
Equity
Evangelize, host content (BW sales as lure in)
Find new large volume target peering customers (Video)
A question that often comes up is, how do IXes attract the 1st ISPs into the Internet Exchange Point? After all, the value of participation is zero or less for the first ISP in the building. How do they convince the first ISPs to populate it?
In Europe, the path has been the London Internet Exchange (LINX) model. A group of interested, and more to the point, motivated ISPs reach an agreement to structure an association to build an Internet Exchange to their mutual benefit. A formal association is set up with bylaws, a charter, a staff and meetings for the members where they vote on things affecting the membership. The physical peering equipment is hosted at a colocation provider that has made arrangements to also colocate the IX peering fabric equipment. The colocation providers typically need to pay for fiber builds from other colocation centers that currently host the IX and provide equipment and space for the IX for free.
In the US the commercial Internet Exchange model dominates the landscape. Commercial exchange points are launched by colocation providers, so to participate in the Internet Exchange, one typically needs to be colocated with the colocation provider that operates the peering fabric. These commercial IXes have built critical mass through a variety of ways that association-based IXes can not.
Commercial IXes target key ISPs to act as draws and offer special incentives to get them in, including a drop in price, or an equity stake in the company so they participate in the upside of the more valuable IX.
Commercial IX operators host events to facilitate the ISPs meeting each other, spending money in elaborate ways that association based IXes tend not to.
Source: The Art of Peering: the IX Playbook

29. Models of IXes IX & Customers IX separate from colo (Europe)
IX owns colo (US)
LINX/AMS-IX/DE-CIX model
US Equinix/PAIX/NOTA model
colo1
colo2
colo3
colo4
colo5
colo6
colo1
Customers IX

30. Internet Service Providers
Neutrality
Carriers
Colocation
Provider
ISP-neutrality?
Carrier-Neutrality?
Carrier and Colo neutrality?
Why is this important?
Turkish Internet example
Internet Service Providers CP
Ecosystem

31. Peering Math Colo $1,500 per mo IX Port $3,500 Total Peering Costs
$5,000
Gbps
Transit $/Mbps
Peering $/Mbps 1
$5.00 2
$4.50
$2.50 3
$1.67 4
$1.25 5
$4.30
$1.00 6
$0.83 7
$0.71 8
$4.00
$0.63

32. Peering v Transit

33. Peering v Transit is always a question
Peering vs. Transit
$/Mbps
Peering
General Observations 10 9 8 7 6 5 4 3 2 1
Peering v Transit is always a question
+Performance
+Marketing Benefits
+Control
Transit
Gbps

34. Conclusion This was an overview (Peering 101)
Concepts and common lexicon
White Papers on the net:
or Google and Bing for ‘william b. norton’