1. Peering & Internet eXchange Points AfNOG 2001 Meeting 12 th May, 2001. Miklin Hotel Accra – Ghana

2. Abstract  Internet Service Provider (ISP) Peering has emerged as one of the most important and effective ways of improving efficiency of service and operation. Peering is the interconnection mutual business arrangement between at least two Service Providers whereby each directly exchanges traffic to and from each other's clients. Peering relationships are sought primarily because peering reduces cost and reliance on purchased Internet bandwidth and/or transit. Secondly, peering lowers inter-AS traffic latency. By avoiding a transit provider hop in between ISPs traffic between peering ISPs has lower latency. So how is peering done and what are the issues and structures of peering?  In peering arrangements, especially when two ISPs of similar size want to interconnect it becomes a problem. The setting up of Internet Exchange Paints (IXP), which is the neutral interconnection point of traffic exchange between peering ISPs has been a way to solving peering problems.  In this paper, focus is made on issues regarding peering and Internet Exchanges, structures, roles and the decision making process which further breaks down into three distinct major processes as follows: Identification (Traffic Engineering Data Collection Analysis), Contact and Qualification (Initial Peering Negotiations) and Implementation Discussion (Peering Methodology). The first phase identifies 'the who‘and 'the why', while the second phase focuses on 'the how'.

3. Background to ISP Peering  The global telephone system and the Internet share a major common attribute of value to all users of their respective services - that of comprehensive connectivity  Preservation of such connectivity became a problem due to the growth of the Internet.  Transition and Financial consideration necessary to guarantee the long term viability of the medium were also problems  Peering is a one way to solving these problems  The Interconnection points for traffic exchange between peering ISPs on the Internet is the Internet eXchange Point  Exchange Points are high-speed (not less than 100mb) switch network, with current physical configuration of today being a mix of FDDI/ATM switches

4. General Peering Models  Takes place within one of three general models  First is that of bilateral settlements  The second model is that of the Sender Keep All (SKA)  The third is that of transit fees  Within the Internet to date only the second and third models can be observed within the deployed infrastructure  SKA arrangement is only stable where the parties involved perceive equal benefit from the interconnection  The third model (transit fees) is commonly seen as a component of the relationship between a transit provider and a local access provider, where the local provider is charged for transit services within a larger service agreement.  There is also that of multilateral settlement, which has been developed at neutral exchanges.

5. Issues & Structures  Within the Internet the common basis of the settlement appears to be one of flat fees, without reference to exchanged data volumes  Significant issues include the status of an ISP - a peer to, or a client of another ISP  Another is that of imbalance in traffic exchange  There are three generic peering structure options: direct point-to-point interconnection, multilateral exchange points and mutual bilateral exchange points

6. Issues & Structures  Direct point-to-point peering Each bilateral peer arrangement is implemented through a physical interconnection  Multilateral exchange points Seen within the CIX exchange architecture, and within the European EBONE structure Architecture is the enforcement of a single multilateral peering structure Stable when all parties are in a position of approximate equal positioning with respect to the mutual traffic exchange  Mutual bilateral exchange points Offers the greatest degree of flexibility Have been used for some years within the United States

7. Decision Making Process  The decision making process involves three main phases  These phases identifies ‘the why’, ‘the who’ and ‘the how’ of Peering and Exchange Points Setup  The Three Major Decision Making Process Identification of Potential Peer: Traffic Engineering Data Collection and Analysis Contact & Qualification: Initial Peering Negotiation Implementation Discussions: Peering Methodology

8. Phase One: Identification of Potential Peer Traffic Engineering Data Collection and Analysis  An interview conducted by W.B. Norton with some ISPs highlighted a common concern: traffic destined for a competitors customer located across the street may need to traverse a couple of transit providers across great distances (with high latency) before interconnecting. The worst example was that traffic between the United Arab Emirates and Saudi Arabia must traverse an exchange point in Washington DC. But through direct interconnections (through direct circuits or regional exchange points) ISP customers realize better performance

9.  Reduces Transit Costs. Internet transit costs dominates the decision making process Peering reduces transit costs  Lower Latency Direct interconnection with peers lowers latency in traffic between peering ISP's High Speed, low latency access to sites for end-users  Scaling Bandwidth Continued success of the Internet depends on increasing bandwidth Since packet loss and latency slows traffic consumption, there is benefit from a low latency, low packet loss Internet  Enables POP Activity Provides a hardened and scalable environment in which to operate POPs Motivations: Why do you need to peer?

10. ISP A ISP B Transit ISP Transit Cost ($$) Peering provides routes only to each others customers Transit vs. Peering Interconnection Peering between ISP A and ISP B Transit from ISP A and ISP B to Upstream or Transit ISP

11.  Variety of Criteria are used to identify Potential Peers, such may include the following: Quantities of traffic Business Credibility Interconnection Policies The Criteria: Whom do you peer with?

12. Yes Part of broad Business Relationship? Dominant Traffic Flow? Large new Customer impact? Traversing Expensive transit cost? Will Peering Have a positive effect on my network? Proceed to Phase 2 Contact Peer A Simplified Peering Selection Decision Tree

13. Migration Path from Transit to Peering  Interviews conducted with some tier 2 ISPs highlighted an emerging peering transition strategy: Access the Internet via transit from a global provider Pursue peering arrangements on public switches at exchange points to reduce load on transit links and improve performance Migrate high traffic peering interconnections to private interconnections (via fiber or direct circuits) Ultimately migrate traffic away from transit purchase and negotiate (free or for-fee) peering with former transit provider Interview was conducted by W. B. Norton (Equinix Inc.) in the U.S.

14. Phase Two: Contact & Qualification Initial Peering Negotiation  ISPs should have a person or group in the organization specifically tasked with traffic engineering issues  The first step in a peering arrangement is for one the parties to initiate the contact with the potential peer partner  Finding the right person to speak with is a difficult intensive task  Second step, though optional is the negotiation and signing of mutual non- disclosures  The third step is to evaluate requirements and capability  Each party then decides whether to stay or to walk away until criteria and requirements are met

15. Principles for Peering Arrangements  However, some principles are needed to be considered during preparations for peering arrangements. These include: providing interconnection agreement under non-discriminatory and transparent terms and conditions; providing interconnection agreement at non-discriminatory cost-oriented rates and comparable quality to that which it provides itself; negotiate interconnection in good faith; and ensure that interconnection agreements, or a reference offer, are made public.  Interconnection is a necessary preliminary to traffic exchange

16. Contact & Qualification Decision Tree Proceed to Phase 3 Implementation of Discussion Peering @ or personal contact Exchange Point Contact list tech-c or admin-c in DNS/ASN Registry Operations Forum Trade Shows Sales Force Larger business transactions Initial Contact Do both parties find motivation to continue peering discussion? Close Discussion Yes No Sign NDA, See Policies Share traffic data, BLPA

17. Phase Three: Implementation Discussions Peering Methodology  Since peering is of mutual benefit, both parties next explore the interconnection method(s) that most effectively exchange traffic to and from each other's customers  The primary goal is to establish mutual point(s) of interconnection, and secondarily detail optimal traffic exchange behaviours  ISPs face three options for interconnections: Direct Circuit Interconnection, Exchange-Based Interconnection or Some global combination thereof  The preferred methodology to adopt depends on the number of peers participating and interconnecting at the Exchange Point  And also on the bandwidth required for the Exchange Interconnections  ISPs that expect to interconnect at high or rapidly increasing bandwidth within the region, or expect interconnections with more than five parties in the region often prefer the exchange-based solution where we have the multilateral exchange points structure  Those that do not anticipate a large number of regional interconnects prefer direct-circuit (the direct point to point structure) and typically decide to split the costs of interconnection with the peer by region

18. Peering Arrangement Goals  get peering set up as soon as possible,  minimize the cost of the interconnection and their transit costs  maximize the benefits of a systematic approach to peering  execute the regional operations plan as strategy dictates (may be architecture / network development group goal), and  fulfill obligations of larger business agreement.

19. Direct-Circuit Interconnection  Interconnection takes place in one of two forms: Peering and Transit  In either case, it requires provision of bandwidth between parties  Direct Point-to-Point interconnection model requires the lease of point-to-point circuits between parties that scale linearly with the number of ISPs  The circuits traverse miles of fiber underground, subject to outages due to construction (and the rapid proliferation of backhoes)  As the bandwidth requirements grow, ISPs need to upgrade circuits to many different places; they are unable to take advantage of traffic aggregation over a very high bandwidth circuit  Cost of this interconnection strategy does not scale either

20. Direct-Circuit Interconnection

21. Point-to-point circuits for ISP interconnection in an exchange (a simpler diagram) ISP B ISP A ISP C R2 R1 R3

22. Exchange-Based Interconnection  An alternative to the direct circuit interconnection model is for ISPs to purchase instead a much larger circuit into the Neutral Internet Business Exchange to take advantage of the relatively inexpensive and the rich connectivity mesh within the exchange  Efficient to scale a single high bandwidth, point-to-point circuit than many lower speed point-to-point circuits  The greater the dependence on the interconnection, the more hardened and scalable that interconnection environment should be  In a Neutral Internet Business Exchange Model, ISPs have access to an additional source of revenue: transit sales to content providers and ISPs

23. Exchange-Based Interconnection ISP A R1 ISP C R3 ISP E R5 ISP F R6 ISP D R4 ISP B R2 R0 Exchange Point Exchange Point High-Speed Switch Each ISP has its own transit uplink

24. Peering Implementation and Operation Exchange Environment Selection Criteria Deployment Issues ISP Current Presences Telecom Access Issues Cost Issues Business Issues Operations Issues Credibility Issues Exchange Population Existing Vs. Emerging Exchange Environment Selection Criteria

25. Telecommunication Cost Issues  Have to do with getting telecommunications services into the exchange  How fast can circuits be bought into the interconnection environment?  How many carriers compete for my business for circuits back to my local Point of Presence (POP)?  For facilities-based ISPs, what is the cost of trenching into the exchange (how far away and what obstacles present themselves)?  Are there nearby fiber providers that lease strands?  These answers will answer the most important question to ISPs: How fast can my peer and I get connectivity into the exchange?  The answers to these questions strongly impact the desirability of the exchange environment

26. Deployment Issues  Have to do with getting equipment into the exchange  How do I get my equipment into the exchange (assuming it supports collocation)?  Do I ship equipment in or do I have to bring it with me as I fly in?  Will someone act as remote hands and eyes to get the equipment into the racks or do I do the installation myself?  What are the costs associated with deployment (travel, staff time, etc.)?  Does the exchange have sufficient space, power?  The answers to these questions impact the deployment schedule for the ISP engineers and the costs of the interconnection method

27. ISP Current Presences  Based on the observations that the most inexpensive and expedient peering arrangements are the ones made between ISPs that are already located in the same exchange  The assumption here is that there is sufficient capacity to interconnect. Cross- connects or switching fabrics can easily establish peering within hours or at most days

28. Operations Issues  Focus should at this stage shift to operations activities allowed within the Exchange  Does the exchange allow private network interconnections?  Are there requirements to connect to a central switch?  How secure is the facility?  Is there sufficient power, capacity at the switch, space for additional racks, real time staff support?  Is it easy to upgrade my presence over time?  Upgrading in this context means the ability to increase the speed of circuits into the exchange, the ability to purchase dark fiber, the ability to increase the number of racks and cross connects in the exchange, the ease of increasing the speed of interconnection. ISPs will prefer bandwidth-rich, ISP-friendly exchanges to those with restrictions over future operations.

29. Business Issues  Perhaps the most far-reaching business issues are strategic, as an ISP, ask these questions:  Do we want to support this exchange operator  Do the interests of Exchange partners enhance or conflict with ours?  “Bandwidth, strategic partner alliances, and corporate ties often override the technical justification.”  Will using this exchange support a competitor (contribute to their net income, their credibility, their positioning)?  Does the exchange have requirements (require use of their carrier or ISP services) that limit the market for services within the exchange?

30. Cost Issues  Crosses all other criteria  What is the cost of using this exchange?  What are the rack fees, cross connect fees, port fees, and installation fees?  What is the future operating fees going to be?  What are the motivations and parameters surrounding these fees?

31. Exchange Population Issues  These issues go to the other (side) benefits to using this exchange, Are there other ISPs there that are peering candidates?  Are there transit sales possible at the exchange?  In the context of the credibility issue discussed above, who will likely be at the exchange in the future, and when will the cost of participation equal the value of the interconnection (also known as the Critical Mass Point)?

32. Existing vs. New Exchanges  There are many existing exchanges and also emerging ones  Which one would benefit you as an ISP intending to go into peering with others?  Preferences for an New Exchange  Chronic traffic congestion can influence the decision to plan to peer in an existing exchange or wait until a better exchange opens  Long-term benefits (scalability) may lead to preferring a next generation exchange. However, all else considered equal, ISPs generally prefer an existing exchange to an emerging one

33. How important are these Issues?  These issues where not listed in any particular order An interview conducted by W.B. Norton with some ISPs in the U.S. reveals that they shared varied weightings of importance of each of these issues. To some the most important issues where business related, and others weighted more highly the flexibility of ongoing operations. Each ISP will place higher or lower importance on different issues and not surprisingly select operations environment based on their specific criteria. It was noted that they all seem to agree upon the general issues within which their criteria operate.

34. Summary  Looked through a rough description of the decision process ISPs follow to obtain peering relationships and establish peering  It focuses on the elements of the decision that lead to the selection of a specific exchange environment for peering  It can be concluded that Peering has some important advantages, with the issue of reduced cost and improvement in performance (lower latency) as the primary benefits  The goals of peering ISPs include: to get peering set up in the soonest possible time; to minimize the cost of the interconnection and their transit costs; to maximize the benefits of a systematic approach to peering; to execute the regional operations plan as strategy dictates (may be architecture/network development group goal) and finally, to fulfill obligations of larger business agreement.  It should also be of critical note to potential peers that the need to make proper decision before joining an Internet eXchange is very critical  One major challenge facing Peering Coordinators is the identification of potential peers and initiating discussions