Internet Exchange Points (IXPs) Philip Smith E2 Workshop, AfNOG 2005

Objectives To be able to explain what is an exchange point To be able to explain why ISPs participate in IXPs To understand why IXPs are important To review some current IXP designs used today To think about how to set up an exchange point in your environment

Introduction to Internet Exchange Points A bit of history What are they? Why use them?

A Bit of History… End of NSFnet – one major backbone move towards commercial Internet private companies selling their bandwidth need for coordination of routing exchange between providers Traffic from ISP A needs to get to ISP B Routing Arbiter project created to facilitate this

What is an Exchange Point Network Access Points (NAPs) established at end of NSFnet original “exchange points” Major providers connect their networks and exchange traffic High-speed network or ethernet switch Simple concept – any place where providers come together to exchange traffic

Internet Exchange Points ISPs connect at Exchange Points or Network Access Points to exchange traffic XP 1 XP 2 ISP A ISP B

Conceptual Diagram of an IXP ISP Router Exchange Point Medium

Why IXPs? Multiple service providers Each with Internet connectivity Internet A B

Why IXPs? Is not cost effective Backhaul issue causes cost to both parties Internet A B

Why IXPs? Domestic Interconnection Internet A B

Exchange Structures layer 2 models (the NAP or IXP) tailored bilateral policies

Why use an IXP? PEERING Shared medium vs. point-to-point Shared can exchange traffic with multiple peers at one location via one interface Point-to-Point for high volumes of traffic

Why use an IXP? KEEP LOCAL TRAFFIC LOCAL!!! ISPs within a region peer with each other at local exchange No need to have traffic go overseas only to come back Much reduced latency and increased performance

Why use an IXP? SAVES MONEY!!! Traffic going overseas means transit charges paid to your upstream ISP Money stays in local economy Used to provide better local infrastructure and services for customers Customers pay less for Internet access Therefore more customers sign up ISP has more customers, better business

Why use an IXP? VASTLY IMPROVES PERFORMANCE!!! Network RTTs between organisations in the local economy is measured in milliseconds, not seconds Packet loss becomes virtually non-existent Customers use the Internet for more products, services, and activities

Why use an IXP? Countries or regions with a successful IXP have a successful Internet economy Local traffic stays local Money spent on local ‘net infrastructure Service Quality not an issue All this attracts businesses, customers, and content providers

The Need for Exchanges AS 2 AS 1 US AFRICA Local traffic travels via the US

The Need for Exchanges AS 2 AS 1 US AFRICA Keep Local Traffic Local!

Exchange Point Design Ethernet switch Has superseded all other types of network devices for an IXP From the cheapest and smallest 12 or 24 port 10/100 switch To the largest 32 port 10GigEthernet switch

What can go wrong? Too many exchange points in one region competing exchanges defeats the purpose Becomes expensive for ISPs to connect to all of them  An IXP is not a competition, it’s not a business

What can go wrong? IXPs try to compete with their membership IXPs run as a closed privileged club For example: Interconnecting IXP locations with their own transit service Providing access to end users rather than just Service Providers Restrictive membership criteria (closed shop) Interfering with ISP business decisions e.g. Mandatory Multi-Lateral Peering

Exchange Point policies/politics AUPs Acceptable Use Policy Minimal rules for connection Fees? Some IXPs charge no fee Other IXPs charge cost recovery A few IXPs are commercial Nobody is obliged to peer Agreements left to ISPs, not mandated by IXP

Exchange Point etiquette Don’t point default route at another IXP participant Be aware of third-party next-hop Only announce your aggregate routes Filter! Filter! Filter! And do reverse path check

Exchange Point examples LINX in London, UK Ethernet switches AMS-IX in Amsterdam, NL Ethernet switches JPNAP in Tokyo, Japan Ethernet switches

Exchange Points in Africa CR-IX – Cairo, Egypt iBiX – Ibadan, Nigeria JINX – Johannesburg, South Africa KINIX – Kinshasa, Dem Rep of Congo KIXP – Nairobi, Kenya MOZIX – Maputo, Mozambique RINEX – Kigali, Rwanda SZIXP – Mbabane, Swaziland TIX – Dar es Salaam, Tanzania UiXP – Kampala, Uganda Source:

Features of IXPs Redundancy multiple switches Support NOC to provide 24x7 support for problems at the exchange DNS, Route Collector, Content & NTP servers CCTLD servers Content redistribution systems such as Akamai Route Collector – Routing Table view

Features of IXPs Location neutral co-location facilities Address space Peering LAN AS If using Route Server (Route servers) Statistics Traffic data – for membership

More info about IXPs Excellent resource for ip address allocation for exchanges, locations of XPs in the world, AUPs and other policies Another excellent resource of IXP locations, papers, IXP statistics, etc

Things to think about... Do you need to be at an Exchange Point? Would you want to start an Exchange Point? Would keeping local traffic local benefit your ISP? Would your environment (politically, etc.) support an Exchange Point?

Discussion How would you build an exchange point in your environment? Who would connect? What services would you provide? What policies would you enforce? What does your environment look like? Is it feasible to set up an IXP?

Important to Remember... Exchange Points can be as simple as an ethernet HUB!!!! Keeping local traffic local improves performance cheaper often simple to do!

Exercise Building an IXP

AS 1 BGP to provider AS 3 AS 5 AS 7 AS 9 AS 2 AS 4 AS 6 AS 8 AS 10 AS100 AS200

AS 1 AS 3 AS 5 AS 7 AS 9 AS 2 AS 4 AS 6 AS 8 AS /28 SWITCH Ethernet to IXP

Introduction to Route Collectors

Route Collector Background What is a Route Collector? Features of a Route Collector Purpose of a Route Collector IXP Design with a Route Collector

What is a Route Collector? Usually a router or Unix box running BGP Gathers routing information from service provider routers at an IXP Does not forward packets

Purpose of a Route Collector To provide a public view of the Routing Information available at the IXP Useful existing members to check functionality of BGP filters Useful for prospective members to check value of joining the IXP Useful for the Internet Operations community for troubleshooting purposes E.g.

Route Collector at an IXP IXP R3R2 R1 ROUTE Collector

Route Collector Requirements Router or Unix system running BGP Peers eBGP with every IXP member Accepts everything; Gives nothing Uses a private ASN Connects to IXP Transit LAN “Back end” connection Second Ethernet globally routed Connection to IXP Website for public access

Route Collector Implementation Most IXPs now implement some form of Route Collector Benefits already mentioned Great public relations tool Unsophisticated requirements Just runs BGP

Introduction to Route Servers Route Collector plus more

Route Server Background What is a Route Server? Features of a Route Server Advantages of using a Route Server Exchange Point Design with a Route Server

What is a Route Server? All the features of a Route Collector But also: Announces routes to participating IXP members according to their routing policy definitions Implemented using the same specification as for a Route Collector

Features of a Route Server Helps scale routing Simplifies Routing Processes on ISP Routers Insertion of RS Autonomous System Number in the Routing Path Handling of Multi-Exit Discriminator Route Flap Damping Uses Policy registered in IRR

Diagram of N-squared Mesh

With the Route Servers

RS based Exchange Point Routing Flow TRAFFIC FLOW ROUTING INFORMATION FLOW

Advantages of Using a Route Server Helps scale Routing Separation of Routing and Forwarding Simplify Routing Configuration Management on ISPs routers Enforce Good Routing Engineering Helps prevent the spread of bogus routing information!

Disadvantages of Using a Route Server ISPs lose direct policy control Dependent on 3 rd party for configuration and troubleshooting Insertion of RS Autonomous System Number in the Routing Path

Peering with the Route Servers Any ISP attached to an IXP can peer with the Route Servers ISP must register their policy in the Internet Routing Registry Most IXPs who provide the RS facility also provide a local IRR for policy registration Must use BGP

Things to think about... Would using a route server benefit you? Helpful when BGP knowledge is limited Avoids having to maintain a large number of eBGP peers

Introduction to the IRR The Internet Routing Registry

What is the Routing Registry Contact names, addresses and telephone numbers for an AS Routing policy for an AS (what other ASes does it connect to, which routes do they exchange) Information about routes (most important is which AS originates the route) Several other types of information

What is the Routing Registry? Distributed database collectively known as Internet Routing Registry (IRR) APNIC, RIPE, ARIN, RADB, etc Providers register routing policy Used for planning, debugging and generating backbone router configs

What is the Routing Registry? Can be used by anyone worldwide debugging configuring engineering routing addressing

What happens if I don’t use the IRR Routing Horror Stories AS7007 announcing bogus routes Inconsistent policy at network borders Peers and upstreams need physical notification of policy changes Mistakes easily made

So, I need to use the database because….. Filters generated off the IRR protect against inaccurate routing information Makes troubleshooting and debugging easier Keep track of policy Security Filter! Filter! Filter!!

Why Bother using the IRR? View of global routing policy in a single cooperatively maintained database to improve integrity of Internet’s routing generate router configs protect against inaccurate routing info distribution verification of Internet routing

Why Bother using the IRR? Many providers require that you register your policy (or they won’t peer with you)

Describing Policy Use the policy languages to describe your relationship with other Peers routes importing routes exporting specific policies interfaces, MEDs, communities register routes with origin AS

Querying the Database whois -h whois.ripe.net AS702 whois -h whois.ripe.net AS1849-MAINT whois -h whois.ripe.net

How to Register your IRR policy Register one or more maintainers Register AS and policy information Register Routes Describes your import and export policy At the very least, provides contact information

Router Configuration Currently configs by hand - slow and inaccurate Configuring routers using the IRR lots of tools available!!! IRRToolSet maintained by ISC route and Aspath filters. Import and export Filtering is a good thing...

Router Configuration IRR Configuration Machine IRR Database Server Router

How do I use the IRR to generate configurations Tools available to generate config files for most BGP implementations IRRToolSet Started off as RAToolSet as a project of ISI Moved to RIPE NCC custodianship and became IRRToolSet Enhanced to support RPSL (RFC2622) Now maintained by ISC

How do I participate? Set up your own registry Private for your ISP? Community for the region? Download the software (from ISC) Use one of the many public IRR systems Ask AfriNIC to set one up?

Things to think about... How would you register your policy? Try to describe it in an aut-num object How would registering your policy benefit you? The community?