IXP Training Workshops

IXP Training Workshops
Contact: WROU03_v1.0

Introduction to The Internet
IXP Training Workshops

Introduction to the Internet
Topologies and Definitions IP Addressing Internet Hierarchy Gluing it all together

Topologies and Definitions
What does all the jargon mean?

Some Icons… Router (layer 3, IP datagram forwarding) Ethernet switch
(layer 2, packet forwarding) Network Cloud

Routed Backbone ISPs build networks covering regions
Regions can cover a country, sub-continent, or even global Each region has points of presence built by the ISP Routers are the infrastructure Physical circuits run between routers Easy routing configuration, operation and troubleshooting The dominant topology used in the Internet today

MPLS Backbones Some ISPs & Telcos use Multi Protocol Label Switching (MPLS) MPLS is built on top of router infrastructure Used replace old ATM technology Tunnelling technology Main purpose is to provide VPN services Although these can be done just as easily with other tunnelling technologies such as GRE

Points of Presence PoP – Point of Presence vPoP – virtual PoP
Physical location of ISP’s equipment Sometimes called a “node” vPoP – virtual PoP To the end user, it looks like an ISP location In reality a back hauled access point Used mainly for consumer access networks Hub/SuperPoP – large central PoP Links to many PoPs

PoP Topologies Core routers Distribution routers Access routers
high speed trunk connections Distribution routers higher port density, aggregating network edge to the network core Access routers high port density, connecting the end users to the network Border routers connections to other providers Service routers hosting and servers Some functions might be handled by a single router

Network Operation Centre
Typical PoP Design Other ISPs Other ISPs Border Backbone link to another PoP Backbone link to another PoP Network Core Network Operation Centre Service Access Service Access Hosted Services Business Customer Aggregation ISP Services (DNS, Mail, News, FTP, WWW) Consumer Aggregation

More Definitions Transit Peering Default
Carrying traffic across a network Usually for a fee Peering Exchanging routing information and traffic Usually for no fee Sometimes called settlement free peering Default Where to send traffic when there is no explicit match in the routing table

Peering and Transit example
provider A Backbone Provider D IXP-East IXP-West provider B provider C A and B peer for free, but need transit arrangements with D to get packets to/from C

Private Interconnect ISP B ISP A Autonomous System 334 border border

Public Interconnect A location or facility where several ISPs are present and connect to each other over a common shared media Why? To save money, reduce latency, improve performance IXP – Internet eXchange Point NAP – Network Access Point

Public Interconnect Centralised (in one facility)
Distributed (connected via WAN links) Switched interconnect Ethernet (Layer 2) Technologies such as SRP, FDDI, ATM, Frame Relay, SMDS and even routers have been used in the past Each provider establishes peering relationship with other providers at IXP ISP border router peers with all other provider border routers

Public Interconnect ISP 1 ISP 4 IXP ISP 5 ISP 2 ISP 3 ISP 6
Each of these represents a border router in a different autonomous system

ISPs participating in Internet
Bringing all pieces together, ISPs: Build multiple PoPs in a distributed network Build redundant backbones Have redundant external connectivity Obtain transit from upstream providers Get free peering from local providers at IXPs

Example ISP Backbone Design
ISP Peer ISP Peer ISP Peer ISP Peer IXP Upstream1 Upstream 2 PoP 2 Upstream 2 Upstream1 Backbone Links PoP 1 PoP 3 Network Core PoP 4

Where to get address space and who from
IP Addressing Where to get address space and who from

IP Addressing Internet uses classless routing
Concept of IPv4 class A, class B or class C is no more Engineers talk in terms of prefix length, for example the class B is now called /16. All routers must be CIDR capable Classless InterDomain Routing RFC1812 – Router Requirements

IP Addressing Pre-CIDR (before 1994)
Big networks got a class A Medium networks got a class B Small networks got a class C The CIDR IPv4 years (1994 to 2010) Sizes of IPv4 allocations/assignments made according to demonstrated need – CLASSLESS IPv6 adoption (from 2011) The size of IPv4 address allocations and assignments are now very limited as IANA’s free pool has run out

IP Addressing IP Address space is a resource shared amongst all Internet users Regional Internet Registries delegated allocation responsibility by the IANA AfriNIC, APNIC, ARIN, LACNIC & RIPE NCC are the five RIRs RIRs allocate address space to ISPs and Local Internet Registries ISPs/LIRs assign address space to end customers or other ISPs All usable IPv4 address space has been allocated to the RIRs by the IANA (February 2011) The time for IPv6 is now

Non-portable Address Space
“Provider Aggregatable” or “PA Space” Customer uses RIR member’s address space while connected to Internet Customer has to renumber to change ISP Aids control of size of Internet routing table Need to fragment provider block when multihoming PA space is allocated to the RIR member All assignments made by the RIR member to end sites are announced as an aggregate to the rest of the Internet

Portable Address Space
“Provider Independent” or “PI Space” Customer gets or has address space independent of ISP Customer keeps addresses when changing ISP Is very bad for size of Internet routing table Is very bad for scalability of the routing system  PI space is rarely distributed by the RIRs

Internet Hierarchy The pecking order

High Level View of the Global Internet
Global Providers Regional Provider 1 Regional Provider 2 Content Provider 1 Content Provider 2 Access Provider 1 Access Provider 2 R4 Internet Exchange Point Customer Networks

Detailed View of the Global Internet
Global Transit Providers Connect to each other Provide connectivity to Regional Transit Providers Regional Transit Providers Provide connectivity to Content Providers Provide connectivity to Access Providers Access Providers Connect to each other across IXPs (free peering) Provide access to the end user

Categorising ISPs Tier 1 ISP Tier 1 ISP $ Tier 1 ISP Tier 1 ISP
IXP IXP Tier 3 ISP Tier 3 ISP Tier 3 ISP Tier 3 ISP Tier 3 ISP Tier 3 ISP

Inter-provider relationships
Peering between equivalent sizes of service providers (e.g. Tier 2 to Tier 2) Shared cost private interconnection, equal traffic flows No cost peering Peering across exchange points If convenient, of mutual benefit, technically feasible Fee based peering Unequal traffic flows, “market position”

NB: is not related to where an ISP is in the hierarchy
Default Free Zone The default free zone is made up of Internet routers which have explicit routing information about the rest of the Internet, and therefore do not need to use a default route NB: is not related to where an ISP is in the hierarchy

Gluing it together

Gluing it together Who runs the Internet? How does it keep working?
No one (Definitely not ICANN, nor the RIRs, nor the US,…) How does it keep working? Inter-provider business relationships and the need for customer reachability ensures that the Internet by and large functions for the common good Any facilities to help keep it working? Not really. But… Engineers keep working together!

Engineers keep talking to each other...
North America NANOG (North American Network Operators Group) NANOG meetings and mailing list Latin America Foro de Redes NAPLA LACNOG – supported by LACNIC Middle East MENOG (Middle East Network Operators Group)

Asia & Pacific APRICOT annual conference APOPS & APNIC-TALK mailing lists mailman.apnic.net/mailman/listinfo/apops mailman.apnic.net/mailman/listinfo/apnic-talk PacNOG (Pacific NOG) mailman.apnic.net/mailman/listinfo/pacnog SANOG (South Asia NOG) to

Europe RIPE meetings, working groups and mailing lists e.g. Routing WG: Africa AfNOG meetings and mailing list And many in-country ISP associations and NOGs IETF meetings and mailing lists

Summary Topologies and Definitions IP Addressing Internet Hierarchy
PA versus PI address space Internet Hierarchy Local, Regional, Global Transit Providers IXPs Gluing it all together Engineers cooperate, common business interests

Introduction to The Internet
ISP Training Workshops

ISP Training Workshops
The Value of Peering ISP Training Workshops

The Internet Internet is made up of ISPs of all shapes and sizes
Some have local coverage (access providers) Others can provide regional or per country coverage And others are global in scale These ISPs interconnect their businesses They don’t interconnect with every other ISP (over distinct autonomous networks) – won’t scale They interconnect according to practical and business needs Some ISPs provide transit to others They interconnect other ISP networks

Categorising ISPs Global ISP Global ISP $ Global ISP Global ISP
Regional ISP Regional ISP Regional ISP Regional ISP IXP IXP Access ISP Access ISP Access ISP Access ISP Access ISP Access ISP

Peering and Transit Transit Peering Carrying traffic across a network
Usually for a fee Example: Access provider connects to a regional provider Peering Exchanging routing information and traffic Usually for no fee Sometimes called settlement free peering Example: Regional provider connects to another regional provider

Private Interconnect Two ISPs connect their networks over a private link Can be peering arrangement No charge for traffic Share cost of the link Can be transit arrangement One ISP charges the other for traffic One ISP (the customer) pays for the link ISP 1 ISP 2

Public Interconnect Several ISPs meeting in a common neutral location and interconnect their networks Usually is a peering arrangement between their networks ISP 1 ISP 2 ISP 3 ISP 6 IXP ISP 5 ISP 4

ISP Goals Minimise the cost of operating the business Transit Peering
ISP has to pay for circuit (international or domestic) ISP has to pay for data (usually per Mbps) Repeat for each transit provider Significant cost of being a service provider Peering ISP shares circuit cost with peer (private) or runs circuit to public peering point (one off cost) No need to pay for data Reduces transit data volume, therefore reducing cost

Transit – How it works Small access provider provides Internet access for a city’s population Mixture of dial up, wireless and fixed broadband Possibly some business customers Possibly also some Internet cafes How do their customers get access to the rest of the Internet? ISP buys access from one, two or more larger ISPs who already have visibility of the rest of the Internet This is transit – they pay for the physical connection to the upstream and for the traffic volume on the link

Peering – How it works If two ISPs are of equivalent sizes, they have:
Equivalent network infrastructure coverage Equivalent customer size Similar content volumes to be shared with the Internet Potentially similar traffic flows to each other’s networks This makes them good peering partners If they don’t peer They both have to pay an upstream provider for access to each other’s network/customers/content Upstream benefits from this arrangement, the two ISPs both have to fund the transit costs

The IXP’s role Private peering makes sense when there are very few equivalent players Connecting to one other ISP costs X Connecting to two other ISPs costs 2 times X Connecting to three other ISPs costs 3 times X Etc… (where X is half the circuit cost plus a port cost) The more private peers, the greater the cost IXP is a more scalable solution to this problem

The IXP’s role Connecting to an IXP
ISP costs: one router port, one circuit, and one router to locate at the IXP Some IXPs charge annual “maintenance fees” The maintenance fee has potential to significantly influence the cost balance for an ISP Generally connecting to an IXP and peering there becomes cost effective when there are at least three other peers The real $ amount varies from region to region, IXP to IXP

Who peers at an IXP? Access Providers Regional Providers
Don’t have to pay their regional provider transit fees for local traffic Keeps latency for local traffic low ‘Unlimited’ bandwidth through the IXP (compared with costly and limited bandwidth through transit provider) Regional Providers Don’t have to pay their global provider transit for local and regional traffic Keeps latency for local and regional traffic low ‘Unlimited’ bandwidth through the IXP (compared with costly and limited bandwidth through global provider)

The IXP’s role Global Providers can be located close to IXPs
Attracted by the potential transit business available Advantageous for access & regional providers They can peer with other similar providers at the IXP And in the same facility pay for transit to their regional or global provider (Not across the IXP fabric, but a separate connection) IXP Transit Access

Connectivity Decisions
Transit Almost every ISP needs transit to reach rest of Internet One provider = no redundancy Two providers: ideal for traffic engineering as well as redundancy Three providers = better redundancy, traffic engineering gets harder More then three = diminishing returns, rapidly escalating costs and complexity Peering Means low (or zero) cost access to another network Private or Public Peering (or both)

Transit Goals Minimise number of transit providers
But maintain redundancy 2 is ideal, 4 or more is bad Aggregate capacity to transit providers More aggregated capacity means better value Lower cost per Mbps 4x 45Mbps circuits to 4 different ISPs will almost always cost more than 2x 155Mbps circuits to 2 different ISPs Yet bandwidth of latter (310Mbps) is greater than that of former (180Mbps) and is much easier to operate

Peering or Transit? How to choose? Or do both?
It comes down to cost of going to an IXP Free peering Paying for transit from an ISP co-located in same facility, or perhaps close by Or not going to an IXP and paying for the cost of transit directly to an upstream provider There is no right or wrong answer, someone has to do the arithmetic

Private or Public Peering
Private peering Scaling issue, with costs, number of providers, and infrastructure provisioning Public peering Makes sense the more potential peers there are (more is usually greater than “two”) Which public peering point? Local Internet Exchange Point: great for local traffic and local peers Regional Internet Exchange Point: great for meeting peers outside the locality, might be cheaper than paying transit to reach the same consumer base

Local Internet Exchange Point
Defined as a public peering point serving the local Internet industry Local means where it becomes cheaper to interconnect with other ISPs at a common location than it is to pay transit to another ISP to reach the same consumer base Local can mean different things in different regions!

Regional Internet Exchange Point
These are also “local” Internet Exchange Points But also attract regional ISPs and ISPs from outside the locality Regional ISPs peer with each other And show up at several of these Regional IXPs Local ISPs peer with ISPs from outside the locality They don’t compete in each other’s markets Local ISPs don’t have to pay transit costs ISPs from outside the locality don’t have to pay transit costs Quite often ISPs of disparate sizes and influences will happily peer – to defray transit costs

Which IXP? How many routes are available?
What is traffic to & from these destinations, and by how much will it reduce cost of transit? What is the cost of co-lo space? If prohibitive or space not available, pointless choosing this IXP What is the cost of running a circuit to the location? If prohibitive or competitive with transit costs, pointless choosing this IXP What is the cost of remote hands/assistance? If no remote hands, doing maintenance is challenging and potentially costly with a serious outage

Example: South Asian ISP @ LINX
Date: October 2011 Facts: Route Server plus bilateral peering offers 81k prefixes IXP traffic averages 55Mbps/15Mbps Transit traffic averages 35Mbps/3Mbps Analysis: 61% of inbound traffic comes from 81k prefixes available by peering 39% of inbound traffic comes from remaining 287k prefixes from transit provider

Example: South Asian ISP @ HKIX
Date: October 2011 Facts: Route Server plus bilateral peering offers 34k prefixes IXP traffic is 130Mbps/30Mbps Transit traffic is 125Mbps/40Mbps Analysis: 51% of inbound traffic comes from 42k prefixes available by peering 49% of inbound traffic comes from remaining 326k prefixes from transit provider

Example: South Asian ISP
Summary: Traffic by Peering: 185Mbps/45Mbps Traffic by Transit: 160Mbps/43Mbps 54% of incoming traffic is by peering 52% of outbound traffic is by peering

Example: South Asian ISP
Router at remote co-lo Benefits: can select peers, easy to swap transit providers Costs: co-lo space and remote hands Servers at remote co-lo Benefits: mail filtering, content caching, etc Overall advantage: Can control what goes on the expensive connectivity “back to home”

Value propositions Peering at a local IXP
Reduces latency & transit costs for local traffic Improves Internet quality perception Participating at a Regional IXP A means of offsetting transit costs Managing connection back to home network Improving Internet Quality perception for customers

Summary Benefits of peering Local versus Regional IXPs Private
Internet Exchange Points Local versus Regional IXPs Local services local traffic Regional helps defray transit costs

Single International Transit Versus Local IXP + Regional IXP + Transit
Worked Example Single International Transit Versus Local IXP + Regional IXP + Transit

Worked Example ISP A is local access provider
Some business customers (around 200 fixed links) Some co-located content provision (datacentre with 100 servers) Some consumers on broadband (5000 DSL/Cable/Wireless) Some consumers on dial (1000 on V.34 type speeds) They have a single transit provider Connect with a 16Mbps international leased link to their transit’s PoP Transit link is highly congested

Worked Example (2) There are two other ISPs serving the same locality
There is no interconnection between any of the three ISPs Local traffic (between all 3 ISPs) is traversing International connections Course of action for our ISP: Work to establish local IXP Establish presence at overseas co-location First Step Assess local versus international traffic ratio Use NetFlow on border router connecting to transit provider

Worked Example (3) Local/Non-local traffic ratio
Local = traffic going to other two ISPs Non-local = traffic going elsewhere Example: balance is 30:70 Of 16Mbps, that means 5Mbps could stay in country and not congest International circuit 16Mbps transit costs $50 per Mbps per month traffic charges = $250 per month, or $3000 per year for local traffic Circuit costs $100k per year: $30k is spent on local traffic Total is $33k per year for local traffic

Worked Example (4) IXP cost:
Simple 8 port 10/100 managed switch plus co-lo space over 3 years could be around US$30k total; or $3k per year per ISP One router to handle 5Mbps (e.g. 2801) would be around $3k (good for 3 years) One local 10Mbps circuit from ISP location to IXP location would be around $5k per year, no traffic charges Per ISP total: $9k Somewhat cheaper than $33k Business case for local peering is straightforward - $24k saving per annum

Worked Example (5) After IXP establishment
5Mbps removed from International link Leaving 5Mbps for more International traffic – and that fills the link within weeks of the local traffic being removed Next step is to assess transit charges and optimise costs ISPs visits several major regional IXPs Assess routes available Compares routes available with traffic generated by those routes from its Netflow data Discovers that 30% of traffic would transfer to one IXP via peering

Worked Example (6) Costs:
Router for Regional IXP (e.g. 2801) at $3k over three years Co-lo space at Regional IXP venue at $3k per year Best price for transit at the Regional IXP venue by competitive tender is $30 per Mbps per month, plus $1k port charge 30% of traffic offloads to IXP, leaving 70% of 16Mbps to transit provider = $330 per month, or $5k per annum Total with this model is $9k per year, plus the cost of the circuit (still $100k) Compare this with paying $50 per Mbps per month to the transit provider = $10k per annum (plus cost of the circuit)

Worked Example (7) Result: Bonuses: ISP co-locates at Regional IXP
Pays reduced transit charges to transit provider (competitive tender) Pays no charges for traffic across Regional IXP Bonuses: Rate limits on router at Regional IXP Co-lo Can prioritise congestion dependent on customer demands Install servers at Regional IXP co-lo facility Filters (spam and viruses) – relieves some capacity on link Caches content – relieves a little more capacity on link

Conclusion Within the original costs of having one international transit provider: ISP has turned up at the local IXP and offloaded local traffic for free ISP has turned up at a major regional IXP and offloaded traffic, avoiding paying transit charges to transit provider ISP has reduced remaining transit charges by competitive tender at the regional IXP co-location facility Caveat These numbers are typical of the Internet today As ever, your mileage may vary – but do the financial calculations first and in the context of potential technical advantages too

The Value of Peering ISP Training Workshops

Introduction to OSPF ISP Training Workshops

OSPF Open Shortest Path First Link state or SPF technology
Developed by OSPF working group of IETF (RFC 1247) OSPFv2 standard described in RFC2328 Designed for: TCP/IP environment Fast convergence Variable-length subnet masks Discontiguous subnets Incremental updates Route authentication Runs on IP, Protocol 89

Link State Z’s Link State Q’s Link State Z Q Y X
2 Q Y B Z 13 C X 13 X Topology Information is kept in a Database separate from the Routing Table X’s Link State 44

Link State Routing Neighbour discovery
Constructing a Link State Packet (LSP) Distribute the LSP (Link State Announcement – LSA) Compute routes On network failure New LSPs flooded All routers recompute routing table

Low Bandwidth Utilisation
LSA X R1 LSA Only changes propagated Uses multicast on multi-access broadcast networks

X Fast Convergence Detection Plus LSA/SPF
Known as the Dijkstra Algorithm Alternate Path R2 X N1 N2 R1 R3 Primary Path

X Fast Convergence Finding a new route LSA flooded throughout area
Acknowledgement based Topology database synchronised Each router derives routing table to destination network LSA X N1 R1

OSPF Areas Area is a group of contiguous hosts and networks
Reduces routing traffic Per area topology database Invisible outside the area Backbone area MUST be contiguous All other areas must be connected to the backbone Area 1 Area 2 Area 3 R1 R2 R3 R6 Area 4 R5 R4 R7 R8 Ra Rd Rb Rc Area 0 Backbone Area 27

Virtual Links between OSPF Areas
Virtual Link is used when it is not possible to physically connect the area to the backbone ISPs avoid designs which require virtual links Increases complexity Decreases reliability and scalability Area 1 R3 R6 Area 4 R5 R4 R7 R8 Ra Rd Rb Rc Area 0 Backbone Area 27

Classification of Routers
Rd Ra Rb Rc IR ABR/BR IR/BR ASBR To other AS Area 1 Area 0 Area 2 Area 3 Internal Router (IR) Area Border Router (ABR) Backbone Router (BR) Autonomous System Border Router (ASBR) 32

OSPF Route Types Intra-area Route Inter-area Route External Route IR
Rd Ra Rb Rc IR ABR/BR ASBR To other AS Area 1 Area 0 Area 2 Area 3 Intra-area Route all routes inside an area Inter-area Route routes advertised from one area to another by an Area Border Router External Route routes imported into OSPF from other protocol or static routes

External Routes Prefixes which are redistributed into OSPF from other protocols Flooded unaltered throughout the AS Recommendation: Avoid redistribution!! OSPF supports two types of external metrics Type 1 external metrics Type 2 external metrics (Cisco IOS default) RIP EIGRP BGP Static Connected etc. OSPF R2 Redistribute 18

External Routes Type 1 external metric: metrics are added to the summarised internal link cost to N1 External Cost = 1 Cost = 10 R2 R1 to N1 External Cost = 2 Cost = 8 R3 Network N1 Type 1 11 10 Next Hop R2 R3 Selected Route 19

External Routes Type 2 external metric: metrics are compared without adding to the internal link cost Cost = 10 to N1 External Cost = 1 External Cost = 2 Cost = 8 Selected Route R3 R1 R2 Network N1 Type 1 1 2 Next Hop 19

Topology/Link State Database
A router has a separate LS database for each area to which it belongs All routers belonging to the same area have identical database SPF calculation is performed separately for each area LSA flooding is bounded by area Recommendation: Limit the number of areas a router participates in!! 1 to 3 is fine (typical ISP design) >3 can overload the CPU depending on the area topology complexity 30

The Hello Protocol Responsible for establishing and maintaining neighbour relationships Elects designated router on multi-access networks Hello 38

The Hello Packet Contains: Router priority Hello interval
Router dead interval Network mask List of neighbours DR and BDR Options: E-bit, MC-bit,… (see A.2 of RFC2328) Hello 39

Designated Router There is ONE designated router per multi-access network Generates network link advertisements Assists in database synchronization Backup Designated Router Designated Router Designated Router Backup Designated Router

Designated Router by Priority
Configured priority (per interface) ISPs configure high priority on the routers they want as DR/BDR Else determined by highest router ID Router ID is 32 bit integer Derived from the loopback interface address, if configured, otherwise the highest IP address R1 DR R2 R1 Router ID = R2 Router ID = 41

Neighbouring States Full Routers are fully adjacent
Databases synchronised Relationship to DR and BDR Full DR BDR

Neighbouring States 2-way Router sees itself in other Hello packets
DR selected from neighbours in state 2-way or greater 2-way DR BDR

When to Become Adjacent
Underlying network is point to point Underlying network type is virtual link The router itself is the designated router or the backup designated router The neighbouring router is the designated router or the backup designated router 44

LSAs Propagate Along Adjacencies
DR BDR LSAs acknowledged along adjacencies 45

Broadcast Networks IP Multicast used for Sending and Receiving Updates
All routers must accept packets sent to AllSPFRouters ( ) All DR and BDR routers must accept packets sent to AllDRouters ( ) Hello packets sent to AllSPFRouters (Unicast on point-to-point and virtual links)

Routing Protocol Packets
Share a common protocol header Routing protocol packets are sent with type of service (TOS) of 0 Five types of OSPF routing protocol packets Hello – packet type 1 Database description – packet type 2 Link-state request – packet type 3 Link-state update – packet type 4 Link-state acknowledgement – packet type 5 5

Different Types of LSAs
Six distinct type of LSAs Type 1 : Router LSA Type 2 : Network LSA Type 3 & 4: Summary LSA Type 5 & 7: External LSA (Type 7 is for NSSA) Type 6: Group membership LSA Type 9, 10 & 11: Opaque LSA (9: Link-Local, 10: Area)

Router LSA (Type 1) Describes the state and cost of the router’s links to the area All of the router’s links in an area must be described in a single LSA Flooded throughout the particular area and no more Router indicates whether it is an ASBR, ABR, or end point of virtual link 21

Network LSA (Type 2) Generated for every transit broadcast and NBMA network Describes all the routers attached to the network Only the designated router originates this LSA Flooded throughout the area and no more 25

Summary LSA (Type 3 and 4) Describes the destination outside the area but still in the AS Flooded throughout a single area Originated by an ABR Only inter-area routes are advertised into the backbone Type 4 is the information about the ASBR 28

External LSA (Type 5 and 7)
Defines routes to destination external to the AS Default route is also sent as external Two types of external LSA: E1: Consider the total cost up to the external destination E2: Considers only the cost of the outgoing interface to the external destination (Type 7 LSAs used to describe external LSA for one specific OSPF area type) 32

Inter-Area Route Summarisation
Prefix or all subnets Prefix or all networks ‘Area range’ command R2 Backbone Area 0 With summarisation Network 1 Next Hop R1 (ABR) R1 Area 1 Without summarisation Network 1.A 1.B 1.C Next Hop R1 1.A 1.B 1.C 16

No Summarisation Specific Link LSA advertised out of each area
Link state changes propagated out of each area 3.A 3.B 3.C 3.D 2.A 2.B 2.C 2.D 1.A 1.B 1.C 1.D Area 0

With Summarisation Only summary LSA advertised out of each area
Link state changes do not propagate out of the area 3.A 3.B 3.C 3.D 2.A 2.B 2.C 2.D 1.A 1.B 1.C 1.D 1 Area 0 2 3

No Summarisation Specific Link LSA advertised in to each area
Link state changes propagated in to each area 3.A 3.B 3.C 3.D 2.A 2.B 2.C 2.D 1.A 1.B 1.C 1.D 2.A 2.B 2.C 2.D 3.A 3.B 3.C 3.D Area 0 1.A 1.B 1.C 1.D

With Summarisation Only summary link LSA advertised in to each area
Link state changes do not propagate in to each area 3.A 3.B 3.C 3.D 2.A 2.B 2.C 2.D 1.A 1.B 1.C 1.D 2 3 Area 0 1

Types of Areas Regular Stub Totally Stubby Not-So-Stubby
Only “regular” areas are useful for ISPs Other area types handle redistribution of other routing protocols into OSPF – ISPs don’t redistribute anything into OSPF The next slides describing the different area types are provided for information only

Regular Area (Not a Stub)
From Area 1’s point of view, summary networks from other areas are injected, as are external networks such as X.1 3.A 3.B 3.C 3.D 2.A 2.B 2.C 2.D 1.A 1.B 1.C 1.D 2 3 Area 0 1 ASBR External networks X.1 X.1 X.1 X.1 X.1 X.1 X.1

Normal Stub Area Summary networks, default route injected
Command is area x stub 3.A 3.B 3.C 3.D 2.A 2.B 2.C 2.D 1.A 1.B 1.C 1.D 2 3 Area 0 1 ASBR External networks X.1 Default Default Default X.1 X.1 X.1

Totally Stubby Area Only a default route injected
Default path to closest area border router Command is area x stub no-summary 3.A 3.B 3.C 3.D 2.A 2.B 2.C 2.D 1.A 1.B 1.C 1.D Area 0 1 3 2 ASBR External networks X.1 Totally Stubby Area Default Default Default X.1 X.1 X.1

Not-So-Stubby Area ASBR Not-So-Stubby Area Area 0
Capable of importing routes in a limited fashion Type-7 LSA’s carry external information within an NSSA NSSA Border routers translate selected type-7 LSAs into type-5 external network LSAs 3.A 3.B 3.C 3.D 2.A 2.B 2.C 2.D 1.A 1.B 1.C 1.D Area 0 1 3 2 ASBR External networks X.1 Not-So-Stubby Area Default Default X.2 Default X.2 X.2 X.2 X.1 X.1 X.2 X.1 External networks

ISP Use of Areas ISP networks use: Backbone area Regular area
No partitioning Summarisation of point to point link addresses used within areas Loopback addresses allowed out of regular areas without summarisation (otherwise iBGP won’t work)

Addressing for Areas Area 0 network range Area 1 network range Area 2 network range Area 3 network range Assign contiguous ranges of subnets per area to facilitate summarisation 69

Summary Fundamentals of Scalable OSPF Network Design Area hierarchy
DR/BDR selection Contiguous intra-area addressing Route summarisation Infrastructure prefixes only 73

Introduction to OSPF ISP Training Workshops

Deploying OSPF for ISPs

Agenda OSPF Design in SP Networks Adding Networks in OSPF
OSPF in Cisco’s IOS

As applicable to Service Provider Networks
OSPF Design As applicable to Service Provider Networks

Service Providers SP networks are divided into PoPs
PoPs are linked by the backbone Transit routing information is carried via iBGP IGP is only used to carry the next hop for BGP Optimal path to the next hop is critical

SP Architecture IP Backbone
Major routing information is ~430K prefixes via BGP Largest known IGP routing table is ~9–10K Total of 440K 10K/440K is 2½% of IGP routes in an ISP network A very small factor but has a huge impact on network convergence! IP Backbone POP Area 1/L1 BGP 1 Area 6/L1 Area 5/L1 Area 4/L1 Area 2/L1 Area 3/L1 Area0/L2

SP Architecture Regional Core You can reduce the IGP size from 10K to approx the number of routers in your network This will bring really fast convergence Optimise where you must and summarise where you can Stops unnecessary flapping RR IGP Access customer customer customer

OSPF Design: Addressing
OSPF Design and Addressing go together Objective is to keep the Link State Database lean Create an address hierarchy to match the topology Use separate Address Blocks for loopbacks, network infrastructure, customer interfaces & customers Customer Address Space Infrastructure PtP Links Loopbacks

OSPF Design: Addressing
Minimising the number of prefixes in OSPF: Number loopbacks out of a contiguous address block But do not summarise these across area boundaries: iBGP peer addresses need to be in the IGP Use contiguous address blocks per area for infrastructure point-to-point links Use area range command on ABR to summarise With these guidelines: Number of prefixes in area 0 will then be very close to the number of routers in the network It is critically important that the number of prefixes and LSAs in area 0 is kept to the absolute minimum

OSPF Design: Areas Examine physical topology
Is it meshed or hub-and-spoke? Use areas and summarisation This reduces overhead and LSA counts (but watch next-hop for iBGP when summarising) Don’t bother with the various stub areas No benefits for ISPs, causes problems for iBGP Push the creation of a backbone Reduces mesh and promotes hierarchy

OSPF Design: Areas One SPF per area, flooding done per area
Watch out for overloading ABRs Avoid externals in OSPF DO NOT REDISTRIBUTE into OSPF External LSAs flood through entire network Different types of areas do different flooding Normal areas Stub areas Totally stubby (stub no-summary) Not so stubby areas (NSSA) 50

OSPF Design: Areas Area 0 must be contiguous
Do NOT use virtual links to join two Area 0 islands Traffic between two non-zero areas always goes via Area 0 There is no benefit in joining two non-zero areas together Avoid designs which have two non-zero areas touching each other (Typical design is an area per PoP, with core routers being ABR to the backbone area 0)

OSPF Design: Summary Think Redundancy Too much redundancy…
Dual Links out of each area – using metrics (cost) for traffic engineering Too much redundancy… Dual links to backbone in stub areas must be the same cost – other wise sub-optimal routing will result Too Much Redundancy in the backbone area without good summarisation will effect convergence in the Area 0

OSPF Areas: Migration Where to place OSPF Areas?
Follow the physical topology! Remember the earlier design advice Configure area at a time! Start at the outermost edge of the network Log into routers at either end of a link and change the link from Area 0 to the chosen Area Wait for OSPF to re-establish adjacencies And then move onto the next link, etc Important to ensure that there is never an Area 0 island anywhere in the migrating network

OSPF Areas: Migration Area 0
B C Area 0 D Area 10 E G F Migrate small parts of the network, one area at a time Remember to introduce summarisation where feasible With careful planning, the migration can be done with minimal network downtime

OSPF for Service Providers
Configuring OSPF & Adding Networks

OSPF: Configuration Starting OSPF in Cisco’s IOS
router ospf 100 Where “100” is the process ID OSPF process ID is unique to the router Gives possibility of running multiple instances of OSPF on one router Process ID is not passed between routers in an AS Many ISPs configure the process ID to be the same as their BGP Autonomous System Number

OSPF: Establishing Adjacencies
Cisco IOS OSPFv2 automatically tries to establish adjacencies on all defined interfaces (or subnets) Best practice is to disable this Potential security risk: sending OSPF Hellos outside of the autonomous system, and risking forming adjacencies with external networks Example: Only POS4/0 interface will attempt to form an OSPF adjacency router ospf 100 passive-interface default no passive-interface POS4/0

OSPF: Adding Networks Option One
Redistribution: Applies to all connected interfaces on the router but sends networks as external type-2s – which are not summarised router ospf 100 redistribute connected subnets Do NOT do this! Because: Type-2 LSAs flood through entire network These LSAs are not all useful for determining paths through backbone; they simply take up valuable space

OSPF: Adding Networks Option Two
Per link configuration – from IOS 12.4 onwards OSPF is configured on each interface (same as ISIS) Useful for multiple subnets per interface interface POS 4/0 ip address ip address secondary ip ospf 100 area 0 ! router ospf 100 passive-interface default no passive-interface POS 4/0

OSPF: Adding Networks Option Three
Specific network statements Every active interface with a configured IP address needs an OSPF network statement Interfaces that will have no OSPF neighbours need passive-interface to disable OSPF Hello’s That is: all interfaces connecting to devices outside the ISP backbone (i.e. customers, peers, etc) router ospf 100 network area 51 network area 51 passive-interface Serial 1/0

OSPF: Adding Networks Option Four
Network statements – wildcard mask Every active interface with configured IP address covered by wildcard mask used in OSPF network statement Interfaces covered by wildcard mask but having no OSPF neighbours need passive-interface (or use passive-interface default and then activate the interfaces which will have OSPF neighbours) router ospf 100 network area 51 passive-interface default no passive interface POS 4/0

OSPF: Adding Networks Recommendations
Don’t ever use Option 1 Use Option 2 if supported; otherwise: Option 3 is fine for core/infrastructure routers Doesn’t scale too well when router has a large number of interfaces but only a few with OSPF neighbours  solution is to use Option 3 with “no passive” on interfaces with OSPF neighbours Option 4 is preferred for aggregation routers Or use iBGP next-hop-self Or even ip unnumbered on external point-to-point links

OSPF: Adding Networks Example One (Cisco IOS ≥ 12.4)
Aggregation router with large number of leased line customers and just two links to the core network: interface loopback 0 ip address ip ospf 100 area 0 interface POS 0/0 ip address interface POS 1/0 ip address interface serial 2/0:0 ... ip unnumbered loopback 0 ! Customers connect here ^^^^^^^ router ospf 100 passive-interface default no passive interface POS 0/0 no passive interface POS 1/0

OSPF: Adding Networks Example One (Cisco IOS < 12.4)
Aggregation router with large number of leased line customers and just two links to the core network: interface loopback 0 ip address interface POS 0/0 ip address interface POS 1/0 ip address interface serial 2/0:0 ... ip unnumbered loopback 0 ! Customers connect here ^^^^^^^ router ospf 100 network area 51 network area 51 network area 51 passive-interface default no passive interface POS 0/0 no passive interface POS 1/0

OSPF: Adding Networks Example Two (Cisco IOS ≥ 12.4)
Core router with only links to other core routers: interface loopback 0 ip address ip ospf 100 area 0 interface POS 0/0 ip address interface POS 1/0 ip address interface POS 2/0 ip address interface POS 2/1 ip address router ospf 100 passive interface loopback 0

OSPF: Adding Networks Example Two (Cisco IOS < 12.4)
Core router with only links to other core routers: interface loopback 0 ip address interface POS 0/0 ip address interface POS 1/0 ip address interface POS 2/0 ip address interface POS 2/1 ip address router ospf 100 network area 0 network area 0 network area 0 network area 0 network area 0 passive interface loopback 0

OSPF: Adding Networks Summary
Key Theme when selecting a technique: Keep the Link State Database Lean Increases Stability Reduces the amount of information in the Link State Advertisements (LSAs) Speeds Convergence Time

Useful features for ISPs
OSPF in Cisco IOS Useful features for ISPs

Areas An area is stored as a 32-bit field:
Defined in IPv4 address format (i.e. Area ) Can also be defined using single decimal value (i.e. Area 0) reserved for the backbone area Area 3 Area 0 Area 2 Area 1

Logging Adjacency Changes
The router will generate a log message whenever an OSPF neighbour changes state Syntax: [no] [ospf] log-adjacency-changes (OSPF keyword is optional, depending on IOS version) Example of a typical log message: %OSPF-5-ADJCHG: Process 1, Nbr on Ethernet0 from LOADING to FULL, Loading Done

Number of State Changes
The number of state transitions is available via SNMP (ospfNbrEvents) and the CLI: show ip ospf neighbor [type number] [neighbor-id] [detail] Detail—(Optional) Displays all neighbours given in detail (list all neighbours). When specified, neighbour state transition counters are displayed per interface or neighbour ID

State Changes (Continued)
To reset OSPF-related statistics, use the clear ip ospf counters command This will reset neighbour state transition counters per interface or neighbour id clear ip ospf counters [neighbor [<type number>] [neighbor-id]]

Router ID If the loopback interface exists and has an IP address, that is used as the router ID in routing protocols – stability! If the loopback interface does not exist, or has no IP address, the router ID is the highest IP address configured – danger! OSPF sub command to manually set the Router ID: router-id <ip address>

Cost & Reference Bandwidth
Bandwidth used in Metric calculation Cost = 108/bandwidth Not useful for interface bandwidths > 100 Mbps Syntax: ospf auto-cost reference-bandwidth <reference-bw> Default reference bandwidth still 100 Mbps for backward compatibility Most ISPs simply choose to develop their own cost strategy and apply to each interface type

Cost: Example Strategy
100GE 100Gbps cost = 1 40GE/OC768 40Gbps cost = 2 10GE/OC192 10Gbps cost = 5 OC48 2.5Gbps cost = 10 GigEthernet 1Gbps cost = 20 OC12 622Mbps cost = 50 OC3 155Mbps cost = 100 FastEthernet 100Mbps cost = 200 Ethernet 10Mbps cost = 500 E1 2Mbps cost = 1000

Default routes Originating a default route into OSPF
default-information originate metric <n> Will originate a default route into OSPF if there is a matching default route in the Routing Table (RIB) The optional always keyword will always originate a default route, even if there is no existing entry in the RIB

Clear/Restart OSPF clear commands clear ip ospf [pid] redistribution
If no process ID is given, all OSPF processes on the router are assumed clear ip ospf [pid] redistribution This command clears redistribution based on OSPF routing process ID clear ip ospf [pid] counters This command clears counters based on OSPF routing process ID clear ip ospf [pid] process This command will restart the specified OSPF process. It attempts to keep the old router-id, except in cases where a new router-id was configured or an old user configured router-id was removed. Since this command can potentially cause a network churn, a user confirmation is required before performing any action

Use OSPF Authentication
Use authentication Too many operators overlook this basic requirement When using authentication, use the MD5 feature Under the global OSPF configuration, specify: area <area-id> authentication message-digest Under the interface configuration, specify: ip ospf message-digest-key 1 md5 <key> Authentication can be selectively disabled per interface with: ip ospf authentication null

Point to Point Ethernet Links
For any broadcast media (like Ethernet), OSPF will attempt to elect a designated and backup designated router when it forms an adjacency If the interface is running as a point-to-point WAN link, with only 2 routers on the wire, configuring OSPF to operate in "point-to-point mode" scales the protocol by reducing the link failure detection times Point-to-point mode improves convergence times on Ethernet networks because it: Prevents the election of a DR/BDR on the link, Simplifies the SPF computations and reduces the router's memory footprint due to a smaller topology database. interface fastethernet0/2 ip ospf network point-to-point

Tuning OSPF (1) DR/BDR Selection ip ospf priority 100 (default 1)
This feature should be in use in your OSPF network Forcibly set your DR and BDR per segment so that they are known Choose your most powerful, or most idle routers, so that OSPF converges as fast as possible under maximum network load conditions Try to keep the DR/BDR limited to one segment each Tuning options in order of priority – 1, 2 and 3 are must do’s; 4 and 5 are only for the advanced practitioner

Tuning OSPF (2) OSPF startup ISIS equivalent:
max-metric router-lsa on-startup wait-for-bgp Avoids blackholing traffic on router restart Causes OSPF to announce its prefixes with highest possible metric until iBGP is up and running When iBGP is running, OSPF metrics return to normal, make the path valid ISIS equivalent: set-overload-bit on-startup wait-for-bgp

Tuning OSPF (3) Hello/Dead Timers LSA Pacing
ip ospf hello-interval 3 (default 10) ip ospf dead-interval 15 (default is 4x hello) This allows for faster network awareness of a failure, and can result in faster reconvergence, but requires more router CPU and generates more overhead LSA Pacing timers lsa-group-pacing 300 (default 240) Allows grouping and pacing of LSA updates at configured interval Reduces overall network and router impact

Tuning OSPF (4) OSPF Internal Timers
timers spf 2 8 (default is 5 and 10) Allows you to adjust SPF characteristics The first number sets wait time from topology change to SPF run The second is hold-down between SPF runs BE CAREFUL WITH THIS COMMAND; if you’re not sure when to use it, it means you don’t need it; default is sufficient 95% of the time

Tuning OSPF (5) LSA filtering/interface blocking
Per interface: ip ospf database-filter all out (no options) Per neighbor: neighbor database-filter all out (no options) OSPFs router will flood an LSA out all interfaces except the receiving one; LSA filtering can be useful in cases where such flooding unnecessary (i.e., NBMA networks), where the DR/BDR can handle flooding chores area <area-id> filter-list <acl> Filters out specific Type 3 LSAs at ABRs Improper use can result in routing loops and black-holes that can be very difficult to troubleshoot

Summary OSPF has a bewildering number of features and options
Observe ISP best practices Keep design and configuration simple Investigate tuning options and suitability for your own network Don’t just turn them on!

Deploying OSPF for ISPs

Introduction to BGP ISP Training Workshops

Border Gateway Protocol
A Routing Protocol used to exchange routing information between different networks Exterior gateway protocol Described in RFC4271 RFC4276 gives an implementation report on BGP RFC4277 describes operational experiences using BGP The Autonomous System is the cornerstone of BGP It is used to uniquely identify networks with a common routing policy

BGP Path Vector Protocol Incremental Updates
Many options for policy enforcement Classless Inter Domain Routing (CIDR) Widely used for Internet backbone Autonomous systems

Path Vector Protocol BGP is classified as a path vector routing protocol (see RFC 1322) A path vector protocol defines a route as a pairing between a destination and the attributes of the path to that destination. / i AS Path

Path Vector Protocol AS11268 AS6337 AS7018 AS500 AS6461 AS600

Definitions Transit – carrying traffic across a network, usually for a fee Peering – exchanging routing information and traffic Default – where to send traffic when there is no explicit match in the routing table

NB: is not related to where an ISP is in the hierarchy
Default Free Zone The default free zone is made up of Internet routers which have explicit routing information about the rest of the Internet, and therefore do not need to use a default route NB: is not related to where an ISP is in the hierarchy

Peering and Transit example
provider A Backbone Provider D IXP-East IXP-West provider B provider C A and B can peer, but need transit arrangements with D to get packets to/from C

Autonomous System (AS)
Collection of networks with same routing policy Single routing protocol Usually under single ownership, trust and administrative control Identified by a unique 32-bit integer (ASN)

Autonomous System Number (ASN)
Two ranges (original 16-bit range) (32-bit range – RFC4893) Usage: 0 and (reserved) (public Internet) (documentation – RFC5398) (private use only) (represent 32-bit range in 16-bit world) (documentation – RFC5398) (public Internet) 32-bit range representation specified in RFC5396 Defines “asplain” (traditional format) as standard notation

Autonomous System Number (ASN)
ASNs are distributed by the Regional Internet Registries They are also available from upstream ISPs who are members of one of the RIRs Current 16-bit ASN allocations up to have been made to the RIRs Around are visible on the Internet Each RIR has also received a block of 32-bit ASNs Out of 3100 assignments, around 2800 are visible on the Internet See

Configuring BGP in Cisco IOS
This command enables BGP in Cisco IOS: router bgp 100 For ASNs > 65535, the AS number can be entered in either plain or dot notation: router bgp or router bgp 2.4 IOS will display ASNs in plain notation by default Dot notation is optional: bgp asnotation dot

BGP Basics AS 100 AS 101 AS 102 Peering E B D A C
Runs over TCP – port 179 Path vector protocol Incremental updates “Internal” & “External” BGP

Demarcation Zone (DMZ)
DMZ Network AS 100 AS 101 B D E AS 102 DMZ is the link or network shared between ASes

BGP General Operation Learns multiple paths via internal and external BGP speakers Picks the best path and installs it in the routing table (RIB) Best path is sent to external BGP neighbours Policies are applied by influencing the best path selection

Constructing the Forwarding Table
BGP “in” process receives path information from peers results of BGP path selection placed in the BGP table “best path” flagged BGP “out” process announces “best path” information to peers Best path stored in Routing Table (RIB) Best paths in the RIB are installed in forwarding table (FIB) if: prefix and prefix length are unique lowest “protocol distance”

Constructing the Forwarding Table
discarded BGP in process in accepted everything bgp peer BGP table routing table best paths forwarding table BGP out process out

eBGP & iBGP BGP used internally (iBGP) and externally (eBGP)
iBGP used to carry Some/all Internet prefixes across ISP backbone ISP’s customer prefixes eBGP used to Exchange prefixes with other ASes Implement routing policy

BGP/IGP model used in ISP networks
Model representation eBGP eBGP eBGP IGP iBGP IGP iBGP IGP iBGP IGP iBGP AS1 AS2 AS3 AS4

External BGP Peering (eBGP)
AS 100 AS 101 C B Between BGP speakers in different AS Should be directly connected Never run an IGP between eBGP peers

Configuring External BGP
ip address on ethernet interface Router A in AS100 interface ethernet 5/0 ip address ! router bgp 100 network mask neighbor remote-as 101 neighbor prefix-list RouterC in neighbor prefix-list RouterC out Local ASN Remote ASN ip address of Router C ethernet interface Inbound and outbound filters

Configuring External BGP
ip address on ethernet interface Router C in AS101 interface ethernet 1/0/0 ip address ! router bgp 101 network mask neighbor remote-as 100 neighbor prefix-list RouterA in neighbor prefix-list RouterA out Local ASN Remote ASN ip address of Router A ethernet interface Inbound and outbound filters

Internal BGP (iBGP) BGP peer within the same AS
Not required to be directly connected IGP takes care of inter-BGP speaker connectivity iBGP speakers must be fully meshed: They originate connected networks They pass on prefixes learned from outside the ASN They do not pass on prefixes learned from other iBGP speakers

Internal BGP Peering (iBGP)
AS 100 B A C D Topology independent Each iBGP speaker must peer with every other iBGP speaker in the AS

Peering between Loopback Interfaces
AS 100 C A B Peer with loop-back interface Loop-back interface does not go down – ever! Do not want iBGP session to depend on state of a single interface or the physical topology

Configuring Internal BGP
ip address on loopback interface Router A in AS100 interface loopback 0 ip address ! router bgp 100 network neighbor remote-as 100 neighbor update-source loopback0 neighbor remote-as 100 neighbor update-source loopback0 Local ASN Local ASN ip address of Router B loopback interface

Configuring Internal BGP
ip address on loopback interface Router B in AS100 interface loopback 0 ip address ! router bgp 100 network neighbor remote-as 100 neighbor update-source loopback0 neighbor remote-as 100 neighbor update-source loopback0 Local ASN Local ASN ip address of Router A loopback interface

Inserting prefixes into BGP
Two ways to insert prefixes into BGP redistribute static network command

Inserting prefixes into BGP – redistribute static
Configuration Example: router bgp 100 redistribute static ip route serial0 Static route must exist before redistribute command will work Forces origin to be “incomplete” Care required!

Inserting prefixes into BGP – redistribute static
Care required with redistribute! redistribute <routing-protocol> means everything in the <routing-protocol> will be transferred into the current routing protocol Will not scale if uncontrolled Best avoided if at all possible redistribute normally used with “route-maps” and under tight administrative control

Inserting prefixes into BGP – network command
Configuration Example router bgp 100 network mask ip route serial0 A matching route must exist in the routing table before the network is announced Forces origin to be “IGP”

Configuring Aggregation
Three ways to configure route aggregation redistribute static aggregate-address network command

Configuring Aggregation
Configuration Example: router bgp 100 redistribute static ip route null0 250 static route to “null0” is called a pull up route packets only sent here if there is no more specific match in the routing table distance of 250 ensures this is last resort static care required – see previously!

Configuring Aggregation – Network Command
Configuration Example router bgp 100 network mask ip route null0 250 A matching route must exist in the routing table before the network is announced Easiest and best way of generating an aggregate

Configuring Aggregation – aggregate-address command
Configuration Example: router bgp 100 network mask aggregate-address [summary-only] Requires more specific prefix in BGP table before aggregate is announced summary-only keyword Optional keyword which ensures that only the summary is announced if a more specific prefix exists in the routing table

Summary BGP neighbour status
Router6>sh ip bgp sum BGP router identifier , local AS number 10 BGP table version is 16, main routing table version 16 7 network entries using 819 bytes of memory 14 path entries using 728 bytes of memory 2/1 BGP path/bestpath attribute entries using 248 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 1795 total bytes of memory BGP activity 7/0 prefixes, 14/0 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd :04: :01: :04: ... BGP Version Updates sent and received Updates waiting

Summary BGP Table Router6>sh ip bgp
BGP table version is 16, local router ID is Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *>i / i *>i / i *>i / i *>i / i *>i / i *> / i *>i / i *>i / i *>i / i *>i / i ...

Summary BGP4 – path vector protocol iBGP versus eBGP
stable iBGP – peer with loopbacks announcing prefixes & aggregates

Introduction to BGP ISP Training Workshops

BGP Policy Control ISP Training Workshops

Applying Policy with BGP
Policy-based on AS path, community or the prefix Rejecting/accepting selected routes Set attributes to influence path selection Tools: Prefix-list (filters prefixes) Filter-list (filters ASes) Route-maps and communities

Policy Control – Prefix List
Per neighbour prefix filter incremental configuration Inbound or Outbound Based upon network numbers (using familiar IPv4 address/mask format) Using access-lists in Cisco IOS for filtering prefixes was deprecated long ago Strongly discouraged!

Prefix-list Command Syntax
[no] ip prefix-list list-name [seq seq-value] permit|deny network/len [ge ge-value] [le le-value] network/len: The prefix and its length ge ge-value: “greater than or equal to” le le-value: “less than or equal to” Both “ge” and “le” are optional Used to specify the range of the prefix length to be matched for prefixes that are more specific than network/len Sequence number is also optional no ip prefix-list sequence-number to disable display of sequence numbers

Prefix Lists – Examples
Deny default route ip prefix-list EG deny /0 Permit the prefix /8 ip prefix-list EG permit /8 Deny the prefix /12 ip prefix-list EG deny /12 In 192/8 allow up to /24 ip prefix-list EG permit /8 le 24 This allows all prefix sizes in the /8 address block, apart from /25, /26, /27, /28, /29, /30, /31 and /32.

Prefix Lists – Examples
In 192/8 deny /25 and above ip prefix-list EG deny /8 ge 25 This denies all prefix sizes /25, /26, /27, /28, /29, /30, /31 and /32 in the address block /8. It has the same effect as the previous example In 193/8 permit prefixes between /12 and /20 ip prefix-list EG permit /8 ge 12 le 20 This denies all prefix sizes /8, /9, /10, /11, /21, /22, … and higher in the address block /8. Permit all prefixes ip prefix-list EG permit /0 le 32 matches all possible addresses, “0 le 32” matches all possible prefix lengths

Policy Control – Prefix List
Example Configuration router bgp 100 network mask neighbor remote-as 110 neighbor prefix-list AS110-IN in neighbor prefix-list AS110-OUT out ! ip prefix-list AS110-IN deny /16 ip prefix-list AS110-IN permit /0 le 32 ip prefix-list AS110-OUT permit /16 ip prefix-list AS110-OUT deny /0 le 32

Policy Control – Filter List
Filter routes based on AS path Inbound or Outbound Example Configuration: router bgp 100 network mask neighbor filter-list 5 out neighbor filter-list 6 in ! ip as-path access-list 5 permit ^200$ ip as-path access-list 6 permit ^150$

Policy Control – Regular Expressions
Like Unix regular expressions . Match one character * Match any number of preceding expression + Match at least one of preceding expression ^ Beginning of line $ End of line \ Escape a regular expression character _ Beginning, end, white-space, brace | Or () brackets to contain expression [] brackets to contain number ranges

Simple Examples .* match anything .+ match at least one character ^$ match routes local to this AS _1800$ originated by AS1800 ^1800_ received from AS1800 _1800_ via AS1800 _790_1800_ via AS1800 and AS790 _(1800_)+ multiple AS1800 in sequence (used to match AS-PATH prepends) _$65530$_ via AS65530 (confederations)

Not so simple Examples ^[0-9]+$ Match AS_PATH length of one ^[0-9]+_[0-9]+$ Match AS_PATH length of two ^[0-9]*_[0-9]+$ Match AS_PATH length of one or two ^[0-9]*_[0-9]*$ Match AS_PATH length of one or two (will also match zero) ^[0-9]+_[0-9]+_[0-9]+$ Match AS_PATH length of three _(701|1800)_ Match anything which has gone through AS701 or AS1800 _1849(_.+_)12163$ Match anything of origin AS12163 and passed through AS1849

Policy Control – Route Maps
A route-map is like a “programme” for IOS Has “line” numbers, like programmes Each line is a separate condition/action Concept is basically: if match then do expression and exit else else etc Route-map “continue” lets ISPs apply multiple conditions and actions in one route-map

Route Maps – Caveats Lines can have multiple set statements
Lines can have multiple match statements Line with only a match statement Only prefixes matching go through, the rest are dropped Line with only a set statement All prefixes are matched and set Any following lines are ignored Line with a match/set statement and no following lines Only prefixes matching are set, the rest are dropped

Route Maps – Caveats Example
Omitting the third line below means that prefixes not matching list-one or list-two are dropped route-map sample permit 10 match ip address prefix-list list-one set local-preference 120 ! route-map sample permit 20 match ip address prefix-list list-two set local-preference 80 route-map sample permit 30 ! Don’t forget this

Route Maps – Matching prefixes
Example Configuration router bgp 100 neighbor route-map infilter in ! route-map infilter permit 10 match ip address prefix-list HIGH-PREF set local-preference 120 route-map infilter permit 20 match ip address prefix-list LOW-PREF set local-preference 80 ip prefix-list HIGH-PREF permit /8 ip prefix-list LOW-PREF permit /8

Route Maps – AS-PATH filtering
Example Configuration router bgp 100 neighbor remote-as 200 neighbor route-map filter-on-as-path in ! route-map filter-on-as-path permit 10 match as-path 1 set local-preference 80 route-map filter-on-as-path permit 20 match as-path 2 set local-preference 200 ip as-path access-list 1 permit _150$ ip as-path access-list 2 permit _210_

Route Maps – AS-PATH prepends
Example configuration of AS-PATH prepend router bgp 300 network mask neighbor remote-as 100 neighbor route-map SETPATH out ! route-map SETPATH permit 10 set as-path prepend Use your own AS number when prepending Otherwise BGP loop detection may cause disconnects

Route Maps – Matching Communities
Example Configuration router bgp 100 neighbor remote-as 200 neighbor route-map filter-on-community in ! route-map filter-on-community permit 10 match community 1 set local-preference 50 route-map filter-on-community permit 20 match community 2 exact-match set local-preference 200 ip community-list 1 permit 150:3 200:5 ip community-list 2 permit 88:6

Community-List Processing
Note: When multiple values are configured in the same community list statement, a logical AND condition is created. All community values must match to satisfy an AND condition ip community-list 1 permit 150:3 200:5 When multiple values are configured in separate community list statements, a logical OR condition is created. The first list that matches a condition is processed ip community-list 1 permit 150:3 ip community-list 1 permit 200:5

Route Maps – Setting Communities
Example Configuration router bgp 100 network mask neighbor remote-as 200 neighbor send-community neighbor route-map set-community out ! route-map set-community permit 10 match ip address prefix-list NO-ANNOUNCE set community no-export route-map set-community permit 20 match ip address prefix-list AGGREGATE ip prefix-list NO-ANNOUNCE permit /16 ge 17 ip prefix-list AGGREGATE permit /16

Route Map Continue Handling multiple conditions and actions in one route-map (for BGP neighbour relationships only) route-map peer-filter permit 10 match ip address prefix-list group-one continue 30 set metric 2000 ! route-map peer-filter permit 20 match ip address prefix-list group-two set community no-export route-map peer-filter permit 30 match ip address prefix-list group-three set as-path prepend

Order of processing BGP policy
For policies applied to a specific BGP neighbour, the following sequence is applied: For inbound updates, the order is: Route-map Filter-list Prefix-list For outbound updates, the order is:

Managing Policy Changes
New policies only apply to the updates going through the router AFTER the policy has been introduced or changed To facilitate policy changes on the entire BGP table the router handles the BGP peerings need to be “refreshed” This is done by clearing the BGP session either in or out, for example: clear ip bgp <neighbour-addr> in|out Do NOT forget in or out — doing so results in a hard reset of the BGP session

Managing Policy Changes
Ability to clear the BGP sessions of groups of neighbours configured according to several criteria clear ip bgp <addr> [in|out] <addr> may be any of the following x.x.x.x IP address of a peer * all peers ASN all peers in an AS external all external peers peer-group <name> all peers in a peer-group

BGP Policy Control ISP Training Workshops

Internet Exchange Point Design

IXP Design Background Why set up an IXP? Layer 2 Exchange Point
Design Considerations Route Collectors & Servers What can go wrong?

A bit of history In a time long gone…

A Bit of History… End of NSFnet – one major backbone
move towards commercial Internet Private companies selling their bandwidth Need for coordination of routing exchange between providers Traffic from ISP A needs to get to ISP B Routing Arbiter project created to facilitate this

What is an Exchange Point
Network Access Points (NAPs) established at end of NSFnet The original “exchange points” Major providers connect their networks and exchange traffic High-speed network or ethernet switch Simple concept – any place where providers come together to exchange traffic

Internet Exchange Points
Layer 2 exchange point Ethernet (100Gbps/10Gbps/1Gbps/100Mbps) Older technologies include ATM, Frame Relay, SRP, FDDI and SMDS Layer 3 exchange point Router based Has historical status now

Why an Internet Exchange Point?
Saving money, improving QoS, Generating a local Internet economy

Internet Exchange Point Why peer?
Consider a region with one ISP They provide internet connectivity to their customers They have one or two international connections Internet grows, another ISP sets up in competition How does traffic from customer of one ISP get to customer of the other ISP? Via the international connections

Yes, International Connections… If satellite, RTT is around 550ms per hop So local traffic takes over 1s round trip International bandwidth Costs significantly more than domestic bandwidth Congested with local traffic Wastes money, harms performance

Solution: Two competing ISPs peer with each other Result: Both save money Local traffic stays local Better network performance, better QoS,… More international bandwidth for expensive international traffic Everyone is happy

A third ISP enters the equation Becomes a significant player in the region Local and international traffic goes over their international connections They agree to peer with the two other ISPs To save money To keep local traffic local To improve network performance, QoS,…

Private peering means that the three ISPs have to buy circuits between each other Works for three ISPs, but adding a fourth or a fifth means this does not scale Solution: Internet Exchange Point

Internet Exchange Point
Every participant has to buy just one whole circuit From their premises to the IXP Rather than N-1 half circuits to connect to the N-1 other ISPs 5 ISPs have to buy 4 half circuits = 2 whole circuits  already twice the cost of the IXP connection

Internet Exchange Point
Solution Every ISP participates in the IXP Cost is minimal – one local circuit covers all domestic traffic International circuits are used for just international traffic – and backing up domestic links in case the IXP fails Result: Local traffic stays local QoS considerations for local traffic is not an issue RTTs are typically sub 10ms Customers enjoy the Internet experience Local Internet economy grows rapidly

Layer 2 Exchange The traditional IXP

IXP Design Very simple concept:
Ethernet switch is the interconnection media IXP is one LAN Each ISP brings a router, connects it to the ethernet switch provided at the IXP Each ISP peers with other participants at the IXP using BGP Scaling this simple concept is the challenge for the larger IXPs

Layer 2 Exchange ISP 5 ISP 4 ISP 6 IXP Management Network
IXP Services: Root & TLD DNS, Routing Registry Looking Glass, etc IXP Management Network Ethernet Switch ISP 1 ISP 3 ISP 2

Layer 2 Exchange ISP 5 ISP 4 ISP 6 IXP Management Network
IXP Services: Root & TLD DNS, Routing Registry Looking Glass, etc IXP Management Network Ethernet Switches ISP 1 ISP 3 ISP 2

Layer 2 Exchange Two switches for redundancy
ISPs use dual routers for redundancy or loadsharing Offer services for the “common good” Internet portals and search engines DNS Root & TLDs, NTP servers Routing Registry and Looking Glass

Layer 2 Exchange Requires neutral IXP management
Usually funded equally by IXP participants 24x7 cover, support, value add services Secure and neutral location Configuration Private address space if non-transit and no value add services Otherwise public IPv4 (/24) and IPv6 (/64) ISPs require AS, basic IXP does not

Layer 2 Exchange Network Security Considerations
LAN switch needs to be securely configured Management routers require TACACS+ authentication, vty security IXP services must be behind router(s) with strong filters

“Layer 3 IXP” Layer 3 IXP is marketing concept used by Transit ISPs
Real Internet Exchange Points are only Layer 2

IXP Design Considerations

Exchange Point Design The IXP Core is an Ethernet switch
It must be a managed switch Has superseded all other types of network devices for an IXP From the cheapest and smallest managed 12 or 24 port 10/100 switch To the largest switches now handling high densities of 10GE and 100GE interfaces

Exchange Point Design Each ISP participating in the IXP brings a router to the IXP location Router needs: One Ethernet port to connect to IXP switch One WAN port to connect to the WAN media leading back to the ISP backbone To be able to run BGP

Exchange Point Design IXP switch located in one equipment rack dedicated to IXP Also includes other IXP operational equipment Routers from participant ISPs located in neighbouring/adjacent rack(s) Copper (UTP) connections made for 10Mbps, 100Mbps or 1Gbps connections Fibre used for 1Gbps, 10Gbps, 40Gbps or 100Gbps connections

Peering Each participant needs to run BGP
They need their own AS number Public ASN, NOT private ASN Each participant configures external BGP directly with the other participants in the IXP Peering with all participants or Peering with a subset of participants

Peering (more) Mandatory Multi-Lateral Peering (MMLP)
Each participant is forced to peer with every other participant as part of their IXP membership Has no history of success — the practice is strongly discouraged Multi-Lateral Peering (MLP) Each participant peers with every other participant (usually via a Route Server) Bi-Lateral Peering Participants set up peering with each other according to their own requirements and business relationships This is the most common situation at IXPs today

Routing ISP border routers at the IXP must NOT be configured with a default route or carry the full Internet routing table Carrying default or full table means that this router and the ISP network is open to abuse by non-peering IXP members Correct configuration is only to carry routes offered to IXP peers on the IXP peering router Note: Some ISPs offer transit across IX fabrics They do so at their own risk – see above

Routing (more) ISP border routers at the IXP should not be configured to carry the IXP LAN network within the IGP or iBGP Use next-hop-self BGP concept Don’t generate ISP prefix aggregates on IXP peering router If connection from backbone to IXP router goes down, normal BGP failover will then be successful

Address Space Some IXPs use private addresses for the IX LAN
Public address space means IXP network could be leaked to Internet which may be undesirable Because most ISPs filter RFC1918 address space, this avoids the problem Some IXPs use public addresses for the IX LAN Address space available from the RIRs IXP terms of participation often forbid the IX LAN to be carried in the ISP member backbone

Hardware Try not to mix port speeds Don’t mix transports
if 10Mbps and 100Mbps connections available, terminate on different switches (L2 IXP) Don’t mix transports if terminating ATM PVCs and G/F/Ethernet, terminate on different devices Insist that IXP participants bring their own router moves buffering problem off the IXP security is responsibility of the ISP, not the IXP

Charging IXPs should be run at minimal cost to participants Examples:
Datacentre hosts IX for free Because ISP participants then use data centre for co-lo services, and the datacentre benefits long term IX operates cost recovery Each member pays a flat fee towards the cost of the switch, hosting, power & management Different pricing for different ports One slot may handle 24 10GE ports Or one slot may handle 96 1GE ports 96 port 1GE card is tenth price of 24 port 10GE card Relative port cost is passed on to participants

Services Offered Services offered should not compete with member ISPs (basic IXP) e.g. web hosting at an IXP is a bad idea unless all members agree to it IXP operations should make performance and throughput statistics available to members Use tools such as MRTG/Cacti to produce IX throughput graphs for member (or public) information

Services to Offer ccTLD DNS Root server Usenet News
the country IXP could host the country’s top level DNS e.g. “SE.” TLD is hosted at Netnod IXes in Sweden Offer back up of other country ccTLD DNS Root server Anycast instances of I.root-servers.net, F.root-servers.net etc are present at many IXes Usenet News Usenet News is high volume could save bandwidth to all IXP members

Services to Offer Route Collector Looking Glass
Route collector shows the reachability information available at the exchange Technical detail covered later on Looking Glass One way of making the Route Collector routes available for global view (e.g. Public or members only access

Services to Offer Content Redistribution/Caching Network Time Protocol
For example, Akamised update distribution service Network Time Protocol Locate a stratum 1 time source (GPS receiver, atomic clock, etc) at IXP Routing Registry Used to register the routing policy of the IXP membership (more later)

Introduction to Route Collectors
What routes are available at the IXP?

What is a Route Collector?
Usually a router or Unix system running BGP Gathers routing information from service provider routers at an IXP Peers with each ISP using BGP Does not forward packets Does not announce any prefixes to ISPs

Purpose of a Route Collector
To provide a public view of the Routing Information available at the IXP Useful for existing members to check functionality of BGP filters Useful for prospective members to check value of joining the IXP Useful for the Internet Operations community for troubleshooting purposes E.g.

Route Collector at an IXP
SWITCH R5 Route Collector

Route Collector Requirements
Router or Unix system running BGP Minimal memory requirements – only holds IXP routes Minimal packet forwarding requirements – doesn’t forward any packets Peers eBGP with every IXP member Accepts everything; Gives nothing Uses a private ASN Connects to IXP Transit LAN “Back end” connection Second Ethernet globally routed Connection to IXP Website for public access

Route Collector Implementation
Most IXPs now implement some form of Route Collector Benefits already mentioned Great public relations tool Unsophisticated requirements Just runs BGP

Introduction to Route Servers
How to scale very large IXPs

What is a Route Server? Has all the features of a Route Collector
But also: Announces routes to participating IXP members according to their routing policy definitions Implemented using the same specification as for a Route Collector

Features of a Route Server
Helps scale routing for large IXPs Simplifies Routing Processes on ISP Routers Optional participation Provided as service, is NOT mandatory Does result in insertion of RS Autonomous System Number in the Routing Path Optionally uses Policy registered in IRR

Diagram of N-squared Peering Mesh
For large IXPs (dozens for participants) maintaining a larger peering mesh becomes cumbersome and often too hard

Peering Mesh with Route Servers
ISP routers peer with the Route Servers Only need to have two eBGP sessions rather than N

RS based Exchange Point Routing Flow
TRAFFIC FLOW ROUTING INFORMATION FLOW

Advantages of Using a Route Server
Advantageous for large IXPs Helps scale eBGP mesh Helps scale prefix distribution Separation of Routing and Forwarding Simplifies BGP Configuration Management on ISP routers

Disadvantages of using a Route Server
ISPs can lose direct policy control If RS is only peer, ISPs have no control over who their prefixes are distributed to Completely dependent on 3rd party Configuration, troubleshooting, etc… Insertion of RS ASN into routing path (If using a router rather than a dedicated route-server BGP implementation) Traffic engineering/multihoming needs more care

Typical usage of a Route Server
Route Servers may be provided as an OPTIONAL service Most common at large IXPs (>50 participants) Examples: LINX, TorIX, AMS-IX, etc ISPs peer: Directly with significant peers With Route Server for the rest

Things to think about... Would using a route server benefit you?
Helpful when BGP knowledge is limited (but is NOT an excuse not to learn BGP) Avoids having to maintain a large number of eBGP peers But can you afford to lose policy control? (An ISP not in control of their routing policy is what?)

The different ways IXP operators harm their IXP…
What can go wrong… The different ways IXP operators harm their IXP…

What can go wrong? Concept
Some Service Providers attempt to cash in on the reputation of IXPs Market Internet transit services as “Internet Exchange Point” “We are exchanging packets with other ISPs, so we are an Internet Exchange Point!” So-called Layer-3 Exchanges — really Internet Transit Providers Router used rather than a Switch Most famous example: SingTelIX

What can go wrong? Financial
Some IXPs price the IX out of the means of most providers IXP is intended to encourage local peering Acceptable charging model is minimally cost-recovery only Some IXPs charge for port traffic IXPs are not a transit service, charging for traffic puts the IX in competition with members (There is nothing wrong with charging different flat fees for 100Mbps, 1Gbps, 10Gbps etc ports as they all have different hardware costs on the switch.)

What can go wrong? Competition
Too many exchange points in one locale Competing exchanges defeats the purpose Becomes expensive for ISPs to connect to all of them An IXP: is NOT a competition is NOT a profit making business

What can go wrong? Rules and Restrictions
IXPs try to compete with their membership Offering services that ISPs would/do offer their customers IXPs run as a closed privileged club e.g.: Restrictive membership criteria IXPs providing access to end users rather than just Service Providers IXPs interfering with ISP business decisions e.g. Mandatory Multi-Lateral Peering

What can go wrong? Technical Design Errors
Interconnected IXPs IXP in one location believes it should connect directly to the IXP in another location Who pays for the interconnect? How is traffic metered? Competes with the ISPs who already provide transit between the two locations (who then refuse to join IX, harming the viability of the IX) Metro interconnections work ok (e.g. LINX, AMS-IX, DE-CIX etc)

What can go wrong? Technical Design Errors
ISPs bridge the IXP LAN back to their offices “We are poor, we can’t afford a router” Financial benefits of connecting to an IXP far outweigh the cost of a router In reality it allows the ISP to connect any devices to the IXP LAN — with disastrous consequences for the security, integrity and reliability of the IXP

What can go wrong? Routing Design Errors
Route Server implemented from Day One ISPs have no incentive to learn BGP Therefore have no incentive to understand peering relationships, peering policies, &c Entirely dependent on operator of RS for troubleshooting, configuration, reliability RS can’t be run by committee! Route Server is to help scale peering at LARGE IXPs

What can go wrong? Routing Design Errors
iBGP Route Reflector used to distribute prefixes between IXP participants Claimed Advantage (1): Participants don’t need to know about or run BGP Actually a Disadvantage IXP Operator has to know BGP ISP not knowing BGP is big commercial disadvantage ISPs who would like to have a growing successful business need to be able to multi-home, peer with other ISPs, etc — these activities require BGP

What can go wrong? Routing Design Errors (cont)
Route Reflector Claimed Advantage (2): Allows an IXP to be started very quickly Fact: IXP is only an Ethernet switch — setting up an iBGP mesh with participants is no quicker than setting up an eBGP mesh

Route Reflector Claimed Advantage (3): IXP operator has full control over IXP activities Actually a Disadvantage ISP participants surrender control of: Their border router; it is located in IXP’s AS Their routing and peering policy IXP operator is single point of failure If they aren’t available 24x7, then neither is the IXP BGP configuration errors by IXP operator have real impacts on ISP operations

Route Reflector Disadvantage (4): Migration from Route Reflector to “correct” routing configuration is highly non-trivial ISP router is in IXP’s ASN Need to move ISP router from IXP’s ASN to the ISP’s ASN Need to reconfigure BGP on ISP router, add to ISP’s IGP and iBGP mesh, and set up eBGP with IXP participants and/or the IXP Route Server

More Information

Exchange Point Policies & Politics
AUPs Acceptable Use Policy Minimal rules for connection Fees? Some IXPs charge no fee Other IXPs charge cost recovery A few IXPs are commercial Nobody is obliged to peer Agreements left to ISPs, not mandated by IXP

Exchange Point etiquette
Don’t point default route at another IXP participant Be aware of third-party next-hop Only announce your aggregate routes Read RIPE-399 first Filter! Filter! Filter!

Exchange Point Examples
LINX in London, UK TorIX in Toronto, Canada AMS-IX in Amsterdam, Netherlands SIX in Seattle, Washington, US PA-IX in Palo Alto, California, US JPNAP in Tokyo, Japan DE-CIX in Frankfurt, Germany HK-IX in Hong Kong … All use Ethernet Switches

Features of IXPs (1) Redundancy & Reliability Support
Multiple switches, UPS Support NOC to provide 24x7 support for problems at the exchange DNS, Route Collector, Content & NTP servers ccTLD & root servers Content redistribution systems such as Akamai Route Collector – Routing Table view

Features of IXPs (2) Location Address space AS Number
neutral co-location facilities Address space Peering LAN AS Number If using Route Collector/Server Route servers (optional, for larger IXPs) Statistics Traffic data – for membership

More info about IXPs http://www.pch.net/documents
Another excellent resource of IXP locations, papers, IXP statistics, etc A collection of IXPs and interconnect points for ISPs

Summary L2 IXP – most commonly deployed
The core is an ethernet switch ATM and other old technologies are obsolete L3 IXP – nowadays is a marketing concept used by wholesale ISPs Does not offer the same flexibility as L2 Not recommended unless there are overriding regulatory or political reasons to do so Avoid!

Internet Exchange Point Design

BGP Configuration for IXPs

Background This presentation covers the BGP configurations required for a participant at an Internet Exchange Point It does not cover the technical design of an IXP Nor does it cover the financial and operational benefits of participating in an IXP See the IXP Design Presentation that is part of this Workshop Material set for financial, technical and operational details

Recap: Definitions Transit – carrying traffic across a network, usually for a fee Traffic and prefixes originating from one AS are carried across an intermediate AS to reach their destination AS Peering – private interconnect between two ASNs, usually for no fee Internet Exchange Point – common interconnect location where several ASNs exchange routing information and traffic

IXP Peering Issues Only announce your aggregates and your customer aggregates at IXPs Only accept the aggregates which your peer is entitled to originate Never carry a default route on an IXP (or private) peering router

ISP Transit Issues Many mistakes are made on the Internet today due to incomplete understanding of how to configure BGP for peering at Internet Exchange Points

Simple BGP Configuration example
Exchange Point Configuration

Exchange Point Example
Exchange point with 6 ASes present Layer 2 – ethernet switch Each ISP peers with the other NO transit across the IXP is allowed

Exchange Point AS100 AS150 A F AS140 AS110 E B AS120 AS130 C D Each of these represents a border router in a different autonomous system

Router configuration IXP router is usually located at the Exchange Point premises Configuration needs to be such that disconnecting it from the backbone does not cause routing loops or traffic blackholes Create a peer-group for IXP peers All outbound policy to each peer will be the same Ensure the router is not carrying the default route Or the full routing table (for that matter)

Creating a peer-group & route-map
router bgp 100 neighbor ixp-peer peer-group neighbor ixp-peer send-community neighbor ixp-peer prefix-list my-prefixes out neighbor ixp-peer route-map set-local-pref in ! ip prefix-list my-prefixes permit /19 route-map set-local-pref permit 10 set local-preference 150 Only allow AS100 address block to IXP peers Prefixes heard from IXP peers have highest preference

Interface and BGP configuration (1)
interface fastethernet 0/0 description Exchange Point LAN ip address mask no ip directed-broadcast no ip proxy-arp no ip redirects ! router bgp 100 neighbor remote-as 110 neighbor peer-group ixp-peer neighbor prefix-list peer110 in neighbor remote-as 120 neighbor peer-group ixp-peers neighbor prefix-list peer120 in IXP LAN BCP configuration

Interface and BGP Configuration (2)
neighbor remote-as 130 neighbor peer-group ixp-peers neighbor prefix-list peer130 in neighbor remote-as 140 neighbor peer-group ixp-peers neighbor prefix-list peer140 in neighbor remote-as 150 neighbor peer-group ixp-peers neighbor prefix-list peer150 in ! ip route null0 ip prefix-list peer110 permit /19 ip prefix-list peer120 permit /19 ip prefix-list peer130 permit /19 ip prefix-list peer140 permit /19 ip prefix-list peer150 permit /19 Peer-group applied to each peer Each peer has own inbound filter

Exchange Point Configuration of the other routers in the AS is similar in concept Notice inbound and outbound prefix filters outbound announces myprefixes only inbound accepts peer prefixes only Notice inbound route-map Set local preference higher than default ensures that if the same prefix is heard via AS100 upstream, the best path for traffic is via the IXP

Exchange Point Ethernet port configuration
Be aware of LAN configuration best practices Switch off proxy arp, redirects and broadcasts (if not already default) IXP border router must NOT carry prefixes with origin outside local AS and IXP participant ASes Helps prevent “stealing of bandwidth”

Exchange Point Issues: Alternative solutions
AS100 needs to know all the prefixes its peers are announcing New prefixes requires the prefix-lists to be updated Alternative solutions Use the Internet Routing Registry to build prefix list Use AS Path filters (could be risky)

More Complex BGP example

Exchange Point Example
Exchange point with 6 ASes present Layer 2 – ethernet switch Each ISP peers with the other NO transit across the IXP allowed ISPs at exchange points provide transit to their BGP customers

Exchange Point AS200 AS150 AS201 AS100 A F AS140 AS110 E B AS120 AS130 C D Each of these represents a border router in a different autonomous system

Exchange Point Router A configuration
interface fastethernet 0/0 description Exchange Point LAN ip address mask no ip directed-broadcast no ip proxy-arp no ip redirects ! router bgp 100 neighbor ixp-peers peer-group neighbor ixp-peers send-community neighbor ixp-peers prefix-list bogons out neighbor ixp-peers filter-list 10 out neighbor ixp-peers route-map set-local-pref in ...next slide Filter by ASN rather than by prefix – and block bogons too

Exchange Point neighbor remote-as 110 neighbor peer-group ixp-peers neighbor prefix-list peer110 in neighbor remote-as 120 neighbor peer-group ixp-peers neighbor prefix-list peer120 in neighbor remote-as 130 neighbor peer-group ixp-peers neighbor prefix-list peer130 in neighbor remote-as 140 neighbor peer-group ixp-peers neighbor prefix-list peer140 in neighbor remote-as 150 neighbor peer-group ixp-peers neighbor prefix-list peer150 in

Exchange Point ip route null0 ! ip as-path access-list 10 permit ^$ ip as-path access-list 10 permit ^200$ ip as-path access-list 10 permit ^201$ ip prefix-list peer110 permit /19 ip prefix-list peer120 permit /19 ip prefix-list peer130 permit /19 ip prefix-list peer140 permit /19 ip prefix-list peer150 permit /19 route-map set-local-pref permit 10 set local-preference 150

Exchange Point Notice the change in router A’s configuration
Filter-list instead of prefix-list permits local and customer ASes out to exchange Prefix-list blocks Special Use Address prefixes – rest get out, could be risky Other issues as previously This configuration will not scale as more and more BGP customers are added to AS100 As-path filter has to be updated each time Solution: BGP communities

More scalable BGP example

Exchange Point Example (Scalable)
Exchange point with 6 ASes present Layer 2 – ethernet switch Each ISP peers with the other NO transit across the IXP allowed ISPs at exchange points provide transit to their BGP customers (Scalable solution is presented here)

Exchange Point A F E B C D AS150 AS100 AS140 AS110 AS120 AS130
Each of these represents a border router in a different autonomous system - each ASN has BGP customers of their own

Router configuration Take AS100 as an example
Has 15 BGP customers, in AS501 to AS515 Create a peer-group for IXP peers All outbound policy to each peer will be the same Communities will be used AS-path filters will not scale well Community Policy AS100 aggregate put into 100:1000 All BGP customer aggregates go into 100:1100

Creating a peer-group & route-map
router bgp 100 neighbor ixp-peer peer-group neighbor ixp-peer send-community neighbor ixp-peer route-map ixp-peers-out out neighbor ixp-peer route-map set-local-pref in ! ip community-list 10 permit 100:1000 ip community-list 11 permit 100:1100 route-map ixp-peers-out permit 10 match community route-map set-local-pref permit 10 set local-preference 150 AS100 aggregate AS100 BGP customers Prefixes heard from IXP peers have highest preference

BGP configuration for IXP router
router bgp 100 neighbor remote-as 110 neighbor peer-group ixp-peer neighbor prefix-list peer110 in neighbor remote-as 120 neighbor peer-group ixp-peers neighbor prefix-list peer120 in ...etc Remaining configuration is the same as earlier Note the reliance again on inbound prefix-lists for peers Peers need to update the ISP if filters need to be changed And that’s what the IRR is for (otherwise use )

BGP configuration for AS100’s customer aggregation router
router bgp 100 network mask route-map set-comm neighbor remote-as 501 neighbor prefix-list as501-in in neighbor prefix-list default out neighbor route-map set-cust-policy in ...etc ! route-map set-comm permit 10 set community 100:1000 route-map set-cust-policy permit 10 set community 100:1100 Set community on AS100 aggregate Set community on BGP customer routes

Scalable IXP policy ISP Community policy is set on ingress
ISP now relies on communities to determine what is announced at the IXP No need to update any as-path filters, prefix-lists, &c If BGP customer announces more prefixes, only the filters at the aggregation edge need to be updated And those new prefixes will automatically be tagged with the community to allow them through to AS100’s IXP peers Consult the BGP community presentation for more extensive examples

Route Servers IXP operators quite often provide a Route Server to assist with scaling the BGP mesh All prefixes sent to a Route Server are usually distributed to all ASNs that peer with the Route Server (although some IXPs offer ISPs the facility to configure specific policies on their Route Server) BGP configuration to peer with a Route Server is the same as for any other ordinary peer But note that the route server will offer prefixes from several ASNs (the IXP membership who choose to participate) Inbound filter should be constructed appropriately

Route Servers Route Server software suppresses the ASN of the RS so that it doesn’t appear in the AS-path IOS by default will not accept prefixes from a neighbouring AS unless that AS is first in the AS-path router bgp 100 no bgp enforce-first-as neighbor x.x.x.a remote-as 65534 neighbor x.x.x.a route-map IXP-RS-in in neighbor x.x.x.a route-map ixp-peers-out out Needed so that IOS can receive prefixes without AS65534 being first in path

Exchange Point Configuration
Summary Exchange Point Configuration

Summary Ensure that BGP is scalable on your IXP peering router
Manually updating filters every time a new customer connects is tiresome and has potential to cause errors Only carry local ASN prefixes and customer routes on the IXP peering router Anything else (e.g. default or full BGP table) has the potential to result in bandwidth theft Filter IXP peer announcements Inbound – use the IRR if maintaining prefix-lists is difficult Outbound – use communities for scalability

BGP Configuration for IXPs

IXP Training Workshops

Similar presentations

Presentation on theme: "IXP Training Workshops"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

IXP Training Workshops

Similar presentations

Presentation on theme: "IXP Training Workshops"— Presentation transcript:

Similar presentations

About project

Feedback