1. Sofía Silva Berenguer sofia @ lacnic.net
Internet Exchange Points Workshop
Paramaribo - Surinam

2. AGENDA How the Internet Works Intro to BGP
IPv4 Exhaustion and IPv6 Deployment
Internet Exchange Points
How to request Internet Resources
Advanced topics
Route Hijacking
Leaks
Attacks against the path
Well known incidents
Securing the Routing System

3. How the Internet Works

4. Interconnection of Networks 4

5. The prefix 200.40.0.0/16 is propagated with BGP to the Internet
Internet Routing
ASN 6057 announces
/16
The prefix /16 is propagated with BGP to the Internet
ASN 8158 receives
/16
Atributos:
/16 AS_PATH ASN1 ASN3 ASN6057

6. Transit and Peering Transit Peering
Traffic and prefixes originating from one AS are carried across an intermediate AS to reach their destination AS
Usually for a fee
Peering
Private interconnect between two ASNs
Usually for no fee

7. Transit and Peering Transit Peering ASN 65538 ASN 64511 ASN 65536

8. Peering in an Internet Exchange Point (IXP)
Common interconnect location where several ASNs exchange routing information and traffic
ASN 65538
ASN 65536
ASN 65537
ASN 65539

9. IP address, where they come from?
Standards
Central Registry
Distribution *
Regional Internet Registries (RIRs) distribute IPv4, IPv6 and Autonomous System Numbers
Distribution
End user *
Sometimes the distribution is done through National Internet Registries (NIRs)
Allocations and Assignments 9

10. Regional Internet Registries

11. Intro to BGP

12. Border Gateway Protocol
A Routing Protocol used to exchange routing information between different networks
Exterior gateway protocol
Described in RFC4271
RFC4276 gives an implementation report on BGP
RFC4277 describes operational experiences using BGP
Works on TCP port 179

13. More about BGP
Learns multiple paths via internal and external BGP speakers – Initial exchange of entire table
Incremental Updates
Picks THE bestpath and installs it in the IP forwarding table – Policies applied by influencing the bestpath selection
Keepalive messages exchanged
Many options for policy enforcement
Classless Inter Domain Routing (CIDR)
Widely used for Internet backbone

14. Neighbors BGP speakers Internal (iBGP) if they are in the same ASN
External (eBGP) if they are in different ASN
eBGP
iBGP
ASN 65536
ASN 65538

15. Where to use BGP: Stub Network
ASN 65536,
Transit Provider
Only one exit for customer
Not really need to add BGP
ASN 65538, Customer

16. Multihomed Network Transit Providers Peering in IXP ASN 65538
Different situations possible
Multiple links to same ISP
Secondary for only backup
Load share between primary and secondary
Selectively use different ISPs
Peering at IXP
ASN 65536
ASN 65537
Peering in IXP

17. Internet Exchange Points

18. Why peer? Consider a region with one ISP
It provides internet connectivity to it’s customers
It has one or two international connections
Internet grows, another ISP sets up in competition
They provide internet connectivity to their customers
They have one or two international connections
How does traffic from customer of one ISP get to customer of the other ISP?
Via the international connections

19. Why peer? (Cont.)
Internet
ASN 65538
ASN 65536

20. Why peer? (Cont.) International bandwidth…
Yes, international connections…
If satellite, RTT is around 550ms per hop
So local traffic takes over 1s round trip
International bandwidth…
Is much more expensive than domestic bandwidth
Becomes congested with local traffic
Wastes money, harms performance

21. Why peer? (Cont.) Solution: Result:
Two competing ISPs peer with each other
Result:
Both save money
Local traffic stays local
Better network performance
More international bandwidth for international traffic

22. Why peer? (Cont.)
Internet
ASN 65538
ASN 65536

23. Why peer? (Cont.) A third ISP enters the equation
Becomes a significant player in the region
Local and international traffic goes over their international connections
They agree to peer with the two other ISPs
To save money
To keep local traffic local
To improve network performance

24. Why peer? (Cont.)
Peering means that the three ISPs have to buy circuits between each other
Works for three ISPs, but adding a fourth or a fifth means this does not scale
Solution:
Internet Exchange Point

25. Why peer? – Non-financial Motivations
Low latency
Control over routing
Redundancy
Aggregation benefits w/peering and Transit at IXP
ISP relationships – be one of the cool kids
Marketing benefits
Network reliability

26. Internet Exchange Point
Every participant has to buy just one whole circuit
From their premises to the IXP
Rather than N-1 half circuits to connect to the N-1 other ISPs
5 ISPs have to buy 4 half circuits = 2 whole circuits -> already twice the cost of the IXP connection

27. Simple Topology
Layer 2 fabric
N^N BGP relations

28. IXP Design
Each ISP participating in the IXP brings a router to the IXP location
Router needs:
One Ethernet port to connect to IXP switch
One WAN port to connect to the WAN media leading back to the ISP backbone
To be able to run BGP

29. IXP Design (Cont.)
IXP switch located in one equipment rack dedicated to IXP
Also includes other IXP operational equipment (Management network, TLD DNS, Routing Registry, Looking Glass, etc.)
Optional: Second switch for redundancy
Routers from participant ISPs located in neighbouring/adjacent rack(s)
Copper (UTP) connections made for 10Mbps, 100Mbps or 1Gbps connections
Fibre used for 10Gbs and higher speeds

30. Peering at an IXP Each participant need to run BGP
They need their own AS number
Public ASN, NOT private ASN
Each participant configures external BGP with the other participants in the IXP
Peering with all participants Or
Peering with a subset of participants

31. IXP - Routing
ISP border routers at the IXP generally should NOT be configured with a default route or carry the full Internet routing table
Carrying default or full table means that this router and the ISP network is open to abuse by non-peering IXP members
ISP border routers at the IXP should not be configured to carry the IXP LAN network within the IGP or iBGP
Set BGP next-hop to local router (Cisco IOS next-hop-self)

32. IP Address Space Some IXPs use private addresses for the IXP LAN
Public address space means the IXP network can be leaked to the Internet, which could be undesirable
Filtering RFC1918 address space by ISPs is Best Practice; this avoids leakage
Some IXPs use public addresses for the IXP LAN
Address space is available from LACNIC
IXP terms of participation usually forbid carrying the IXP LAN addressing in the ISP backbone

33. Hardware The IXP Core is an Ethernet Switch (Mandatory)
Therefore invest in the best and most expandable equipment that its financial circumstances allow
Having 2 switches is good for redundancy if the funds can allow
Route Server (Optional)
Provides ease of configuration for new members
Direct peering between the IXP members can be implemented in the absence of the Route Server

34. Hardware (Cont.) Other optional equipment
Web Server (website, monitoring, etc.)
Mail Server ( , mailing list, etc.)
Transit Router (to provide Internet access to the IXP website, and staff Internet access)
Route Collector (Looking glass which assists IXP members with troubleshooting. It can also be used to collect routes for statistics measurements)

35. Hardware - Suggestions
Try not to mix port speeds
If 10Mbps and 100Mbps connections available, terminate on different switches
Insist that IXP participants bring their own router
Moves buffering problem off the IXP
Ensures integrity of the IXP
Security is responsibility of the ISP, not the IXP

36. Location The location of the IXP is very important.
The IXP location should be neutral and low cost.
In considering the IXP location the following factors should be considered:
Space
Environment Control
Security
Power
Access to terrestrial Infrastructure
Cabling
Support

37. Recommendations and Best Practices
Only announce your aggregates and your customer aggregates at IXPs
Only accept the aggregates which your peer is entitled to originate
Never carry a default route on an IXP (or private) peering router
Failing to do so leads to route-hijacks and leaks

38. General Info about IXPs. .
Source:

39. General Info about IXPs.
Source:

40. General Info about IXPs
Source:

41. How to request internet resources

42. Go to https://solicitudes.lacnic.net/
Or fill out a form and send it in the body of a message to
You can find templates at:
Once the online request or the form has been processed by the system, the requestor will receive a confirmation with a ticket number.
After that the hostmasters will analyze the request.
If the request is approved, it may be necessary to pay a fee and to sign the Registration Service Agreement.

43. Who can request resources?
The person allowed to request resources for an organization is the Administrative POC.
To request resources through the new Requests System you will have to log in using the Administrative POC handle.

44. Requesting an ASN
In order to qualify for an ASN allocation the organization should have:
A unique routing policy, meaning a policy that differs from that applied by the upstream provider.
Or, a network with more than one independent connection to the Internet. (Multi-homed site)
From January 1, 2007 to December 31, 2010 Lacnic assigned ASN of 16 and 32bits upon request. However, since January 1, 2011 Lacnic stopped making distinctions between the assignment of 16- and 32-bit Autonomous Systems Numbers (ASNs) and will only assign ASNs from a general 32-bit pool. This change will be introduced to comply with the Global Policy "Internet Assigned Numbers Authority (IANA) Policy for Allocation of ASN Blocks to Regional Internet Registries" adopted in September 2010.

45. Micro-assignments to Critical Infrastructure
Micro-assignment -> prefixes between /24 and /20.
For projects and network infrastructure that are key or critical for the region, such as IXPs (Internet Exchange Points), NAPs (Network Access Points), RIRs, ccTLDs, among others.
IXPs or NAPs must meet the following requirements:
Duly document the following aspects:
Prove by means of their bylaws their IXP or NAP capacity. The organization shall have at least three members and an open policy for the association of new members.
Submit a diagram of the organization's network structure.
Document the numbering plan to be implemented.
Provide a utilization plan for the following three and six months.
If the applicant does not already have an IPv6 block assigned by LACNIC, simultaneously request an IPv6 block in accordance with the corresponding applicable policy.
The rest of the applications shall be studied based on the analysis of the documentation justifying the critical and/or key aspects of the project.
Organizations receiving micro-assignments shall not sub-assign these IPv4 addresses.

46. Requesting an IPv4 block for ISPs
To qualify for the allocation of a /22 block the org must:
Prove usage or immediate necessity of a /24
Submit a detailed one-year usage plan for a /23
Agree to renumber from previously allocated space and return those IP addresses to their ISPs within 12 months
If the applicant does not already have an IPv6 block assigned by LACNIC, simultaneously request an IPv6 block in accordance with the corresponding applicable policy.
For a larger block additional requirements apply

47. Requesting an IPv6 block for ISPs
To qualify for an initial allocation of a /32 block the organization should:
Be a LIR (Local Internet Registry), which means being an organization that assigns address spaces for its network services customers
Not be an end site (end user)
Document a detailed plan for the services and IPv6 connectivity to be offered to other organizations (clients)
Announce the allocated block in the Internet inter-domain routing system, with the minimum possible level of disaggregation to the one that is publishing the IP blocks, within a period no longer than 12 months.
Offer IPv6 services to clients physically located within the region covered by LACNIC within a period not longer than 24 months

48. More info Policy Manual Registration Services
Registration Services

49. Advanced topics

50. Route Hijacking
This occurs when a participant in the Internet Routing announces a prefix for which it has no authority
Malicious or by operational errors
More know cases:
Pakistan Telecom vs. You Tube (2008)
China Telecom (2010)
Google in Eastern Europe (various AS, 2010)
Latin American cases (beginning 2011)

51. Route Hijacking AS 6057 announces 200.40/16
** Recordar que las rutas mas especificas son preferidas
Luego del primer anuncio de rutas, el trafico normal fluye entre el AS 6057 y el AS 8158
Luego del segundo anuncio, hay trafico dirigido desde y hacia un /24 que se ve desviado hacia el AS 15358
AS announces /24
ASN 8158 receives
/16 y /24
/16 AS_PATH ASN1 ASN3 ASN6057
/24 AS_PATH ASN1 ASN15358
ASN 8158 receives
/16

52. Leaks There is not a standard definition of leaks
But it happens when an ASN “leaks” non-customer or self-originated routes to other peers.
The effects is to give transit to those networks for the peers of the ASN

53. Route Leaks How this should work without leaks ASN 64511 Transit
2001:db8::/
2001:db8:100::/
How this should work without leaks
ASN 64511
ASN 65536
ASN 65537
2001:db8:100:/40
2001:db8::/40
Traffic to whole 2001:db8::/40 goes this way
Transit
2001:db8::/40
2001:db8:1/48
Peering
2001:db8::/40
2001:db8:100:/40

54. Route Leaks Now a Route Leak ASN 64511 Transit ASN 65536 ASN 65537
2001:db8::/
2001:db8:100::/
2001:db8:1::/
Now a Route Leak
ASN 64511
ASN 65536
ASN 65537
2001:db8:100:/40
2001:db8::/40
2001:db8:1::/48
2001:db8::/40
Transit
Traffic to 2001:db8:1::/48 goes this way
2001:db8::/40
2001:db8:1/48
2001:db8::/40
Peering
2001:db8:100:/40

55. Attacks against the path
AS Insertion:
A router might insert one or more ASNs, other than its own ASN, into an update message
False (Route) Origination with valid ASN:
A router might originate a route for a prefix using an ASN not authorized to originate routes for that prefix.

56. Attacks against the path
AS 6057 announces /16
** Recordar que las rutas mas especificas son preferidas
Luego del primer anuncio de rutas, el trafico normal fluye entre el AS 6057 y el AS 8158
Luego del segundo anuncio, hay trafico dirigido desde y hacia un /24 que se ve desviado hacia el AS 15358
ASN 6057
Fake AS 6057 announces /24
ASN 8158 receives
/16 y /24
/16 AS_PATH ASN1 ASN3 ASN6057
/24 AS_PATH ASN1 ASN15358
ASN 8158 receives
/16

57. Well know incidents

58. Pakistan Telecom vs. Youtube
On Sunday, 24 February 2008, Pakistan Telecom (AS17557) started an unauthorized announcement of the prefix /24 (Youtube)
One of Pakistan Telecom's upstream providers, PCCW Global (AS3491) forwarded this announcement to the rest of the Internet, which resulted in the hijacking of YouTube traffic on a global scale.
Reason: “Fat fingers”
Video de RIPE NCC

59. Moratel vs Google Reported by Cloudflare on November 06, 2012
Google's services experienced a limited outage for about 27 minutes over some portions of the Internet.
Moratel (23947) was “leaking” Google one route and packets were going through Indonesia
Reason: “Fat fingers”

60. ASN and Prefix Hijack
On 2011 one large European ISP complain that one of their prefixes was being announced by a Mexican ISP
The Mexican ISP review their network but could not found the problem
Later it appears that a Brazilian ISP was using the Mexican ISP’s ASN to announce the European prefix
Reason: Poor BGP knowledge

61. Securing the routing system

62. Recall how Internet Resources are managed

63. Recall how Internet Resources are managed
IANA
ARIN
ISP
End users
LACNIC
NIC.br
NIC.mx
ISP mx
ISP #1
APNIC
LIRs/ISPs
RIPE NCC
AfriNIC
Each RIR is an authoritative source of information about the relation “user” <-> “resource”
Each RIR operates its registration data base
Members and RIRs sign Service Agreements between them 63

64. Who has the "right" to use resources?
When an ISP obtains resources from its RIR (IPv6/IPv4/ASN):
The ISP has to notify its upstream ASs which prefixes are going to be announced via BGP
This is usually done via , web forms or by updating an IRR (Internet Routing Registry)
Upstreams verify (or at least they should) the right of use for the announced resources
RIR WHOIS Text-based and not really suitable for automatic usage
IRR WHOIS Non-signed information, little additional tools provided for verification of usage rights except for names, phone numbers and POCs
This verification process is sometimes not as thorough as it should be

65. What is RPKI?
RPKI (Resource Public Key Infrastructure) allows the validation of an organization right to use of a certain resource (IPv4, IPv6, ASN)
RPKI combines the hierarchy of the Internet resource assignment model through RIRs with the use of digital certificates based on standard X.509
RPKI is standardized in the IETF through the SIDR WG. It has produced RFCs 6480 – 6492

66. RPKI All RPKI signed objects are listed in public repositories
After verification, these objects can be used to configure filtering in routers
Validation Process
Signed objects have references to the certificate used to sign them
Each certificate has a pointer to an upper level certificate
The resources listed in a certificate MUST be valid subsets of the resources listed in its parent's certificate
In this way a trust chain can be traced to a "trust anchor" both cryptographically as well as in CIDR terms

67. X.509 v3 certificates with RFC 3779 extensions
Signature Algorithm
Serial Number
Version
Issuer
Subject
Subject Public Key
Extensions
Addr:
Asid: 65535
Subject Information
Authority (SIA)
Authority Information
Access (AIA)
X.509 Digital Certificates
Subject, validity period, public key and other fields
With extensions:
RFC 3779 defines extensions that allow the representation of Internet resources as certificate fields
List of IPv4, IPv6 and ASNs assigned to an organization
Implemented in OpenSSL 1.0c onwards

68. ROAs
Using Certificates we can create objects describing the origin of a prefix
ROAs: Routing Origin Authorization
ROAs contain data on the allowed origin-as for a set of prefixes
ROAs are signed using the certificates generated by the RPKI
Signed ROAs are copied to the repository

69. ROAs (ii) A simplified ROA contains the following information:
These ROAs states that:
"The prefix /17 will be originated by ASN 6057 and could be de-aggregated up to /20" "This statement is valid starting on Jan 2, 2013 until Dec 31, 2013"
Other ROA content
ROAs contain cryptographic material that allows validation of the ROAs content
Prefix
MaxLen
Origin AS
Valid Since
Valid Until
/17 20
6057
/22 24
28000

70. ROAs (iii) - Validation
ROAs validation process includes:
Criptographic validation of End Entity certificates (EE) that are included in each ROA
CIDR validation of resources listed in the EE against the resources listed in the issuing certificate
Verification that prefixes listed in the route origin attestations are included in the prefixes listed in the EE certificates of each ROA

71. Creation of ROAs

72. Creation of ROAs

73. Origin Validation
Routers build a database with the information they receive from the caches
This table contains
Prefix, Min length, Max length, Origin-AS
By applying a set of rules a validity status is assigned to each UPDATE prefix
Network operators can use “validity” attribute to construct routing policies

74. Origin Validation VALID
UPDATE /9 ORIGIN-AS 20
IP prefix/[min_len – max_len]
Origin AS
/ [16-20] 10
/[8-21] 20
If the "UPDATE pfx" is not covered by any entry in the DB -> "not found"
If the "UPDATE pfx" is covered by at least one entry in the DB, and the origin-AS matches the ASNs in the DB -> "valid"
If the origin-AS does NOT match -> "invalid"

75. Origin Validation INVALID
UPDATE /22 ORIGIN-AS 20
IP prefix/[min_len – max_len]
Origin AS
/ [16-20] 10
/[8-21] 20
If the "UPDATE pfx" is not covered by any entry in the DB -> "not found"
If the "UPDATE pfx" is covered by at least one entry in the DB, and the origin-AS matches the ASNs in the DB -> "valid"
If the origin-AS does NOT match -> "invalid"

76. Origin Validation INVALID
UPDATE /9 ORIGIN-AS 66
IP prefix/[min_len – max_len]
Origin AS
/ [16-20] 10
/[8-21] 20
If the "UPDATE pfx" is not covered by any entry in the DB -> "not found"
If the "UPDATE pfx" is covered by at least one entry in the DB, and the origin-AS matches the ASNs in the DB -> "valid"
If the origin-AS does NOT match -> "invalid"

77. Origin Validation NOT FOUND
UPDATE /9 ORIGIN-AS 66
IP prefix/[min_len – max_len]
Origin AS
/ [16-20] 10
/[8-21] 20
If the "UPDATE pfx" is not covered by any entry in the DB -> "not found"
If the "UPDATE pfx" is covered by at least one entry in the DB, and the origin-AS matches the ASNs in the DB -> "valid"
If the origin-AS does NOT match -> "invalid"

78. Routing Policies with Origin-validation
Using the BGP validation attribute network operators can construct routing policies
For example:
Assign higher preference to routes with status “valid” than routes with status “unknown”
Drop routes with status of “invalid”
Very important, RPKI is a source of data, operators are free to use it as it suits them better

79. RPKI Management System
RPKI in action
Cache
RPKI Management System
Repository
Routers validate updates from other BGP peers
Caches feeds routers using RTR protocol with ROA information
Caches retrieve and cryptographically validates certificates and ROAs from repositories

80. RPKI Operation Modes “Hosted” mode “Delegated” mode
LACNIC issues certificates and keeps public and private keys in its systems
Certificates are issued at the request of member organizations
LACNIC provides a web interface for management
“Delegated” mode
Each organization has it’s own certificate, signed by LACNIC’s CA
The organization sends signing requests to LACNIC, who returns them signed
“Up-down” protocol

81. RPKI today
5 CAs and repositories working (RIRs) in hosted and delegated mode (just APNIC)
Validation, CA and origin validation working software and devices
Some other tools built around (bgpmon, LACNIC Labs, RIPE Labs)
~ 11,700 routes signed (~ 8.9% 1,226 invalid or hijacks)
Origin Validation available in Cisco, Juniper, Quagga, BIRD

82. LACNIC’s RPKI Structure
Self-signed RTA
LACNIC RTA
LACNIC’s Resources
LACNIC Production
<<INHERITED>>
ISP #2
Resouces of ISP #2
ROA
End Entity cert.
ISP #1
Resources of ISP #1
End User CA #1
(Resources of EU#1)
Signing chain
Explicar la idea del offline
Comentar sobre el single trust anchor
LACNIC RTA es la raiz de la rPKI para la region LAC
Es un certificado **auto firmado**
La clave privada de LACNIC RTA esta offline
LACNIC Produccion es un certificado intermedio, el que efectivamente firma los objetos con su clave privada
ISP #n son CAs que pueden firmar otras Cas
Cada uno de ellos lista los recursos del ISP correspondiente
Los ROAs son objetos firmados
Contienen informacion de enrutamiento (origin-as)
Usan un certificado end-entity cuya clave privada se usa una vez y luego se descarta

83. LACNIC’s RPKI Structure (ii)
CAs
Entity that issues certificates (bit CA=1)
ISPs can use the certificate to sign it’s clients’ certificates
Certificates Repository
Repository of certificates, CRLs and manifests
Accesible through “rsync”
Management Interface
User web interface for those who prefer “hosted” mode

84. RPKI CA Services
Children certificates issuance when the registration database is updated or on user demand
Children certificates revocation in a centralized manner or on user demand
Periodic issuance of CRL for CA certificate
CA Certificate and children certificates publication in a public repository (rsync)

85. Conclusions
The routing system is one of the core operations of the Internet
Still is vulnerable to attacks and bad configs
Some work has been done (RPKI, Origin Validation)
We need to continue our work
Protocol specification
Deployment (Filtering, RPKI, Origin validation)

86. Links / References LACNIC’s RPKI System LACNIC’s RPKI Repository
LACNIC’s RPKI Repository
rsync://repository.lacnic.net/rpki/
To see the repository
rsync --list-only rsync://repository.lacnic.net/rpki/lacnic/
RPKI Statistics

87. Questions?
Comments?

88. CASA DE INTERNET DE LATINOAMÉRICA Y EL CARIBE
twitter.com/LACNIC facebook.com/ LACNIC youtube.com/user/lacnicstaff gplusme.at/LACNIC

89. Sofía Silva Berenguer sofia @ lacnic.net
Thank you!