London Internet Exchange Point Update Keith Mitchell, Executive Chairman NANOG15 Meeting Denver, Jan 1999

LINX Update LINX now has 63 members Second site now in use New Gigabit backbone in place Renumbered IXP LAN Some things we have learned! Statistics What’s coming in 1999

What is the LINX ? UK National IXP Not-for-profit co-operative of ISPs Main aim to keep UK domestic Internet traffic in UK Increasingly keeping EU traffic in EU

LINX Status Established Oct 94 by 5 member ISPs Now has 7 FTE dedicated staff Sub-contracts co-location to 2 neutral sites in London Docklands: Telehouse TeleCity Traffic doubling every 4-6 months !

LINX membership Now totals since Oct 98 Recent UK members: RedNet XTML Mistral ICLnet Dialnet Recent non-UK members: Carrier 1 GTE Above Net Telecom Eireann Level 3

LINX Members by Country

Second Site Existing Telehouse site full until 99Q3 extension ready TeleCity is new dedicated co-lo facility, 3 miles from Telehouse Awarded LINX contract by open tender (8 submissions) LINX has 16-rack suite Space for 800 racks

Second Site LINX has diverse dark fibre between sites (5km) Same switch configuration as Telehouse site Will have machines to act as hot backups for the servers in Telehouse Will have a K.root server behind a transit router soon

LINX Traffic Issues Bottleneck was inter-switch link between Catalyst 5000s Cisco FDDI could no longer cope 100baseT nearly full Needed to upgrade to Gigabit backbone within existing site 98Q3

Gigabit Switch Options Looked at 6 vendors: Cabletron/Digital, Cisco, Extreme, Foundry, Packet Engines, Plaintree Some highly cost-effective options available But needed non-blocking, modular, future-proof equipment, not workgroup boxes

Old LINX Infrastructure 5 Cisco Switches: 2 x Catalyst 5000, 3 x Catalyst Plaintree switches 2 x WaveSwitch 4800 FDDI backbone Switched FDDI ports 10baseT & 100baseT ports Media convertors for fibre ether (>100m)

Old LINX Topology

New Infrastructure Catalyst and Plaintree switches no longer in use Catalyst 5000s appeared to have broadcast scaling issues regardless of Supervisor Engine Plaintree switches had proven too unstable and unmanageable Catalyst 1200s at end of useful life

New Infrastructure Packet Engines PR-5200 Chassis based 16 slot switch Non-blocking 52Gbps backplane Used for our core, primary switches One in Telehouse, one in TeleCity Will need a second one in Telehouse within this quarter Supports 1000LX, 1000SX, FDDI and 10/100 ethernet

New Infrastructure Packet Engines PR-1000: Small version of PR U switch; 2x SX and 20x 10/100 Same chipset as 5200 Extreme Summit 48: Used for second connections Gives vendor resiliency Excellent edge switch - low cost per port and 2x Gigabit, 48x 10/100 ethernet

New Infrastructure Topology changes: Aim to be able to have major failure in one switch without affecting member connectivity Aim to have major failures on inter-switch links with out affecting connectivity Ensure that inter-switch connections are not bottlenecks

New backbone All primary inter-switch links are now gigabit New kit on order to ensure that all inter-switch links are gigabit Inter-switch traffic minimised by keeping all primary and all backup traffic on their own switches

IXP Switch Futures Vendor claims of 1000baseProprietary 50km+ range are interesting Need abuse prevention tools: port filtering, RMON Need traffic control tools: member/member bandwidth limiting and measurement

Address Transition Old IXP LAN was /24 New allocation /19 New IXP LAN /23 “Striped” allocation on new LAN 2 addresses per member, same last octet About 100 routers involved

Address Migration Plan Configured new address(es) as secondaries Brought up peerings with their new addresses When all peers are peering on new addresses, stopped old peerings Swap over the secondary to the primary IP address

Address Migration Plan Collector dropped peerings with old /24 addresses Anyone not migrated at this stage lost direct peering with AS5459 Eventually, old addresses no longer in use

What we have learned... the hard way! Problems after renumbering Some routers still using /24 netmask Some members treating the /23 network as two /24s Big problem if proxy ARP is involved! Broadcast traffic bad for health We have seen >50 ARP requests per second at worst times.

ARP Scaling Issues Renumbering led to lots of ARP requests for unused IP addresses ARP no-reply retransmit timer fixed time-out Maintenance work led to groups of routers going down/up together  Synchronised “waves” of ARP requests

New MoU Prohibitions Proxy ARP ICMP redirects Directed broadcasts Spanning Tree IGP broadcasts All non-ARP MAC layer broadcasts

Statistics LINX total traffic 300 M/sec avg, 405 M/sec peak Routing table 9,200 out of 55,000 routes k.root-servers 2.2 Mbit/sec out, 640 Kbit/sec in nic.uk 150 Kbit/sec out, 60 Kbit/sec in

Statistics and looking glass at

Things planned for ‘99 Infrastructure spanning tree implementation Completion of Stratum-1 NTP server Work on an ARP server Implementation of route server Implementation of RIPE NCC test traffic box