Bas Kreukniet, Sr Network Specialist at SURF SARA NL-T1 Expectations, findings, and innovation Geneva Workshop 10 Februari 2014

Outline 1.Expectations from NL-T1 grid administrators 2.Findings while connecting to the LHCONE 3.Innovation: Ethernet OAM and NSI

Expectations from NL-T1 administrators Advice from grid administrators NL-T1: Bulk data with simple applications (grid-FTP). “Keep it simple” Network provisioning from application is considered “complex” Don’t rush to merge LHCOPN with LHCONE

NL-T1 connected to LHCONE since 20 Jan 2014

BGP routing starts at the connected site. BGP routing for T1’s: Not only a “NREN thing” – it already starts at your organisation! Focus on some BGP topics for connecting sites to LHC networks

LHCOPN connectivity

LHCONE connectivity /16 *[BGP/170] 1w2d 04:07:47, MED 120, localpref 100 AS path: I > to via xe-4/0/ NL-T1 AS1162 GÉANT AS20965 LHCONE-RS CERN AS20641 CERN AS513 LHCONE.inet.0: 133 destinations, 133 routes (133 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both

Preferred route: LHCOPN, LHCONE or Internet primary: direct T1–T1 over LHCOPN secondary: T1–T1 over LHCOPN via another T1 tertiary: LHCONE quaternary: Internet

Preferred route: route (a)symmetry If everyone makes this choice, all connections are symmetric. But: sites may have different preferences: 10 Gb/s 1 Gb/s 100 Gb/s site 1 site 2 ISP-B ISP-A

Tie-breaker between LHCOPN and LHCONE CNAF routers at NL-T1 LHCOPN /17 *[BGP/170] 14:58:18, localpref 100 AS path: I LHCONE /17 *[BGP/170] 1w2d 04:23:40, MED 120, localpref 100 AS path: I

LHCOPN / LHCONE route preference BNL routes at NL-T1 LHCOPN /24 *[BGP/170] 2w5d 09:35:43, MED 10, localpref 100 AS path: I > to via xe-1/1/0.0 [BGP/170] 3d 01:15:22, MED 51, localpref 100 AS path: I > to via xe-2/1/0.0 LHCONE /24 *[BGP/170] 1w2d 04:27:43, MED 120, localpref 100 AS path: I > to via xe-4/0/1.2012

Problems and concerns regarding BGP A site advertising his routes has no control who to send a route to. At best they can give hints with BGP communities. The site receiving a route decides which route to accept and how to accept.

BGP in LHC networks Only As is not enough. More info needed.  community for origin of a route T1/T2? Or even better: -site connected to LHCOPN -site connected to LHCONE - Site connected to both: Type A “prefer LHCONE for this route” Type B “prefer LHCOPN for this route” - Specials: dedicated link between two (T1) sites. (“VPN”or “private link”)

Route Preference Solutions Idea: Tag routes with two types of communities: One for origin or source One for destinations See also: BGP hinting by Martin Sweeny (Indiana U) BGP Always-compare-MED always on. We sometimes add metrics on incoming routes.

Operational Issues We received routes over LHC from a site, but traffic we send is blackholed by that site The site was still reachable over the Internet. This happened to us twice recently: on LHCOPN (accidental route redistribution) and LHCONE (incoming IP filter). Configuration errors will be made (we’re also just human) LHCOPN Link NL-T1 – TRIUMF link is still unstable 31 outages last 4 months  Monitoring remains important!

LHCONE and LHCOPN layer 2 monitoring Ethernet OAM monitoring or Layer 2 monitoring NIKHEF, TRIUMF to participate as Measurement Points Looking for T1’s and T2 to participate Advantages: Layer2 keep-alive, ping and traceroute Interdomain, intervendor solution L2 devices can be made visible unidirectional fibercuts can be signalled

LHCONE Innovation NL-T1 likes to partcipate in NSI for LHCONE NSI experiences so far: Use-case: Life Science Grid (LSG) in Holland makes use of NSI since autumn Freek contributed to standard Sander wrote NSI client and implemented it for “Cloud Bypassing” in the Life Science Grid.

Cloud Bypassing on Life Science Grid (LSG) Compute clusters at ±10 locations in the Netherlands

Cloud Bypassing on Life Science Grid (LSG) Offloading campus networks (some only have 1 Gb/s Internet)

LHCONE Innovation: Offloading is cheaper Internet full routing (incl backup): € 8k – €10k per month for 10 Gb/s LHCONE or dynamic lightpath: € 2k – 3k per month for 10 Gb/s

Questions? Erik Ruiter Bas Kreukniet Diederik Vandevenne Sander Boele Farhad Davani Freek Dijkstra