1. Bas Kreukniet, Sr Network Specialist at SURF SARA NL-T1 Expectations, findings, and innovation Geneva Workshop 10 Februari 2014

2. Outline 1.Expectations from NL-T1 grid administrators 2.Findings while connecting to the LHCONE 3.Innovation: Ethernet OAM and NSI

3. Expectations from NL-T1 administrators Advice from grid administrators NL-T1: Bulk data with simple applications (grid-FTP). “Keep it simple” Network provisioning from application is considered “complex” Don’t rush to merge LHCOPN with LHCONE

4. NL-T1 connected to LHCONE since 20 Jan 2014

5. BGP routing starts at the connected site. BGP routing for T1’s: Not only a “NREN thing” – it already starts at your organisation! Focus on some BGP topics for connecting sites to LHC networks

6. LHCOPN connectivity

7. LHCONE connectivity 128.142.0.0/16 *[BGP/170] 1w2d 04:07:47, MED 120, localpref 100 AS path: 20965 20641 513 I > to 62.40.126.161 via xe-4/0/1.2012 NL-T1 AS1162 GÉANT AS20965 LHCONE-RS CERN AS20641 CERN AS513 LHCONE.inet.0: 133 destinations, 133 routes (133 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both

8. Preferred route: LHCOPN, LHCONE or Internet primary: direct T1–T1 over LHCOPN secondary: T1–T1 over LHCOPN via another T1 tertiary: LHCONE quaternary: Internet

9. Preferred route: route (a)symmetry If everyone makes this choice, all connections are symmetric. But: sites may have different preferences: 10 Gb/s 1 Gb/s 100 Gb/s site 1 site 2 ISP-B ISP-A

10. Tie-breaker between LHCOPN and LHCONE CNAF routers at NL-T1 LHCOPN 131.154.128.0/17 *[BGP/170] 14:58:18, localpref 100 AS path: 34878 137 I LHCONE 131.154.128.0/17 *[BGP/170] 1w2d 04:23:40, MED 120, localpref 100 AS path: 20965 137 I

11. LHCOPN / LHCONE route preference BNL routes at NL-T1 LHCOPN 130.199.185.0/24 *[BGP/170] 2w5d 09:35:43, MED 10, localpref 100 AS path: 513 43 I > to 192.16.166.73 via xe-1/1/0.0 [BGP/170] 3d 01:15:22, MED 51, localpref 100 AS path: 39590 513 43 I > to 109.105.124.17 via xe-2/1/0.0 LHCONE 130.199.185.0/24 *[BGP/170] 1w2d 04:27:43, MED 120, localpref 100 AS path: 20965 293 43 I > to 62.40.126.161 via xe-4/0/1.2012

12. Problems and concerns regarding BGP A site advertising his routes has no control who to send a route to. At best they can give hints with BGP communities. The site receiving a route decides which route to accept and how to accept.

13. BGP in LHC networks Only As is not enough. More info needed.  community for origin of a route T1/T2? Or even better: -site connected to LHCOPN -site connected to LHCONE - Site connected to both: Type A “prefer LHCONE for this route” Type B “prefer LHCOPN for this route” - Specials: dedicated link between two (T1) sites. (“VPN”or “private link”)

14. Route Preference Solutions Idea: Tag routes with two types of communities: One for origin or source One for destinations See also: BGP hinting by Martin Sweeny (Indiana U) BGP Always-compare-MED always on. We sometimes add metrics on incoming routes.

15. Operational Issues We received routes over LHC from a site, but traffic we send is blackholed by that site The site was still reachable over the Internet. This happened to us twice recently: on LHCOPN (accidental route redistribution) and LHCONE (incoming IP filter). Configuration errors will be made (we’re also just human) LHCOPN Link NL-T1 – TRIUMF link is still unstable 31 outages last 4 months  Monitoring remains important!

16. LHCONE and LHCOPN layer 2 monitoring Ethernet OAM monitoring or Layer 2 monitoring NIKHEF, TRIUMF to participate as Measurement Points Looking for T1’s and T2 to participate Advantages: Layer2 keep-alive, ping and traceroute Interdomain, intervendor solution L2 devices can be made visible unidirectional fibercuts can be signalled

17. LHCONE Innovation NL-T1 likes to partcipate in NSI for LHCONE NSI experiences so far: Use-case: Life Science Grid (LSG) in Holland makes use of NSI since autumn 2013. Freek contributed to standard Sander wrote NSI client and implemented it for “Cloud Bypassing” in the Life Science Grid.

18. Cloud Bypassing on Life Science Grid (LSG) Compute clusters at ±10 locations in the Netherlands

19. Cloud Bypassing on Life Science Grid (LSG) Offloading campus networks (some only have 1 Gb/s Internet)

20. LHCONE Innovation: Offloading is cheaper Internet full routing (incl backup): € 8k – €10k per month for 10 Gb/s LHCONE or dynamic lightpath: € 2k – 3k per month for 10 Gb/s

21. Questions? Erik Ruiter Bas Kreukniet Diederik Vandevenne Sander Boele Farhad Davani Freek Dijkstra