T1-NREN Luca dell’Agnello CCR, 21-Ottobre-2005

The problem Computing for LHC experiments –Multi tier model (MONARC) –LHC computing based on grid –Experiment computing models LHC Computing Grid (LCG) –Policy Executive Board (PEB) –Grid Deployment Board (GDB) Transfer LHC data from CERN to external computing facilities –Players: TO, T1s, NREN, DANTE

Task forces & activities Group of interest on network set up in 2004 by NRENs –Some T1s, NRENs, CERN (D Foster), DANTE Service Challenge activity (December 2004  service phase) –Progressive tests of data transfer chain in cooperation with experiments –ramp up to LHC data taking phase Second group (LHCOPN wg) started by GDB in January 2005 –T1s, CERN, NRENs, DANTE represented European NRENs, ESNET, CANARIE, ASNET –Chair: D. Foster (CERN) –Meetings: January 2005 (SARA/NIKHEF) 8 April 2005 (SARA/NIKHEF) 19 July 2005 (CERN) Next: November 14 (SC05, Seattle)

LHCOPN wg activity Preparation and control of plans for implementation of WAN connectivity requested from the LHC experiments’ Computing Models Ensure that individual agreements among T1s will provide a coherent infrastructure to satisfy the LHC experiments' Computing Models and that there is a credible solution for the management of the end-to-end network services First priority: to plan the networking for the Tier-0 and Tier-1 centres –should also cover Tier-2 centres, as appropriate information on requirements becomes available The group reports regularly to the GDB Subgroups –IP Addressing and Routing –Monitoring –Operations –Security

LHC “instrument” The LHC and its data collection systems The data processing and storage units at CERN, i.e. T0 The data processing and storage sites called T1 The data processing and storage sites called T2 Associated networking between all T0, T1, and T2 sites. TO-T1s 10 Gbps permanent light paths form the Optical Private Network (OPN) for the LHC instrument LHCOPN

Who does what CERN will –provide interfaces for T1’s link terminations at CERN –host T1's equipment for T0-T1 link termination at CERN (if requested) T1s will –organise physical connectivity from the T1's premises to the T0 –make available network equipment for termination point of T1-T0 link at the T1 side light path termination will be at T1 premises (as in INFN-CNAF case) or at NREN POP –be ready at full bandwidth not later than Q SCs need to test the system (from network to applications) up to full capacity production environment

Network architecture (1) At least one dedicated 10 Gbit/s light path between T0 and each T1 –every T0-T1 link should handle only production LHC data –T1 to T1 traffic via the T0 allowed BUT T1s encouraged to provision direct T1-T1 connectivity –T1-T2 and T0-T2 traffic will normally be handled by the normal L3 connectivity provided by NRENs Backup through L3 paths across NRENs discouraged (potential heavy interference with general purpose Internet connectivity of T0 or the T1s) Main connection Backup connection L3 Backbones Tier0 Tier1s Tier2s main connection backup connection

Network architecture (2) LHC prefixes –Every T1 and the T0 must allocate publicly routable IP address space (the "LHC prefixes”) aggregated into single CIDR block –T1s and T2s to exchange traffic directly with the T0 must provide the T0 with the list of its LHC prefixes (  next slide) –LHC prefixes should be dedicated to the LHC network traffic Routing among T0 and T1s sites will be achieved using eBGP (no static routes!) –No default route must be used in T1-T0 routing –Every T1 will announce its own LHC prefixes to T0 –T0 will accept from a T1 only it own LHC prefixes plus prefixes of any T1 or T2 for which that T1 is willing to provide transit for –TO will announce its LHC prefixes and all the LHC prefixes received in BGP to every peering T1 inter T1s traffic through T0 allowed but not encouraged –T1 will accept T0's prefixes, plus, if desired, some selected T1's prefixes –T0 and T1s should announce their LHC prefixes to their upstream continental research networks (GÉANT2, Abilene, ESnet) in order to allow connectivity towards the T2s Every Tier must make sure that any of its own machines within the LHC prefix ranges can reach any essential service (for instance the DNS system)

Network architecture: the T2s T2s usually upload and download data via a particular T1 It is possibile to provide transit for a T2 to the T0, if a T1 announces the T2's prefixes to T0 and the T0 open all the security barriers for it –BUT this assumes a “static” allocation of a T2 to a particular T1 It is assumed that the T1-T2 and T0-T2 traffic will be handled by the normal L3 connectivity provided by NRENs. The announcement of prefixes associated with Tier 2 sites is deprecated and any site has the right ignore such announcements.

Security on LHCOPN: the basics Security contact person (+ deputy) needed for each site –Note that a T1 has also security contact persons for its own NREN, LCG/EGEE and maybe at HEPix level Incident handling and reporting –Local site procedures for security incident handling will take precedence –Security incident at an OPN site will be reported to the LHCOPN representatives –The report will provide a description of the incident with an assessment of the risk posed to other OPN sites –A site receiving such an incident report, the nominated OPN Security representative will share this information and will abide by the requirements of the local Security officer

Security on LHCOPN: implementation It is not possible to rely on firewalls –throughput and performance problems ACL-based network security acceptable –It is not a general access network –Low number of expected LHC prefixes –Low number of expected (Grid) applications ACL general schema (discussion still on going) –Applied inbound and outbound at T0 and T1s borders –Only traffic originating & directed to LHC prefixes allowed on LHCOPN –IP based ACLs to control traffic from and to end points –Transit T1-T1 through TO for LHC prefixes allowed –Extended ACLs should also be used where source/destination port numbers can be associated with data flows

LHCOPN Operation Issues –Circuits provided and managed by NRENs, GEANT,… –IP termination points at T0/T1s Need for some sort of coordination and monitor of LHCOPN (LHCOPN NCC ?) –Model still under discussion –Proactive IP level monitoring Coordination of multi circuit problems Escalation to circuit provider/NREN –Tiers contact LHCOPN NCC

Italian LHCOPN Architecture PoP Geant2 - IT AT GARR CNAF RT1.BO1 RT.RM1 RT.MI1 CERN eBGP AS513 – AS137 T1 T0 T1 T2 10GE- LAN STM-64 IP access 10GE-LAN lightpath access GFP-F STM-64 10G – leased λ’s PoP Geant2 - CH

LAN connectivity based on 10GE technology with capacity for 10 GE link aggregation Data flows will terminate on disk buffer system (possibly CASTOR, but also other SRM systems under evaluation) 1 network prefix for LHCOPN Security model will be based on L3 filters (ACLs) within the L3 equipment Monitoring via snmp (presently v2) INFN Tier1

GARR T1 INFN-CNAF WAN connectivity BD CISCO 7600 CNAF Juniper GARR 10 Gbps 1 Gbps /20 default Link LHCOPN default T / /16 GEANT

Glossary LHC Network Traffic: The data and control traffic that flows between T0, the T1s, and the T2s. LHC prefixes: IP address space allocated by the T0 and T1s and assigned to the machines connected to the LHC-OPN. Light path: 1.a point to point circuit based on WDM technology 2.a circuit-switched channel between two end points with deterministic behaviour based on TDM technology 3.concatenations of (1) and (2). NREN: usually National Research and Education Network; here used as the collective name for a network that service either the research community or the education community or both.