BNL Network Status and dCache/Network Integration Dantong Yu USATLAS Computing Facility Brookhaven National Lab

2 USATLAS Tier 1 Network Outline  Tier 1 Networks.  dCache and Network Integration.  Tier 0 Data Exports Performance.  Tier 2 Sites Networks.  Network Monitoring, and 24*7 Operations.  Network Research  Network Future Plan- Direct Tier 1 to Tier 1, Tier 1 and USATLAS Tier 2 connectivity.

BNL Tier 1 Networks: A Zoom-out View

4 BNL 20 Gig-E Architecture Based on CISCO65xx  20 GBps LAN for LHCOPN  20GBps for Production IP  Full Redundant and survive the failure of any network switch.  No Firewall for LHCOPN, as shown in the green lines.  Two Firewalls for all other IP networks.  Cisco Firewall Services Module (FWSM), a line card plugged into CISCO chassis with 5*1Gbps capacity, allows outgoing connection (except http and https ports).

5 BNL and Long Island MAN Ring ESnet demarc Cisco 6509 LI MAN – Diverse dual core connection 32 AoA, NYC Brookhaven National Lab, Upton, NY 10GE circuit BNL IP/LHC gateway 10 Gb/s circuits International USLHCnet circuits (proposed) production IP core SDN/provisioned virtual circuits 2007 circuits/facilities LI MAN DWDM MAN LAN GEANT CERN Chicago (ESnet IP core) Europe Washington (ESnet IP core) Abilene NYSERNet SINet (Japan) CANARIE (Canada) HEANet (Ireland) Qatar DWDM ring (KeySpan Communications) T320 ESnet IP core 2007 or 2008 second MAN switch USLHCnet Chi USLHCnet ESnet MAN ESnet MAN

6 Other connections MAN LAN CERN (?) NLR ESnet GEANT, etc. BNL internal BNL Redundant Diverse Network Connection

TWC SNLL YUCCA MT BECHTEL-NV PNNL LIGO INEEL LANL SNLA Allied Signal PANTEX ARM KCP NOAA OSTI ORAU SRS JLAB PPPL Lab DC Offices MIT ANL BNL FNAL AMES NREL LLNL GA DOE-ALB OSC GTN NNSA International (high speed) 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (≥ 10 G/s) Lab supplied links OC12 ATM (622 Mb/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less NNSA Sponsored (12) Joint Sponsored (3) Other Sponsored (NSF LIGO, NOAA) Laboratory Sponsored (6) 42 end user sites SINet (Japan) Russia (BINP) CA*net4 France GLORIAD (Russia, China) Korea (Kreonet2 Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren ESnet IP core: Packet over SONET Optical Ring and Hubs ELP DC commercial peering points MAE-E PAIX-PA Equinix, etc. PNWGPoP/ PAcificWave ESnet core hubs IP Abilene high-speed peering points with Internet2/Abilene Abilene CERN (USLHCnet DOE+CERN funded) GÉANT - France, Germany, Italy, UK, etc NYC Starlight SNV Abilene JGI LBNL SLAC NERSC SNV SDN SDSC Equinix SNV ALB ORNL CHI MREN Netherlands StarTap Taiwan (TANet2, ASCC) NASA Ames AU SEA CHI-SL MAN LAN Abilene Specific R&E network peers Other R&E peering points UNM MAXGPoP AMPATH (S. America) ESnet Science Data Network (SDN) core R&E networks Office Of Science Sponsored (22) ATL NSF/IRNC funded Equinix IARC ESnet3 Today Provides Global High-Speed Internet Connectivity for DOE Facilities and Collaborators

dCache WAN interface architecture and integration

9 20 Gb/s HPSS Mass Storage System dCache SRM and Core Servers Gridftp door (7 nodes) WAN 2x10 Gb/s LHC OPN VLAN Write Pool (13 nodes / TB) Farm Pool (434 nodes / 360 TB) 7 x 1 Gb/s Tier 1 VLANS 10 Gb/s 7 x 1 Gb/s dCache.... N x 1 Gb/s Gb/s Logical Connections FTS controlled Srmcp path T0 Export Pool (>=30 nodes) New Farm Pool (80 nodes, 360TB Raw ) Thumpers (30 nodes, 720TB Raw ) dCache and Network Integration 5.4 TB storage on the write pool is off-line

10 BNL dCache and Network Integration  Data Import and Export:  Preferred and fully supported: FTS Glite-URL-COPY, data transfer goes through GridFtp server nodes.  Less desired and partially supported: srmcp direct end to end transfer. goes through CISCO firewall, bottleneck <<5*1Gbps.  Advantages  Less exposure to WAN, only limited nodes with firewall conduits.  Bulk of data transfer managed by FTS, and by-passing firewall.  Firewall can handle a negligible load generated by users directly using srmcp.  Performance can be scaled up by adding extra Grid server nodes.

Tier 0 Data Exports Performance

12 Megatable Extract Tier1 Centre ALICEATLASCMSLHCbTarget IN2P3, Lyon GridKA, Germany CNAF, Italy FNAL, USA BNL, USA RAL, UK NIKHEF, NL ASGC, Taipei PIC, Spain Nordic Data Grid Facility TRIUMF, Canada US ALICE TOTALS

13 ATLAS Tier 0 Data Export Dashboard Last hour Last Four Hours Last Day

14 Ganglia Plots for the Aggregated Data Into dCache

Tier 2 Network Connectivity

16 ATLAS Great Lakes Tier 2

17 Midwest Tier 2

18 Northeast Tier 2

19 Southwest Tier2

20 Western Tier 2: SLAC

21 Network Operations and Monitoring  Cacti  Replacement for MRTG  SNMP monitoring tool  Tracks most BNL core network interfaces  Firewall Service Module EtherChannel interfaces also  Public available at

22 BNL Off-Hour Coverage for Network Operation  Off-hour phone calls are handled by a trained helpdesk analyst 24 hours a day 7 days a week.  Help desk does initial triage and forwards the call to Network on-call person.  On-call person has all contacts information to ESnet NOC, USLHCNOC, CERN NOC.

23 TeraPaths  The problem: support efficient/reliable/predictable peta-scale data movement in modern high-speed networks  Multiple data flows with varying priority  Default “best effort” network behavior can cause performance and service disruption problems  Solution: enhance network functionality with QoS features to allow prioritization and protection of data flows  Treat network as a valuable resource  Schedule network usage (how much bandwidth and when)  Techniques: DSCP, MPLS, and VLAN.  Collaborate with ESnet (OSCAR) and Internet 2 (DRAGON) to dynamically create end to end paths, and dynamically forward traffic into the paths. Is being deployed to USATLAS Tier 2 sites.  Option 1: Layer 3: MPLS (supported)  Option 2: Layer 2: VLAN (under development)

24 TeraPaths System Architecture Site A (initiator) Site B (remote) WAN chain web services WAN monitoring WAN web services hardware drivers Web page APIs Cmd line QoS requests user manager scheduler site monitor … router manager user manager scheduler site monitor … router manager WAN chain WAN web services

Conclusions and Network Discussion Points

26 Conclusions  BNL Network has been stable and significantly improved since we had 20Gbps upgrade.  Tier 0 and Tier 1 data transfer rides on LHCOPN network, while Tier 1 to BNL still uses IP production network.  BNL Network utilization is less than 30% of 20Gbps. We have not been able to push data transfer close to the network bandwidth limitation.  Full redundancy has been built in LAN.  WAN (USLHCNetwork) redundancy is being investigated.  dCache and BNL LAN is fully integrated. It is an optimized trade-off between network security and performance.

27 Discussion Points  T1 to T1 transit via T0: data transfers between T1 centers transiting to the T0, is technically feasible, however, it is not implemented.  Direct Tier 1 and Tier 1 connection:  A Layer 2 connection between FNAL and IN2P3 via ESnet and GEANT was setup.  BNL/TRIUMF (ready) and BNL/Prague (in planning).  How about BNL/IN2P3 and BNL/FZK?  BNL needs to work with ESnet, (USLHCNET) and IN2P3 and FZK with GEANT. We need to work on both ends simultaneously.  Tier 1 and Tier 2.