1 ESnet Planning for the LHC T0-T1 Networking William E. Johnston ESnet Manager and Senior Scientist Lawrence Berkeley National Laboratory

ESnet Science Data Network (SDN) core TWC SNLL YUCCA MT BECHTEL-NV PNNL LIGO INEEL LANL SNLA Allied Signal PANTEX ARM KCP NOAA OSTI ORAU SRS JLAB PPPL INEEL-DC ORAU-DC LLNL/LANL-DC MIT ANL BNL FNAL AMES 4xLAB-DC NREL LLNL GA DOE-ALB GTN&NNSA International (high speed) 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (≥ 10 G/s) OC12 ATM (622 Mb/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less Office Of Science Sponsored (22) NNSA Sponsored (12) Joint Sponsored (3) Other Sponsored (NSF LIGO, NOAA) Laboratory Sponsored (6) QWEST ATM 42 end user sites ESnet IP core SINet (Japan) Japan – Russia (BINP) CA*net4France GLORIADKreonet2 MRENNetherlands StarTapTANet2 Taiwan (ASCC) Australia CA*net4 Taiwan (TANet2) Singaren ESnet IP core: Packet over SONET Optical Ring and Hubs ELP HUB ATL HUB DC HUB peering points MAE-E PAIX-PA Equinix, etc. PNWGPoP SEA HUB ESnet High-Speed Physical Connectivity to DOE Facilities and Collaborators (Summer 2005) IP core hubs SNV HUB Abilene high-speed peering points Abilene CERN (LHCnet – part DOE funded) GEANT - Germany, France, Italy, UK, etc NYC HUB Starlight Chi NAP CHI-SL HUB SNV HUB Abilene SNV SDN HUB JGI LBNL SLAC NERSC SND core hubs SDSC HUB Equinix MAN LAN Abilene MAXGPoP SoXGPoP SNV SDN HUB ALB HUB ORNL CHI HUB SINet, AARnet

3 ESnet The IP core is primarily a layer 3 infrastructure o However, supports layer 2 via MPLS o Directly connects sites o Provides global peering for sites The SDN core is primarily a layer 2 infrastructure o Targeted at providing virtual circuit services Site connectivity is evolving to be Metro Area Network rings Goal is two independent core rings that are interconnected at various locations with metro area rings

Metropolitan Area Rings Primary DOE Labs IP core hubs ESnet Target Architecture: IP Core+Science Data Network Core+Metro Area Rings New hubs SDN/NLR hubs GEANT (Europe) Asia-Pacific New York Chicago Washington, DC El Paso Atlanta CERN Seattle Albuquerque Aus. Australia Science Data Network Core (SDN) (NLR circuits) IP Core San Diego LA Gb/s circuits Production IP core Science Data Network core Metropolitan Area Networks Lab supplied International connections Sunnyvale

Metropolitan Area Rings Primary DOE Labs IP core hubs Evolution of ESnet – Step One: SF Bay Area MAN and West Coast SDN New hubs SDN/NLR hubs GEANT (Europe) Asia-Pacific New York Chicago Washington, DC El Paso Atlanta CERN Seattle Albuquerque Aus. Australia Science Data Network Core (SDN) (NLR circuits) IP Core (Qwest) San Diego LA Production IP core Science Data Network core Metropolitan Area Networks Lab supplied International connections Sunnyvale In service by Sept., 2005 planned

6 LHC Requirements 10G “light paths” from FNAL and BNL to CERN o CERN/LHCnet will provide10G circuit to Starlight and likely another to 32 AoA BNL must communicate with TRIUMF (Vancouver) o Data rates? Backup paths must be provided o 10G Starlight to MAN LAN o Others Access to T2 sites o What characteristics? o At least 1G guaranteed

7 ESnet Approach for LHC Requirements Starlight CERN FNAL Triumf CANARIE 32 AoA CERN 60 Hudson (or 32 AoA) Sunnyvale BNL SDN Qwest Chicago IP SDN GEANT All paths are 10 Gb/s FNALESnetIP SDN ESnetFNAL ESnet IP core ESnet SDN core SLLHC MAN LAN

8 DEN ELP ALB ATL Metropolitan Area Rings ESnet Goal – 2007/2008 Aus. CERN Europe SDG AsiaPac SEA Major DOE Office of Science Sites High-speed cross connects with Internet2/Abilene New ESnet hubs ESnet hubs SNV Europe Japan CHI Science Data Network core Lab supplied Major international Production IP ESnet core DC Japan NYC Aus. Metropolitan Area Rings 10 Gbps enterprise IP traffic Gbps circuit based transport ESnet Science Data Network (2nd Core – Gbps, National Lambda Rail) ESnet IP Core (≥10 Gbps) 10Gb/s 30Gb/s 40Gb/s CERN Europe

9 Proposed ESnet Upgrade Schedule Fiscal Year 2005 o FNAL-ANL-Chicago Metropolitan Area Network Fiscal Year 2006 o BNL MAN Fiscal Year 2007 o Extend Science Data Network (SDN) from Seattle=>New York/Washington, DC at 10G. Jefferson Lab MAN. Fiscal Year 2008 o Science Data Network (SDN) loop around the country at10G. ORNL MAN. Fiscal Year 2009 o Double main Science Data Network (SDN) links to 20 Gigabits. Fiscal Year 2010 o Increase main Science Data Network (SDN) links to 30 Gigabits per sec. PPNL MAN.

Denver Seattle Sunnyvale LA San Diego Chicago Pitts Wash DC Raleigh Jacksonville Atlanta KC Baton Rouge El Paso - Las Cruces Phoenix Pensacola Dallas San Ant. Houston Albuq. Tulsa New York Clev ESnet Lambda Infrastructure Based on National Lambda Rail (incremental install ) NLR wavegear sites NLR regeneration / OADM sites Boise

11 FNAL ESnet site gateway router site equip. Qwest hub (NBC Bld.) Starlight ESnet IP core FNAL Shared w/ IWire ESnet ESnet Near-Term Planning for FNAL (  mid-late 2005) FNAL CERN ESnet Switch/RTR ESnet/ Qwest ORNL OC192 T320 NRL, UltraScienceNet, etc. T320 ESnet SDN core All circuits are 10Gb/s ESnet fiber (single lambda) Notes Qwest IWire FNAL T320 Starlight sw

12 ESnet Planning for BNL and Long Island MAN Ring ESnet demarc Cisco 6509 LI MAN – Diverse dual core connection 32 AoA, NYC Brookhaven National Lab, Upton, NY 10GE circuit 10GE circuit BNL IP gateway 1 GE or 10 GE ESnet IP core Chicago (Syracuse – Cleveland – Chicago) ESnet SDN core (NLR circuits) Washington 10 Gb/s circuits International LHCnet circuits (proposed) production IP core SDN/provisioned virtual circuits 60 Hudson, NYC ESnet SDN core router NLR OADM LI MAN DWDM Engineering Study for LI MAN MAN LAN GEANT CERN Chicago (Qwest hub) Europe Washington (Qwest hub) Starlight ESnet IP core (Qwest) Abilene NYSERNet SINet (Japan) CANARIE (Canada) HEANet (Ireland) Qatar T320 M G

13 ESnet Planning for BNL and Long Island MAN Ring ESnet demarc Cisco 6509 LI MAN – Diverse dual core connection 32 AoA, NYC Brookhaven National Lab, Upton, NY 10GE circuit 10GE circuit BNL IP gateway 1 GE or 10 GE ESnet IP core Chicago (Syracuse – Cleveland – Chicago) ESnet SDN core (NLR circuits) Washington 10 Gb/s circuits International LHCnet circuits (proposed) production IP core SDN/provisioned virtual circuits 60 Hudson, NYC ESnet SDN core router NLR OADM LI MAN DWDM Engineering Study for LI MAN MAN LAN GEANT CERN Chicago (Qwest hub) Europe Washington (Qwest hub) Starlight ESnet IP core (Qwest) Abilene NYSERNet SINet (Japan) CANARIE (Canada) HEANet (Ireland) Qatar T320 M G

14 Abilene NYSERNet SINet (Japan) CANARIE (Canada) HEANet (Ireland) Qatar ESnet Planning for BNL and Long Island MAN Ring ESnet demarc Cisco 6509 LI MAN – Non-diverse dual core connection 32 AoA, NYC Brookhaven National Lab, Upton, NY 10GE circuit 10GE circuit BNL IP gateway 1 GE or 10 GE MAN LAN ESnet IP core GEANT Europe Washington (Qwest hub) Chicago (Syracuse – Cleveland – Chicago) ESnet SDN core (NLR circuits) Washington 10 Gb/s circuits International LHCnet circuits (proposed) production IP core SDN/provisioned virtual circuits ESnet SDN core router LI MAN DWDM Engineering Study for LI MAN ESnet IP core (Qwest) CERN Chicago (Qwest hub) Starlight M G T320

15 ESnet Planning for BNL and Long Island MAN Ring Engineering Study for LI MAN ** CONFIDENTIAL ** Other connections MAN LAN CERN (?) NLR ESnet GEANT, etc. BNL internal LI MAN – ring detail