1 ESnet Update ESnet/Internet2 Joint Techs Albuquerque, New Mexico February 6, 2005 R. Kevin Oberman ESnet Network Engineer Lawrence Berkeley National Laboratory

2 Outline  Physical Connectivity oWhat’s new oBAMAN in production oLIMAN nearing production oCHIMAN under construction oNew Circuits  Traffic Driving: oInterconnectivity oArchitecture

IP core to Chicago (Qwest) IP core to El Paso NLR to Seattle NLR to San Diego SF Bay Area λ 4 future λ 3 future λ 2 SDN/circuits λ 1 production IP SLAC Qwest / ESnet hub SNLL Joint Genome Institute LBNL NERSC LLNL NASA Ames Level 3 hub ESnet hubs and sites DOE Ultra Science Net (research net) ESnet MAN ring (Qwest circuits) ESnet SF Bay Area MAN Ring 10 Gb/s optical channels 2 λs (2 X 10 Gb/s channels) in a ring configuration, and delivered as 10 GigEther circuits X current site bandwidth Dual site connection (independent “east” and “west” connections) to each site Used as a 10 Gb/s production IP ring and 2 X 10 Gb/s paths (for circuit services) to each site Project completed 9/2005, on time and on budget

4 Abilene NYSERNet SINet (Japan) CANARIE (Canada) HEANet (Ireland) Qatar Long Island MAN Initial Configuration ESnet boarder demarcation device 32 AoA, NYC Brookhaven National Lab, Upton, NY 10GE circuit BNL IP gateway 1 GE or 10 GE MAN LAN ESnet IP core GEANT Europe Washington (Qwest hub) 10 Gb/s circuits International LHCnet circuits (proposed) production IP core SDN/provisioned virtual circuits LI MAN DWDM ESnet IP core (Qwest) CERN Chicago (Qwest hub) Starlight T320 USLHCnet Cisco 6509

5 Chicago MAN Starlight455 N. City Front FermilabArgonne T320 USLHCnet ESnet IP SDN 3x10G 4x10G

6 New 10G Circuits ESnet is nearing completion of a process to turn up 44 new 10G circuits in less than 12 months including: o MANs -16 circuits composing the San Francisco Bay Area MAN -2 circuits comprising the Long Island MAN (New York to Brookhaven) -15 circuits comprising the Chicago MAN (FNAL, ANL and StarLight) o NLR -3 circuits: San Diego to Sunnyvale, Sunnyvale to Seattle, hub to hub in Sunnyvale o Sites -1 circuit JLAB (via, MAX/NGIX-E, MATP, E-LITE) -1 circuit between NYC and DC to reach GEANT -1 circuit ORNL – Starlight - CHI-HUB(NBC) o Peering -1 circuit GEANT -1 circuit SNV – NASA-AMES

7 Other Circuits Completed: o OC48 ALB-ELP o DS3 IARC-SNV (New Site) o OC3 ALB-KCP In the works: o OC3 ALB-HUB-PANTEX <-- In build-out

8 The Office of Science U.S. Community* Institutions supported by SC Major User Facilities DOE Multiprogram Laboratories DOE Program-Dedicated Laboratories DOE Specific-Mission Laboratories Pacific Northwest National Laboratory Ames Laboratory Argonne National Laboratory BrookhavenNationalLaboratory Oak Ridge National Laboratory Los Alamos National Laboratory Lawrence Livermore &Sandia National Laboratories Lawrence Berkeley National Laboratory FermiNationalAcceleratorLaboratory PrincetonPlasmaPhysicsLaboratory Thomas Jefferson National Accelerator Facility National Renewable Energy Laboratory StanfordLinearAcceleratorCenter Idaho National Laboratory SC Program sites General Atomics Sandia National Laboratory

9 NLR Supplied Circuits LVK SNLL YUCCA MT BECHTEL-NV PNNL LIGO INEEL SNLA PANTEX ARM KCP NOAA OSTI ORAU SRS JLAB PPPL MIT ANL BNL FNAL AMES LLNL GA OSC GTN NNSA International (high speed) Lab Supplied 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (≥ 10 G/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less Office Of Science Sponsored (22) NNSA Sponsored (12) Joint Sponsored (4) Other Sponsored (NSF LIGO, NOAA) Laboratory Sponsored (6) 42 end user sites ESnet IP core CA*net4 France GLORIAD (Russia, China) Korea (Kreonet2 Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren ESnet IP core: Packet over SONET Optical Ring and Hubs ELP ATL DC commercial and R&E peering points MAE-E PAIX-PA Equinix, etc. PNWGPoP/ PAcificWave  ESnet’s Physical Connectivity (Spring 2006)* ESnet core hubs IP Abilene high-speed peering points with Internet2/Abilene Abilene CERN (USLHCnet CERN+DOE funded) GÉANT - France, Germany, Italy, UK, etc NYC Starlight Chi NAP CHI-SL SNV Abilene SNV SDN JGI LBNL SLAC NERSC SDSC Equinix MAXGPoP SoXGPoP SNV ALB ORNL CHI MREN Netherlands StarTap Taiwan (TANet2) UltraLight AU SEA SINet (Japan) Russia (BINP) LANL NREL AMPATH (S. America) AMPATH MAN LAN Abilene MAN rings Abilene USN IARC NASA Ames KCP-ALB UNM LLNL/LANL DC Offices LBNL DC ORAU DC DOE-ALB

10 Network Utilization TBytes/Month ESnet Accepted Traffic (Bytes) Jan, 1990 to Dec., 2005 ESnet is Currently Transporting About 750 Terabytes/mo. and this volume is increasing exponentially

11 Network Utilization Oct., 1993 Aug., 1990 Jul., months 57 months 42 months ESnet traffic has increased by about 10X every 46 months, on average, since 1990 Dec., 2001 TBytes/Month

12  OSCARS: Guaranteed Bandwidth VC Service For SC Science ESnet On-demand Secured Circuits and Advanced Reservation System (OSCARS) To ensure compatibility, the design and implementation is done in collaboration with the other major science R&E networks and end sites o Internet2: Bandwidth Reservation for User Work (BRUW) -Development of common code base o GEANT: Bandwidth on Demand (GN2-JRA3), Performance and Allocated Capacity for End-users (SA3-PACE) and Advance Multi-domain Provisioning System (AMPS) -Extends to NRENs o BNL: TeraPaths - A QoS Enabled Collaborative Data Sharing Infrastructure for Peta- scale Computing Research o GA: Network Quality of Service for Magnetic Fusion Research o SLAC: Internet End-to-end Performance Monitoring (IEPM) o USN: Experimental Ultra-Scale Network Testbed for Large-Scale Science In its current phase this effort is being funded as a research project by the Office of Science, Mathematical, Information, and Computational Sciences (MICS) Network R&D Program A prototype service has been deployed as a proof of concept o To date more then 20 accounts have been created for beta users, collaborators, and developers o More then 100 reservation requests have been processed

13 Observation In 4 years, we can expect a 10x increase in traffic over current levels without the addition of production LHC traffic o Nominal average load on busiest backbone links is ~1.5Gbps today o In 4 years that figure will be ~15Gbps if current trends continue Measurements of this kind are science-agnostic o It doesn’t matter who the users are, the traffic load on the backbone is increasing Bandwidth trends drive requirement for a new network architecture

Gb/s circuits Production IP core Science Data Network core Metropolitan Area Networks International connections Metropolitan Area Rings Primary DOE Labs IP core hubs ESnet Target Architecture: IP Core+Science Data Network Core+Metro Area Rings possible hubs SDN hubs Europe (GEANT) Asia-Pacific New York Chicago Washington, DC Atlanta CERN Seattle Albuquerque Aus. Australia San Diego LA Sunnyvale Denver South America (AMPATH) South America (AMPATH) Canada (CANARIE) CERN Loop off Backbone Canada (CANARIE) Europe (GEANT) SDN Core IP Core