ESnet Network Engineer Lawrence Berkeley National Laboratory ESnet Update ESnet/Internet2 Joint Techs Albuquerque, New Mexico February 6, 2005 R. Kevin Oberman ESnet Network Engineer Lawrence Berkeley National Laboratory

Physical Connectivity Outline Physical Connectivity What’s new BAMAN in production LIMAN nearing production CHIMAN under construction New Circuits Traffic Driving: Interconnectivity Architecture

ESnet SF Bay Area MAN Ring NLR to Seattle IP core to Chicago (Qwest) 2 λs (2 X 10 Gb/s channels) in a ring configuration, and delivered as 10 GigEther circuits - 10-50X current site bandwidth Dual site connection (independent “east” and “west” connections) to each site Used as a 10 Gb/s production IP ring and 2 X 10 Gb/s paths (for circuit services) to each site Project completed 9/2005, on time and on budget Joint Genome Institute 10 Gb/s optical channels LBNL λ1 production IP λ2 SDN/circuits NERSC λ3 future λ4 future SF Bay Area LLNL SNLL SLAC ESnet MAN ring (Qwest circuits) Qwest / ESnet hub DOE Ultra Science Net (research net) Level 3 hub ESnet hubs and sites NLR to San Diego NASA Ames IP core to El Paso

Long Island MAN Initial Configuration CERN Chicago (Qwest hub) Starlight Europe Brookhaven National Lab, Upton, NY Abilene NYSERNet SINet (Japan) CANARIE (Canada) HEANet (Ireland) Qatar GEANT ESnet boarder demarcation device 10GE circuit MAN LAN USLHCnet LI MAN DWDM T320 Cisco 6509 ESnet IP core 1 GE or 10 GE 32 AoA, NYC BNL IP gateway ESnet IP core (Qwest) 10 Gb/s circuits International LHCnet circuits (proposed) production IP core SDN/provisioned virtual circuits Washington (Qwest hub)

Chicago MAN 4x10G Starlight 455 N. City Front 3x10G Fermilab Argonne USLHCnet 4x10G T320 Starlight 455 N. City Front 3x10G ESnet IP SDN Fermilab Argonne

New 10G Circuits ESnet is nearing completion of a process to turn up 44 new 10G circuits in less than 12 months including: MANs 16 circuits composing the San Francisco Bay Area MAN 2 circuits comprising the Long Island MAN (New York to Brookhaven) 15 circuits comprising the Chicago MAN (FNAL, ANL and StarLight) NLR 3 circuits: San Diego to Sunnyvale, Sunnyvale to Seattle, hub to hub in Sunnyvale Sites 1 circuit JLAB (via, MAX/NGIX-E, MATP, E-LITE) 1 circuit between NYC and DC to reach GEANT 1 circuit ORNL – Starlight - CHI-HUB(NBC) Peering 1 circuit GEANT 1 circuit SNV – NASA-AMES

Other Circuits Completed: In the works: OC48 ALB-ELP DS3 IARC-SNV (New Site) OC3 ALB-KCP In the works: OC3 ALB-HUB-PANTEX <-- In build-out

The Office of Science U.S. Community* Pacific Northwest National Laboratory Idaho National Laboratory Ames Laboratory Argonne National Laboratory Brookhaven National Laboratory Fermi National Accelerator Laboratory Lawrence Berkeley National Laboratory Stanford Linear Accelerator Center Princeton Plasma Physics Laboratory Lawrence Livermore & Sandia National Laboratories Thomas Jefferson National Accelerator Facility Correct SLAC representation General Atomics Oak Ridge National Laboratory Sandia National Laboratory Los Alamos National Laboratory Institutions supported by SC Major User Facilities National Renewable Energy Laboratory DOE Specific-Mission Laboratories DOE Program-Dedicated Laboratories SC Program sites DOE Multiprogram Laboratories

ESnet’s Physical Connectivity (Spring 2006)* Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren SINet (Japan) Russia (BINP) GÉANT - France, Germany, Italy, UK, etc CA*net4 France GLORIAD (Russia, China) Korea (Kreonet2 MREN Netherlands StarTap Taiwan (TANet2) UltraLight PNWGPoP/ PAcificWave CERN (USLHCnet CERN+DOE funded) SEA NLR Supplied Circuits LIGO MAN LAN Abilene PNNL AU ESnet IP core MAN rings Starlight CHI-SL Abilene NYC MIT USN Abilene INEEL Chi NAP BNL JGI LVK FNAL LLNL LBNL AMES ANL ORAU DC PPPL SNLL NERSC LBNL DC SNV CHI LLNL/LANL DC Offices MAE-E SLAC PAIX-PA Equinix, etc. OSC GTN NNSA Equinix SNV SDN BECHTEL-NV NREL NASA Ames USN KCP DC JLAB ORNL YUCCA MT OSTI ORAU LANL MAXGPoP IARC ARM Abilene ATL SDSC SNLA NOAA AU GA SRS AMPATH ALB KCP-ALB PANTEX UNM SoXGPoP 42 end user sites DOE-ALB AMPATH (S. America) Office Of Science Sponsored (22) ELP NNSA Sponsored (12) International (high speed) Lab Supplied 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (≥ 10 G/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less Joint Sponsored (4) Other Sponsored (NSF LIGO, NOAA) ESnet IP core: Packet over SONET Optical Ring and Hubs Laboratory Sponsored (6) commercial and R&E peering points IP Abilene SNV ESnet core hubs high-speed peering points with Internet2/Abilene

Network Utilization ESnet Accepted Traffic (Bytes) Jan, 1990 to Dec., 2005 ESnet is Currently Transporting About 750 Terabytes/mo. and this volume is increasing exponentially TBytes/Month

ESnet traffic has increased by about 10X every 46 months, on average, Network Utilization ESnet traffic has increased by about 10X every 46 months, on average, since 1990 Dec., 2001 Jul., 1998 42 months TBytes/Month Oct., 1993 57 months Aug., 1990 39 months

OSCARS: Guaranteed Bandwidth VC Service For SC Science ESnet On-demand Secured Circuits and Advanced Reservation System (OSCARS) To ensure compatibility, the design and implementation is done in collaboration with the other major science R&E networks and end sites Internet2: Bandwidth Reservation for User Work (BRUW) Development of common code base GEANT: Bandwidth on Demand (GN2-JRA3), Performance and Allocated Capacity for End-users (SA3-PACE) and Advance Multi-domain Provisioning System (AMPS) Extends to NRENs BNL: TeraPaths - A QoS Enabled Collaborative Data Sharing Infrastructure for Peta- scale Computing Research GA: Network Quality of Service for Magnetic Fusion Research SLAC: Internet End-to-end Performance Monitoring (IEPM) USN: Experimental Ultra-Scale Network Testbed for Large-Scale Science In its current phase this effort is being funded as a research project by the Office of Science, Mathematical, Information, and Computational Sciences (MICS) Network R&D Program A prototype service has been deployed as a proof of concept To date more then 20 accounts have been created for beta users, collaborators, and developers More then 100 reservation requests have been processed

Measurements of this kind are science-agnostic Observation In 4 years, we can expect a 10x increase in traffic over current levels without the addition of production LHC traffic Nominal average load on busiest backbone links is ~1.5Gbps today In 4 years that figure will be ~15Gbps if current trends continue Measurements of this kind are science-agnostic It doesn’t matter who the users are, the traffic load on the backbone is increasing Bandwidth trends drive requirement for a new network architecture

Metropolitan Area Rings ESnet Target Architecture: IP Core+Science Data Network Core+Metro Area Rings Europe (GEANT) Canada (CANARIE) CERN Asia-Pacific Canada (CANARIE) CERN Europe (GEANT) Seattle SDN Core Australia Chicago New York Denver IP Core Sunnyvale Washington, DC Metropolitan Area Rings Atlanta Loop off Backbone On this diagram you can see how all the components come together. You can see how the MANs tie the two backbone rings together How the most international peering points are connected to both backbones via the MANs LA Aus. Albuquerque South America (AMPATH) San Diego IP core hubs 10-50 Gb/s circuits Production IP core Science Data Network core Metropolitan Area Networks International connections SDN hubs South America (AMPATH) Primary DOE Labs possible hubs