1. 1 The Evolution of ESnet (Summary) William E. Johnston ESnet Manager and Senior Scientist Lawrence Berkeley National Laboratory

2. 2 Summary – 1 ESnet’s mission to support the large-scale science of the DOE Office of Science results in a very unique network o The top 100 data flows each month account for about 25-40% of the total monthly network traffic – that is, 100-150 Terabytes out of about 450 Terabytes (450,000,000 Megabytes) o These top 100 flows represent massive data flows from science experiments to analysis sites and back At the same time ESnet supports all of the other DOE collaborative science and the Lab operations o The other 60-75% of the ESnet monthly traffic is in 6,000,000,000 flows

3. 3 Summary – 2 ESnet must have an architecture that provides very high capacity and high reliability at the same time o Science demands both o Lab operations demand high reliability To meet the challenge of DOE science ESnet has developed a new architecture that has two national core rings and many metropolitan area rings o One national core is specialized for massive science data o The other core is for general science and Lab operations o Each core is designed to provide backup for the other o The metropolitan rings reliably connect the Labs at high- speed to the two cores and connect the cores together

4. 4 Summary – 3 ESnet is designed, built, and operated as a collaboration between ESnet and the DOE science community and the Labs o ESnet planning, configuration, and even operation have input and active participation from the DOE science community and the DOE Labs ESnet also provides a collection of value added services (“science services”) that support the process of DOE collaborative science o National and international trust management services for strong user authentication across all DOE science collaborators (Lab, US university, and international research and education institutions) o Audio and video conferencing that can be scheduled world wide

5. 5 Summary – 4 Taken together, these points demonstrate that ESnet is an evolving creation that is uniquely tailored to meet the needs of the large-scale science of the Office of Science o This is not a network that can be purchased from a commercial telecom – and if it were, it would be very expensive o A very specialized set of services has been combined into a unique facility to support the Office of Science mission

6. 6 DOE Office of Science Drivers for Networking The large-scale science that is the mission of the Office of Science is dependent on networks for o Sharing of massive amounts of data o Supporting thousands of collaborators world-wide o Distributed data processing o Distributed simulation, visualization, and computational steering o Distributed data management These issues were explored in two Office of Science workshops that formulated networking requirements to meet the needs of the science programs (see refs.)

7. 7  Science Requirements for Networking  The network and middleware requirements to support DOE science were developed by the OSC science community representing major DOE science disciplines: o Climate simulation o Spallation Neutron Source facility o Macromolecular Crystallography o High Energy Physics experiments o Magnetic Fusion Energy Sciences o Chemical Sciences o Bioinformatics o The major supercomputing facilities and Nuclear Physics were considered separately Available at www.es.net/#research  Conclusions: the network is essential for o long term (final stage) data analysis and collaboration o “control loop” data analysis (influence an experiment in progress) o distributed, multidisciplinary simulation August, 2002 Workshop Organized by Office of Science Mary Anne Scott, Chair, Dave Bader, Steve Eckstrand. Marvin Frazier, Dale Koelling, Vicky White Workshop Panel Chairs: Ray Bair, Deb Agarwal, Bill Johnston, Mike Wilde, Rick Stevens, Ian Foster, Dennis Gannon, Linda Winkler, Brian Tierney, Sandy Merola, and Charlie Catlett

8. 8 Evolving Quantitative Science Requirements for Networks Science Areas considered in the Workshop (not Nuclear Physics and Supercomputing) Today End2End Throughput 5 years End2End Documented Throughput Requirements 5-10 Years End2End Estimated Throughput Requirements Remarks High Energy Physics 0.5 Gb/s100 Gb/s1000 Gb/shigh bulk throughput Climate (Data & Computation) 0.5 Gb/s160-200 Gb/sN x 1000 Gb/shigh bulk throughput SNS NanoScienceNot yet started1 Gb/s1000 Gb/s + QoS for control channel remote control and time critical throughput Fusion Energy0.066 Gb/s (500 MB/s burst) 0.198 Gb/s (500MB/ 20 sec. burst) N x 1000 Gb/stime critical throughput Astrophysics0.013 Gb/s (1 TBy/week) N*N multicast1000 Gb/scomputational steering and collaborations Genomics Data & Computation 0.091 Gb/s (1 TBy/day) 100s of users1000 Gb/s + QoS for control channel high throughput and steering

9. 9  Observed Drivers for the Evolution of ESnet ESnet Monthly Accepted Traffic Feb., 1990 – Feb. 2005 ESnet is Currently Transporting About 430 Terabytes/mo. (=430,000 Gigabytes/mo. = 430,000,000 Megabytes/mo.) and this volume is increasing exponentially TBytes/Month

10. 10 Observed Drivers for the Evolution of ESnet Oct., 1993 Aug., 1990 Jul., 1998 39 months 57 months 42 months ESnet traffic has increased by 10X every 46 months, on average, since 1990 Dec., 2001 TBytes/Month

11. 11 ESnet Science Traffic Since SLAC and FNAL based, high energy physics experiment data analysis started, the top 100 ESnet flows have consistently accounted for 25% - 40% of ESnet’s monthly total traffic o Much of this data goes to sites in Europe for analysis As LHC (CERN high energy physics accelerator) data starts to move, the large science flows will increase a lot (200-2000 times) o Both LHC, US tier 1 data centers are at DOE Labs – Fermilab and Brookhaven -All of the data from the two major LHC experiments – CMS and Atlas – will be stored at these centers for analysis by groups at US universities

12. 12 Source and Destination of the Top 30 Flows, Feb. 2005 Terabytes/Month Fermilab (US)  WestGrid (CA) SLAC (US)  INFN CNAF (IT) SLAC (US)  RAL (UK) Fermilab (US)  MIT (US) SLAC (US)  IN2P3 (FR) IN2P3 (FR)  Fermilab (US) SLAC (US)  Karlsruhe (DE) Fermilab (US)  Johns Hopkins 12 10 8 6 4 2 0 LIGO (US)  Caltech (US) LLNL (US)  NCAR (US) Fermilab (US)  SDSC (US) Fermilab (US)  Karlsruhe (DE) LBNL (US)  U. Wisc. (US) Fermilab (US)  U. Texas, Austin (US) BNL (US)  LLNL (US) Fermilab (US)  UC Davis (US) Qwest (US)  ESnet (US) Fermilab (US)  U. Toronto (CA) BNL (US)  LLNL (US) CERN (CH)  BNL (US) NERSC (US)  LBNL (US) DOE/GTN (US)  JLab (US) U. Toronto (CA)  Fermilab (US) NERSC (US)  LBNL (US) CERN (CH)  Fermilab (US) DOE Lab-International R&E Lab-U.S. R&E (domestic) Lab-Lab (domestic) Lab-Comm. (domestic)

13.  Enabling Future OSC Science: ESnet’s Evolution over the Next 5-10 Years Based both on the o projections of the science programs o changes in observed network traffic and patterns over the past few years it is clear that the network must evolve substantially in order to meet the needs of OSC science

14. 14 DOE Science Requirements for Networking - 1 1)Network bandwidth must increase substantially, not just in the backbone but all the way to the sites and the attached computing and storage systems o The 5 and 10 year bandwidth requirements mean that the network bandwidth has to almost double every year o Upgrading ESnet to accommodate the anticipated increase from the current 100%/yr traffic growth to 300%/yr over the next 5-10 years is priority number 7 out of 20 in DOE’s “Facilities for the Future of Science – A Twenty Year Outlook” The primary network requirements to come out of the Office of Science workshops were

15. 15 DOE Science Requirements for Networking - 2 2)A highly reliable network is critical for science – when large-scale experiments depend on the network for success, the network must not fail 3)There must be network services that can guarantee various forms of quality-of-service (e.g., bandwidth guarantees) 4)A production, extremely reliable, IP network with Internet services must support the process of science This network must have backup paths for high reliability This network must be able to provide backup paths for large-scale science data movement

16. 16 ESnet Evolution With the old architecture (to 2004) ESnet can not meet the new requirements The current core ring cannot handle the anticipated large science data flows at affordable cost The current point-to-point tail circuits to sites are neither reliable nor scalable to the required bandwidth

17. 17 ESnet’s Evolution – The Network Requirements Based on the growth of DOE large-scale science, and the resulting needs for remote data and experiment management, the architecture of the network must change in order to support the general requirements of both 1) High-speed, scalable, and reliable production IP networking for -University and international collaborator and general science connectivity -Highly reliable site connectivity to support Lab operations -Global Internet connectivity 2) High bandwidth data flows of large-scale science -Very high-speed network connectivity to specific sites -Scalable, reliable, and very high bandwidth site connectivity -Provisioned circuits with guaranteed quality of service (e.g. dedicated bandwidth) and for traffic isolation

18. 18 ESnet’s Evolution – The Network Requirements In order to meet these requirements, the capacity and connectivity of the network must increase to provide o Fully redundant connectivity for every site o High-speed access to the core for every site -at least 20 Gb/s, generally, and 40-100 Gb/s for some sites o 100 Gbps national core/backbone bandwidth by 2008 in two independent backbones

19. 19 10GE RTR Wide Area Network Technology RTR optical fiber ring “DWDM” Dense Wave (frequency) Division Multiplexing provides the circuits today typically 64 x 10 Gb/s optical channels per fiber channels (referred to as “lambdas”) are usually used in bi-directional pairs Lambda (optical) channels are converted to electrical channels usually SONET data framing or Ethernet data framing ESnet hub router ESnet core “tail circuit” “local loop” site LAN ESnet hub (e.g. Sunnyvale, Chicago, NYC, Washington, Atlanta, Albuquerque) ESnet site RTR A ring topology network is inherently reliable – all single point failures are mitigated by routing traffic in the other direction around the ring. RTR Site – ESnet network policy demarcation (“DMZ”) ESnet border router Site IP “gateway” router

20. 20 ESnet Strategy For A New Architecture A three part strategy for the evolution of ESnet 1) Metropolitan Area Network (MAN) rings to provide -dual site connectivity for reliability -much higher site-to-core bandwidth -support for both production IP and circuit-based traffic 2) A Science Data Network (SDN) core for -provisioned, guaranteed bandwidth circuits to support large, high-speed science data flows -very high total bandwidth -multiply connecting MAN rings for protection against hub failure -alternate path for production IP traffic 3) A High-reliability IP core (e.g. the current ESnet core) to address -general science requirements -Lab operational requirements -Backup for the SDN core -vehicle for science services

21. GEANT (Europe) Asia- Pacific New York Chicago Sunnyvale Washington, DC El Paso (ELP) Primary DOE Labs IP core hubs ESnet Target Architecture: IP Core + Science Data Network + MANs Possible new hubs Atlanta (ATL) CERN Seattle Albuquerque (ALB) SDN/NLR hubs Aus. Production IP core Science Data Network core Metropolitan Area Networks Lab supplied International connections ESnet Science Data Network Core (SDN) (NLR circuits) ESnet IP Core (Qwest) Metropolitan Area Rings San Diego LA

22. 22 First Two Steps in the Evolution of ESnet 1) The SF Bay Area MAN will provide to the five OSC Bay Area sites o Very high speed site access – 20 Gb/s o Fully redundant site access 2) The first two segments of the second national 10 Gb/s core – the Science Data Network – will be San Diego to Sunnyvale to Seattle

23. GEANT (Europe) Asia- Pacific New York Chicago Sunnyvale Washington, DC El Paso (ELP) Primary DOE Labs IP core hubs Science Data Network – Step One: SF BA MAN and West Coast SDN Possible new hubs Atlanta (ATL) CERN Seattle Albuquerque (ALB) SDN/NLR hubs Aus. Production IP core Science Data Network core Metropolitan Area Networks Lab supplied International connections ESnet Science Data Network Core (SDN) (NLR circuits) ESnet IP Core (Qwest) Metropolitan Area Rings San Diego LA

24. 24 ESnet SF Bay Area MAN Ring (Sept., 2005) Chicago (Qwest) El Paso Seattle and Chicago (NLR) LA and San Diego SF Bay Area λ 4 future λ 3 future λ 2 SDN/circuits λ 1 production IP SLAC Qwest / ESnet hub SNLL Joint Genome Institute LBNL NERSC LLNL 2 λs (2 X 10 Gb/s channels) in a ring configuration, and delivered as 10 GigEther circuits Dual site connection (independent “east” and “west” connections) to each site Will be used as a 10 Gb/s production IP ring and 2 X 10 Gb/s paths (for circuit services) to each site Qwest contract signed for two lambdas 2/2005 with options on two more Project completion date is 9/2005 ESnet IP core ring (Qwest circuits) ESnet SDN core (NLR circuits) ESnet MAN ring (Qwest circuits) NASA Ames Level 3 hub ESnet hubs and sites DOE Ultra Science Net

25. 25 References – DOE Network Related Planning Workshops  1) High Performance Network Planning Workshop, August 2002 http://www.doecollaboratory.org/meetings/hpnpw  2) DOE Science Networking Roadmap Meeting, June 2003 http://www.es.net/hypertext/welcome/pr/Roadmap/index.html 3) DOE Workshop on Ultra High-Speed Transport Protocols and Network Provisioning for Large-Scale Science Applications, April 2003 http://www.csm.ornl.gov/ghpn/wk2003 4) Science Case for Large Scale Simulation, June 2003 http://www.pnl.gov/scales/ 5) Workshop on the Road Map for the Revitalization of High End Computing, June 2003 http://www.cra.org/Activities/workshops/nitrd http://www.sc.doe.gov/ascr/20040510_hecrtf.pdfhttp://www.sc.doe.gov/ascr/20040510_hecrtf.pdf (public report) 6) ASCR Strategic Planning Workshop, July 2003 http://www.fp-mcs.anl.gov/ascr-july03spw 7) Planning Workshops-Office of Science Data-Management Strategy, March & May 2004 o http://www-conf.slac.stanford.edu/dmw2004 http://www-conf.slac.stanford.edu/dmw2004