1. Recent and Planned Developments in ESnet Michael Ernst, BNL Slides courtesy of Mike Bennett, ESnet Dedicated Meeting with NII May 14, 2013 Tokyo

2. 2 Energy Sciences Network Overview 5/31/2016 2 A national network, optimized for science: connecting 40 labs, facilities with >100 networks optimized for massive science data flows offering capabilities not available commercially $34.5M in FY12 (40 staff) $62M ARRA grant funding: world’s first 100G network at continental scale optical fiber assets, access to spectrum easier (and cheaper) growth On the web: www.es.net fasterdata.es.net my.es.net

3. 3 Not the Commercial Internet  Engineered for massive traffic flows:  emphasis on lossless transport  bandwidth guarantees spanning multiple networks  high-speed (n x 100G) data transport  distributed performance monitoring platform  Optimized for DOE science missions:  extensive scientific outreach, requirements-gathering  visualization tools for monitoring health of science data flows  global leadership in network standards, middleware, architecture  Integrated with scientific user support, advocacy, outreach:  rapid diagnosis of data transport issues, around the world  global advocacy for science-optimized network /security architectures  partnership and outreach with facilities, VOs, peer organizations 5/31/2016

4. 4 SNLL PNNL LANL SNLA JLAB PPPL MIT/ PSFC BNL AMES LLNL GA JGI LBNL SLAC NERSC ORNL ANL FNAL Salt Lake INL GFDL PU Physics SEAT STAR CHIC WASH commercial peering points R&E network peering locations NASH ATLA HOUS NEWY AOFA BOST KANS DENV ALBQ LASV BOIS SACR ELPA SDSC LOSA Routed IP 100 Gb/s Routed IP 4 X 10 Gb/s 3 rd party 10Gb/s Express / metro 100 Gb/s Express / metro 10G Express multi path 10G Lab supplied links Other links Tail circuits Major Office of Science (SC) sites LBNL Major non-SC DOE sites LLNL CLEV ESnet optical transport nodes (only some are shown) MANLAN ESnet managed 10G router 10 ESnet managed 100G routers 100 Site managed routers 100 10 100 10 1 100 10 100 10 100 10 100 10 100 SUNN ESnet PoP/hub locations LOSA ESnet optical node locations (only some are shown) 10 100 EQCH Geography is only representational ESnet5 March 2013 1 100 10 SUNN 100 10 100 CHAT

5. 5 The Trend Continues … ’90 ’91 ’92 ‘93 ’94 ’95 ’96 ’97 ’98 ’99 ’00 ’01 ’02 ’03 ’04 ’05 ’06 ’07 ’08 ’09 ’10 ’11 ’12 ’13 Year 100 PB 10 PB 1 PB 100 TB 10 TB 1 TB 100 GB Bytes/month transferred by ESnet

6. 6 Facilities now Coupled by Networks Analyzer Crystals XES Detector Diffraction Detector Injector X-ray Beam Apertures LCLS Beam X-ray diffraction (structure) Liquid-jet Injection of  m-size crystals X-ray emission spectroscopy (Chemistry at the catalytic site) charge density/spin state ligand environment Using free-electron laser (LCLS) at SLAC to take ‘snapshots’ of catalytic reaction in Photosystem II Data transported to HPC resource (NERSC) for real-time computational analysis One experiment triples NERSC’s network utilization Kern et al (2012) PNAS 109: 9721 Sierra et al (2012) Acta Cryst D68: 1584 Mori et al (2012) PNAS 109: 19103 Optical pump Source: Nicholas Sauter, LBNL

7. 7 SLAC / LCLS  NERSC (http://my.es.net) 5/31/2016 All NERSC Traffic Photosystem II X-Ray Study

8. 8 Near Term Developments Work Completed Recently  Link acceptance testing at: o LBNL, ANL  100G production connections to: o ORNL and ANL  100G production connections to exchange points & R&E peers: o MANLAN, CENIC, I2 (Chicago), SINET (WIX) Work in progress  100G production connections to: o BNL, FNAL, LBNL, LLNL, and NERSC o 100G production connections to exchange points & R&E peers: o Starlight, PACWAVE  Diverse fiber laterals & diverse optical nodes at ANL & FNAL  Lots of cleanup & consolidation at the hubs, moving connections from the Juniper MX’s to the ALUs  Normalize 100G Testbed infrastructure

9. 9 A First Pass at Planning the Future  Rough Approximation o Look at bisection bandwidth – Cut the network in half (vertically) and examine capacity o Look at traffic coming into the network – Most of the traffic coming into ESnet terminates there » No transit services o Try to forecast when we’ll need to make changes – Use our traffic growth trend – Try to forecast when component costs decrease – Put some milestones on the plot  This gives us lots to think about o Large margin of error, but it’s a start

10. 10 Backbone & Traffic Metrics  Backbone Bisection Bandwidth:  Sum of inbound traffic crossing red line from Sites & Peers, converted to Gb/s ESnet Site Peers Site Peers  Average Accepted Traffic:  Sum of Links crossing the red line

11. 11

12. 12 new waves starting Early 2014 Optical system full in 2020; 88 x 100G 10x100G on all routes by 2017; start deploying ESnet6 routed net exceeds ESnet4 complexity Internet2 contract expires Add routers, optical chassis incrementally starting 2015

13. 13 Things ESnet is considering  How much ‘head-room’ is enough o Need less ‘bisection bandwidth’ if traffic is local vs cross-country o Need more for unpredictable users o Need less for predictable steady-state users – Could automated science work-flow systems be more predictable, and sustain utilization >75%?  Some changes that could affect growth rates o Science DMZ adoption reducing impacts of poorly performing firewalls & security infrastructure o PerfSONAR adoption & deployment allowing people to discover & fix broken infrastructure o Data mobility tools enhancement & adoption o Changes in computing models o Adoption rate of 100GE will drive the price curve of 100G components down

14. 14 Summary  Lots of changes in the last nine months and still lots of work to do o Finish connecting sites at 100G o Clean up and optimize hubs o Add 100G peers  The dust hasn’t settled on Esnet5 and they’re already thinking about turning up new waves – Start this all over again in a couple of years! – Start on ESnet6 deployment as early as 2017

15. 15 backup

16. 16 perfSONAR: New Dashboard for users  Visualization of perfSONAR throughput and packet loss data  Easy to build a table with a specific community of interest  Results displayed as a grid of green/yellow/red making it easy to identify problems  Dashboards for a few communities developed and available at http://ps-dashboard.es.net/

17. 17 World-Wide perfSONAR-PS Deployments: 536 bwctl nodes, 505 owamp nodes as of Jan 7 5/31/201617