1. Internet2: Supporting Next Generation Networking Christian Todorov August 16, 2010 NASA IT Summit 2010

2. Session Agenda Internet2 Overview Network Architecture Network Services IP Circuit Service Use of Dynamic Circuits Waves and Private Networking Supporting Advanced Research Discussion

3. INTERNET2 OVERVIEW

4. The Internet2 Network Overview Layer 1: Managed wavelengths from Level(3) Communications Level(3) owns and manages Infinera optical gear: responsible for software upgrades, equipment maintenance, remote hands, sparing, NOC services Internet2 NOC has total provisioning control Layer 2: Internet 2 owned and managed equipment Using OSCARS and DRAGON GMPLS control plane Layer 3: Internet2 owned and managed Juniper MX960s Expanded Observatory Platform for layer 1/3 network performance data collection, collocation, experimentation perfSONAR integration for intra- & inter-network performance analysis International connectivity Layer 1 network extended to international exchange points in Seattle, Chicago and New York City Peering points in Seattle, PAIX, Equinix Chicago 4

5. Internet2 Connected Community 217 Leading US research institutions ~50,000 K20, libraries, museums, zoos and aquariums 38 states covered by SEGP program Federal network peerings and International peerings reaching 80 countries Private networking provided to NOAA and Department of Energy (ESnet); Piloting with USFS Named participant in the FCC’s RHCPP; connecting rural healthcare networks Actively working with the VA JPL and Goddard are Affiliate Members Internet2 peers with NASA networks: NREN and NISN as well as others such as DREN and USGS 5

6. Internet2 Universities 217 University Members as of June 2010

7. Sample Listing of Government Affiliates

8. Internet2 Affiliate Members ACUTA: The Assn. for Information Communications Professionals in Higher Ed. Altarum Ames Laboratory, US Department of Energy Argonne National Laboratory Association of Universities for Research in Astronomy Berklee College of Music Brookhaven National Laboratory CineGrid, Inc. Coalition for NeCleveland Institute of Music tworked Information (CNI) Department of Veterans Affairs (VA) Desert Research Institute EDUCAUSE ESnet Federal Highway Administration, Turner- Fairbank Highway Research Center Fermi National Accelerator Laboratory HIMSS (Healthcare Information and Management Systems Society) Howard Hughes Medical Institute IEEAF (Internet Educational Equal Access Foundation) Indian Health Services Indiana Higher Education Telecommunication System (IHETS) ISOC (Internet Society) Jet Propulsion Lab JSTOR Laboratory for Telecommunications Sciences (LTS) Lawrence Berkeley National Laboratory Library of Congress Los Alamos National Laboratory Manhattan School of Music NASA Goddard Space Flight Center National Archives and Records Administration National Emergency Number Association (NENA)

9. Internet2 Affiliate Members National Geographic National Institutes of Health National Science Foundation New World Symphony NIST (National Institute of Standards and Technology) NITLE (National Institute for Technology and Liberal Education) NOAA (National Oceanic & Atmospheric Administration, Washington, D.C.) Oak Ridge National Laboratory OCLC Online Computer Library Center Pacific Northwest National Laboratory Princeton Plasma Physics Laboratory Ruth Lilly Health Education Center Stanford Linear Accelerator Center SURA (Southeastern Universities Research Association) The Children's Hospital of Philadelphia The Philadelphia Orchestra Association The World Bank Thomas Jefferson National Accelerator TOPIX (Torino E Piemonte Exchange Point) UCAR/NCAR (University Corporation for Atmospheric Research) United Nations System of Organizations (UN) United States Antarctic Program (USAP) United States Department of Agriculture, Forest Service (USFS) United States Department of Commerce Boulder Labs United States Department of State United States Holocaust Memorial Museum University of North Carolina General Administration

10. 10 The Internet2 Network Built on dedicated fiber from Level(3) Communications – 13,000+ mile footprint Currently based on 10G waves; moving to 100G Built on Infinera innovative optical technology Simple and convenient add/drop technology Simple and convenient wave setup Demonstrated high reliability in initial period of operation on the Level(3) network Economics of Infinera system are disruptive in the market place Architecture has maximum flexibility Every direct connector can access every wave on the system if needed Adding add/drop points doesn’t require network redesign Network Design

12. Internet2 Network International Connectivity Slide 12 T640 PacWave: APAN/TransPac2, AARnet, KOREN/KREONet2, CA*Net4, GEMNET, REANNZ, France Telecom (v6), HARNET, CERN StarLight: CERN, JGN2, ASNet, CERN, KOREN/KREONet2, CA*Net4, CERN, JGN2, SINET, GLORIAD, TANet2/TWAREN NGIX-Ames: AARnet PacWave:UNINET, SINET, QatarFN, APAN/TransPac2 TANet2/TWAREN, SingAREN redCLARA, CUDI via CALREN/PacWave CUDI via UTEP / UT AMPATH/SFGP:ANSP, REACCIUN-2, redCLARA NGIX-East/A-Wave: CLARA, GEANT, RNP2 MANLAN:TANet2/TWAREN, TENET, MCIT/ENERGI, LHCnet, QatarFN, CA*Net4, SURFNet, CERN, GEANT(2), SINET, NetherLight/IEEAF

13. Europe-Middle East Austria (ACOnet) Belgium (BELNET) Croatia (CARNet) Czech Rep. (CESNET) Cyprus (CYNET) Denmark (Forskningsnettet) Estonia (EENet) Finland (Funet) France (Renater) Germany (G-WIN) Greece (GRNET) Hungary (HUNGARNET) Iceland (RHnet) Ireland (HEAnet) Israel (IUCC) Italy (GARR) Latvia (LATNET) Lithuania (LITNET) Asia-Pacific Americas Argentina (RETINA) Brazil (RNP2/ANSP) Canada (CA*net) Chile (REUNA) Mexico (Red-CUDI) United States (Abilene) Peru (RAAP) Venezuela (REACCIUN-2) Last updated: April 2005 M Luxembourg (RESTENA ) alta (Univ. Malta) Netherlands (SURFnet) Norway (UNINETT) Poland (POL34) Portugal (RCTS2) Qatar (Qatar FN) Romania (RoEduNet) Russia (RBnet) Slovakia (SANET) Slovenia (ARNES) Spain (RedIRIS) Sweden (SUNET) Switzerland (SWITCH) United Kingdom (JANET) Turkey (ULAKBYM) *CERN Australia (AARNET) China (CERNET, CSTNET, NSFCNET) Hong Kong (HARNET) Japan (SINET, WIDE, JGN2) Korea (KOREN, KREONET2) Singapore (SingAREN) Philippines (PREGINET) Taiwan (TANet2, ASNet) Thailand (UNINET, ThaiSARN) Algeria (CERIST) Egypt (EUN/ENSTIN) Morocco (CNRST) Tunisia (RFR) South Africa (TENET) Central Asia Africa Armenia (ARENA) Georgia (GRENA) Kazakhstan (KAZRENA) Tajikistan (TARENA) Uzbekistan (UZSCI) Internet2 Network International Connectivity Slide 13

14. Internet2 Network Federal Peering Slide 14 T640 PacWave:DREN, NREN, ESnet StarLight:TeraGrid, NREN, DREN, NISN, USGS, ESNet NGIX-Ames: NREN, DREN, NISN, USGS DC: DRAGON NGIX:DREN, USGS, NISN, NREN, Wright Patt AFB, ESNet New York: ESnet

15. Network Architecture

16. Objectives The vision for the Internet2 Network is a seamless integrated network facility that allows for applications and users to transparently utilize the services and network layers that most appropriately serve their needs, when they need it, in a cost effective manner. This network facility will allow users to focus on their work and not on the network.

17. Scalability and Operability The Internet2 Network is based on a unique arrangement with Level 3 that represents a hybrid approach to carrier provided resources. Internet2 has full control over the provisioning on the network but does not bear the responsibility of supporting and maintaining the physical infrastructure: fiber, amps, transport equipment, etc. Level 3’s support of the physical network frees Internet2 of having higher levels of specialized engineering resources dedicated to network support. The Internet2 NOC has a full view into the underlying transport equipment and works jointly with a dedicated NOC group within Level 3. The Internet2 network is constructed on a dedicated fiber pair and with dedicated transport equipment The Infinera, and Juniper equipment used in the network are 40G capable and each has 100G on their roadmaps 17

18. Collocation Facilities 30 facilities across the US 27x Level 3 1x Westin Building, Seattle 1x 32 AoA, New York 1x 710 N. Lakeshore Drive, Chicago Dedicated suites or colo areas – secured Supports Internet2, ESnet, NOAA and some Internet2 connector equipment

19. Optical Network Infinera optical transmission equipment Managed by Level 3 Communications Internet2 has direct provisioning control Internet2 has direct TL-1 monitoring Infinera nodes Redundant Add / Drop (28) Regeneration (28) Optical Amplification (193)

20. Internet2 Network IP Network Services Carrier-class IP service Natively supports IPv4 and IPv6, as well as multicast and jumbo frames Carried over 10Gbps waves on the extremely reliable Infinera backbone 40Gbps and 100Gbps are future enhancements Backhaul to the nearest router is included in the IP connection Redundancy options One option is a geographically diverse backup connection. This connection is viewed as a backup and the aggregate traffic flow over the primary and redundant circuit must not exceed that of the primary circuit Internet2 currently supports 2 x 5 Gig and 2 x 10 Gig connections as well. These can be in geographically diverse locations. Protection against fiber cuts using the automatic restoration capabilities of the Juniper MX’s MPLS Fast Reroute as an optional service Additional IP circuits are also available Slide 20

21. Internet2 Network WaveCo Static Circuit Services Are currently set up manually by the Internet2 NOC. Can have any bandwidth up to a full 10Gbps wave or multiple 10G waves Can be provisioned directly over the Infineras (waves) or through the Junipers (sub-rate circuits) This provisioning can occur over the same physical interface that is used for the Dynamic Circuit Network Circuits can be offered both as protected and unprotected Can be ordered for a period of weeks up to years Have a fee for service that is based on distance, bandwidth, duration and protection scheme. Slide 21

22. Internet2 Network Dynamic Circuit Network The dynamic circuit network (ION) automatically provisions circuits among participants deploying the dynamic networking control plane and protocol Each participant may connect to one or more circuits allowing them to customize the infrastructure to meet their application needs The ION service is currently based on connections into the Juniper MX960s. As the ION service grows and other R&E networks across the world “peer” with the ION Network, through exchange points, world wide reachability will grow. Slide 22

23. ION Service ION has moved from a SONET-based network on the Ciena’s to an MPLS-based service operating on the current IP network. This transition took place during the summer ION connectivity and provisioning options for Internet2 Network Participants will increase. A Connector need only maintain a single connection that can provide both IP and ION services on the same port. The Connector is empowered to appropriate resources in a way that works with their regional model. The shared MPLS model can increase the efficiency of the backbone. Bandwidth reserved for a circuit instantiation is available for use by other network users when the circuit owner is not utilizing the circuit for transfer There’s opportunity to provide circuits that can burst above their requested commit rate, if there is sufficient headroom available. ION is a production service managed by the IU NOC. ION circuits are provisioned using a simple and secure web based interface. ION is available to any institution regardless of their membership status. The goal is to make the service available to the entire connector community. Connectors or sites that use ION should contact us to discuss the best way to utilize this new model. Contact Dale Finkelson (dmf@internet2.edu) for further information.

24. I2 DCN Software Suite OSCARS (IDC) Originally an ESnet Project, now joint development by Internet2 and ESnet Open source project maintained by Internet2 and ESNet DRAGON (DC) NSF-funded Open source project maintained by USC ISI EAST and MAX

25. InterDomain Controller (IDC) Protocol (IDCP) Developed via collaboration with multiple organizations Internet2, ESnet, GEANT2, Nortel, University of Amsterdam, others The following organizations have implemented/deployed systems which are compatible with this IDCP Internet2 Dynamic Circuit Network (ION) ESNet Science Data Network (SDN) GÉANT2 AutoBahn System Ciena (Nortel) (via a wrapper on top of their commercial DRAC System) Surfnet (via use of above Nortel solution) LHCNet (use of I2 ION Software Suite) Nysernet (use of I2 ION Software Suite) University of Amsterdam (use of I2 ION Software Suite) DRAGON Network The following "higher level service applications" have adapted their existing systems to communicate via the user request side of the IDCP: LambdaStation (FermiLab) TeraPaths (Brookhaven) Phoebus

26. ION Control Plane

27. Multi-Service/Domain/Layer/ Vendor Provisioning Regional Network Internet2 Network ESNet Dynamic Ethernet TDM GEANT IP Network (MPLS, L2VPN) Ethernet Router SONET Switch Ctrl Element Domain Controller LSP Data Plane Control Plane Adjacency Multi-Domain Provisioning Interdomain ENNI (Web Service and OIF/GMPLS) Multi-domain, multi-stage path computation process AAA Scheduling TDM Slide from Tom Lehman, ISI-East GUI XML API

28. Application Interfaces to ION Phoebus is network middleware to facilitate high throughput for the non-wizard Based on a “Session” Protocol (Layer 5) that talks to Phoebus Gateways These Gateways do Transport protocol translation and tuning Transparently allocates circuits over the Internet2 ION TeraPaths is a system from BNL Creates end-to-end virtual paths and guaranteed bandwidth for specific flows with DiffServ marking Web interface for manual reservations and a Web Services interface for automatic reservation Allocates resources on the ESNet SDN or Internet2 ION

29. Phoebus Gateways are placed at network access points and can be thought of as network “on-ramps At a high level, the goal is to get network data to these gateways which can then intelligently move the data as efficiently as possible to the other edge of the network Phoebus will automatically utilize ION resources as needed and as available At the other edge, it again appears as legacy traffic does today, thus neither client or server need to be modified

30. Waves and Private Networking Internet2 makes dedicated waves available over its network to support researchers and organizations Internet2 has provided nation-wide dedicated network infrastructure to support the needs of advanced research networks like ESnet (DoE) and N-Wave (NOAA) allowing for private research networking that has access to the broader Internet2 community

31. Monitoring and Measurement

32. Dependence on the network is increasing Distributed applications & moving larger data sets Network is growing much more complex Need to better understand the network Transparent network operation User must know what performance levels to expect Network operators must be able to demonstrate that the network meets or exceeds those expectations. Application developers must have access to tools that differentiate between network problems and application problems. Internet2 Network Observatory Slide 32

33. Observatory Services Several tools and services are available for network troubleshooting: Active testing to the middle to divide and conquer OWAMP (latency), BWCTL (throughput), NDT (last-mile/end station) perfSONAR (cross-domain tool results, link utilization, and test point locations) BGP feeds to RouteViews Link utilization at 10 second intervals “visible backbone” gives router configuration snapshots “Router” proxy for interactive query http://www.internet2.edu/observatory/ (data views) http://noc.net.internet2.edu/ 33

35. perfSONAR A way to export measurement data from your network, and consume it from many networks (including your own) “Interoperable network measurement middleware” A set of schemas / “APIs” and protocols 35

36. perfSONAR Integrates Network measurement tools and archives (e.g. stored measurement results) Data manipulation Information Services Discovery Topology Authentication and authorization 36

37. 37 – 8/29/2015, © 2009 Internet2 37 FNAL (AS3152) [US] ESnet (AS293) [US] GEANT (AS20965) [Europe] DFN (AS680) [Germany] DESY (AS1754) [Germany] measurement archive m1 m4 m3 measurement archive m1 m4 m3 measurement archive m1 m4 m3 m1 m4 m3 m1 m4 m3 measurement archive performance GUI user Analysis tool Many collaborations are inherently multi-domain, so for an end-to-end monitoring tool to work everyone must participate in the monitoring infrastructure

38. Meeting Upcoming Application Demands 38

40. Demands on the Network Entering the age of large scientific facilities Large Hadron Collider at CERN Very Long Baseline Arrays (radio astronomy) Large Synoptic Survey Telescope (2010-13) – 30TB/night An increasingly diverse set of demanding applications are utilizing network resources Telemedicine: BIRN project, proteomics, tele-surgery, remote ICU, radiology: high-resolution 3D color fMRI brain scan = 4.5PB Telepresence: master classes, virtual classrooms, tele-psychiatry High performance video delivery: Uncompressed HD, Cinegrid Disaster Recovery and distributed storage 40

41. Tier 1 to Tier 2 Traffic

42. The New Requirements High performance applications are dependent on high performance networks Networks must be fast, reliable, scalable, have flexible architectures, be cost effective, capable of delivering multiple services across multiple network layers, easy to operate and maintain, and have a view towards the future Enable the user – the network as a service 42

43. Questions? 43

44. 44